Clemens Holzmann Spatial Awareness of Autonomous Embedded Systems
VIEWEG+TEUBNER RESEARCH
Clemens Holzmann
Spatial Awareness of Autonomous Embedded Systems
VIEWEG+TEUBNER RESEARCH
Bibliographic information published by the Deutsche Nationalbibliothek The Deutsche Nationalbibliothek lists this publication in the Deutsche Nationalbibliografie; detailed bibliographic data are available in the Internet at http://dnb.d-nb.de.
Dissertation Universität Linz, 2008
1st Edition 2009 All rights reserved © Vieweg+Teubner | GWV Fachverlage GmbH, Wiesbaden 2009 Editorial Office: Christel A. Roß | Anita Wilke Vieweg+Teubner is part of the specialist publishing group Springer Science+Business Media. www.viewegteubner.de No part of this publication may be reproduced, stored in a retrieval system or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording, or otherwise, without the prior written permission of the copyright holder. Registered and/or industrial names, trade names, trade descriptions etc. cited in this publication are part of the law for trade-mark protection and may not be used free in any form or by any means even if this is not specifically marked. Cover design: KünkelLopka Medienentwicklung, Heidelberg Printing company: STRAUSS GMBH, Mörlenbach Printed on acid-free paper Printed in Germany ISBN 978-3-8348-0798-4
To my mother and father, who have always believed in me and supported me all the way. Thank you.
Acknowledgements I would like to especially thank the following people who contributed to this thesis in various ways and without whose support it would not have been possible to finish it in such a short time and with the present extent and quality: • My supervisor Univ.-Prof. Mag. Dr. Alois Ferscha, for giving me the opportunity to work at his department and supporting me in every way with this dissertation. He offered me an interesting topic to work on, provided generous financial support with regard to the professional equipment I could use in my research as well as for being able to publish and present results at international conferences, he was always available for discussions, helped with useful suggestions to keep the thesis on track and in time, and allowed me to realize and develop my ideas in an industrial research project with Siemens AG Germany. Last but not least, he gave me all the time I needed when actually writing the thesis. • Dipl.-Ing. Manfred Hechinger, who implemented major parts of the software framework presented in Chapter 6 as well as the vibro-tactile space awareness application presented in Section 7.2. Also the framework architecture and many of the related concepts were developed in close cooperation with him. • Univ.-Prof. Mag. Dr. Gabriele Kotsis, for reading this thesis as the second supervisor and being a great help particularly in its initial stage by providing her expertise with regard to technical issues and writing a dissertation in general. • The project partners of Siemens AG Germany, and in particular Dipl.-Inform. Dr. Andreas Zeidler, for their critical and useful feedback on parts of my work as well as for sharing their knowledge on real-world requirements. • The co-authors and reviewers of my publications, who provided helpful suggestions for their improvement and thus also contributed to a higher quality of the respective sections in this thesis. I would like to especially thank the reviewers of my submission to the Ubicomp 2005 doctoral colloquium,
VIII
Acknowledgements
who suggested to extend my sole focus on the spatial direction of artifacts at that time to different types of spatial relationships and anchor the work in real-world application domains. • Dominik Hochreiter and Dipl.-Ing. (FH) Bernadette Emsenhuber for their work on realizing the hardware of the vibro-tactile belt (cf. Section 7.2.1), and in this regard Dipl.-Ing. Andreas Riener for bringing in his expertise on haptic perception. • My partner Mag. Anja Razenböck, for supporting me emotionally and showing understanding for the many evenings and weekends I was working on this thesis. • All the colleagues I have come to know since my employment at the Johannes Kepler University Linz, who contributed to an inspirational and pleasant working environment and made the time at the university enjoyable and fruitful.
Dipl.-Ing. Mag. Clemens Holzmann
Abstract The invisible integration of technology into everyday objects like home appliances, cars and mobile phones, which is the declared vision and fundament of research in the field of pervasive computing, leads to huge quantities of smart objects which are situated in the surrounding physical space. Equipped with embedded systems technology, they become increasingly heterogeneous and interconnected, raising the challenge of a semantically meaningful interplay with each other. In this thesis, such physical objects with embedded computing and communication technology are referred to as digital artifacts. One approach to meet this challenge is to design and implement systems that are able to operate autonomously in the background, namely with as little human intervention as possible, and interact with humans in a more unobtrusive way. In order to achieve autonomy, two aspects are particularly important: (i) context-awareness, which refers to the ability of a digital artifact to acquire environmental information in order to become aware of its situation and adapt to changing situations at runtime, and (ii) context sharing, which relates to an artifact’s ability of exchanging context information with other artifacts in communication range. An essential part of the context of spatially distributed objects is their position, direction and spatial extension with respect to an external reference system or with respect to other objects. The focus of this thesis is to make digital artifacts aware of such context information in order to enable their autonomous adaptation to spatial changes in the environment, whereas our main interest is on qualitatively abstracted spatial relations and their use at the application level. Concepts for recognizing, representing and reasoning about qualitative spatial relations among autonomous artifacts and their changes over time are presented, as well as an according architecture which has been implemented in a flexible software framework that builds upon qualitative relationship abstractions and the rule-based inference of conclusions from them. Special attention is paid to the adaptivity to different application domains, which influence the semantics of spatial abstractions and thus the behavior of digital artifacts. Evaluation results show the feasibility of the proposed concepts for developing spatially aware applications which involve one or more spontaneously interacting digital artifacts, and in particular that qualitatively abstracted relations can constitute an adequate basis for it.
Contents 1
Introduction 1.1 Problem Statement . . . . . . . . . . . . . . . . . . . . . . . . . 1.2 Contribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.3 Thesis Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2
Spatial Awareness 2.1 Context and Awareness . . . . . . . . 2.2 Autonomous Embedded Systems . . . 2.2.1 Exchange of Self-Descriptions 2.3 Mechanisms of Self-Organization . . 2.3.1 Self-Organization in Space . . 2.4 Spatial Context in Time . . . . . . . . 2.5 An Architecture for Spatial Awareness 2.6 Related Work . . . . . . . . . . . . . 2.6.1 Projects . . . . . . . . . . . . 2.6.2 Comparison . . . . . . . . . .
3
1 4 5 7
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
9 9 12 15 16 19 20 23 26 27 41
Representation of Space 3.1 Quantitative Representation . . . . . . . . . . . . . . . . . . 3.1.1 Spatial Abstraction with Zones-of-Influence . . . . . 3.1.2 Reference Systems for Position and Direction . . . . 3.2 Qualitative Representation . . . . . . . . . . . . . . . . . . 3.2.1 Qualitative Abstractions of Space . . . . . . . . . . 3.2.2 Recognition and Representation of Spatial Relations 3.2.3 Static and Dynamic Spatial Relations . . . . . . . . 3.2.4 Frames of Reference . . . . . . . . . . . . . . . . . 3.2.5 Spatiotemporal Relations . . . . . . . . . . . . . . . 3.3 Structure of Self-Descriptions . . . . . . . . . . . . . . . . 3.3.1 Quantitative Spatial Properties . . . . . . . . . . . . 3.3.2 Qualitative Spatial Relations . . . . . . . . . . . . . 3.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . .
. . . . . . . . . . . . .
. . . . . . . . . . . . .
45 46 46 49 51 51 53 56 60 61 66 67 70 72
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
XII
4
5
6
7
Contents
Distributed Spatial Reasoning 4.1 Overview of Qualitative Approaches . . . . . . . . . . . . . . 4.1.1 Properties and Closures of Binary Relations . . . . . . 4.1.2 Compositional Reasoning . . . . . . . . . . . . . . . 4.2 Reasoning about Positional and Directional Relations . . . . . 4.2.1 Related Approaches . . . . . . . . . . . . . . . . . . 4.2.2 Composition of Positional Relations . . . . . . . . . . 4.3 Spatial Relationship Inference and Distribution . . . . . . . . 4.3.1 General Concept by Exploiting Relation Properties . . 4.3.2 Distribution Algorithm Using Compositional Inference 4.4 Simulation Results . . . . . . . . . . . . . . . . . . . . . . . 4.5 Findings and Discussion . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . .
75 . 75 . 77 . 79 . 80 . 81 . 82 . 87 . 87 . 92 . 96 . 103
Rule-Based Spatial Awareness 5.1 Achieving Spatially Aware Behavior . . . . . . . . . . . . 5.1.1 Reasoning with Rules . . . . . . . . . . . . . . . 5.1.2 Using a Rule Engine . . . . . . . . . . . . . . . . 5.2 Rule-Based Qualitative Spatial Reasoning . . . . . . . . . 5.2.1 Inferring Relations and Application-Level Actions 5.3 Proof of Concept . . . . . . . . . . . . . . . . . . . . . . 5.4 Summary and Open Issues . . . . . . . . . . . . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
105 106 106 108 109 112 115 119
Zones-of-Influence Framework 6.1 Architecture . . . . . . . . . . . . 6.1.1 Design Considerations . . 6.1.2 Architecture Overview . . 6.1.3 Runtime Platform . . . . . 6.2 Components . . . . . . . . . . . . 6.2.1 Digital Artifact Service . . 6.2.2 Zones-of-Influence Service 6.2.3 Relations Service . . . . . 6.3 Runtime Behavior . . . . . . . . . 6.4 Discussion . . . . . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
121 122 122 123 126 127 127 130 135 140 143
Framework Evaluation 7.1 Development of Spatially Aware Applications 7.2 Application Scenarios . . . . . . . . . . . . . 7.2.1 Vibro-Tactile Space Awareness . . . . 7.2.2 Focus/Nimbus Awareness . . . . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
147 147 149 151 158
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
Contents
7.3 8
XIII
7.2.3 Spatiotemporal Awareness . . . . . . . . . . . . . . . . . 163 Comparison and Discussion . . . . . . . . . . . . . . . . . . . . 169
Conclusion 177 8.1 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179 8.2 Outlook . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185
Bibliography
191
List of Figures 2.1 2.2 2.3 2.4 2.5
Components of a digital artifact. . . . . . . . . . Exchange and comparison of self-descriptions. . Emerging “glider” pattern in “Game of Life”. . . Static spatial properties and relations of artifacts. General architecture for spatial awareness. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
Zone-of-Influence for a car’s braking distance [FHR+ 08]. . . . . . Types of Zones-of-Influence in 2-D. . . . . . . . . . . . . . . . . WGS84 spherical (ϕ,λ ,h) and Cartesian (x,y,z) coordinates. . . . . Recognition of spatial relations by comparing self-descriptions. . . Orientation relations of nearby extended objects, after [Her94]. . . Definition of static qualitative spatial relations in 2-D. . . . . . . . Cone-/projection-based orientation and qualitative distance [CFH97, RM04]. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.8 Definition of dynamic qualitative spatial relations in 2-D. . . . . . 3.9 Epsilon-neighborhood of a static qualitative orientation relation. . 3.10 Intrinsic and extrinsic orientation relations. . . . . . . . . . . . . 3.11 Qualitative temporal interval relations, cf. [All83]. . . . . . . . . . 3.12 Qualitative spatiotemporal relations. . . . . . . . . . . . . . . . . 3.1 3.2 3.3 3.4 3.5 3.6 3.7
4.1 4.2 4.3 4.4 4.5 4.6 4.7 4.8 4.9
14 16 18 21 24 47 48 50 54 55 56 57 58 59 60 63 65
An example for compositional reasoning. . . . . . . . . . . . . . 80 Composition of positional relations with same (top) and similar directions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84 Distribution of the transitive relation left. . . . . . . . . . . . . . . 90 Distribution of the symmetric and reflexive relation near. . . . . . 91 Recognition, composition and intersection of qualitative positional relations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94 Compositional inference and distribution started at artifact b. . . . 98 Simulated topologies full binary tree, line, mesh and ring. . . . . . 101 Traffic induced by different algorithms depending on the topology. 102 Relative accuracy of different algorithms compared with flooding. 103
XVI
List of Figures
5.1 5.2 5.3 5.4
Rule-based reasoning [PNF+ 08]. . . . . . . . . . . . . . . . . Acquisition of position and direction from two IS-900 trackers. Trajectories of quantitative spatial relations. . . . . . . . . . . Recognized and inferred qualitative relations. . . . . . . . . .
. . . .
108 116 117 119
6.1 6.2 6.3 6.4 6.5 6.6 6.7 6.8
Overview of the Zones-of-Influence Framework architecture. . . . Layered architecture of the OSGi framework, after [All05, All07]. Interfaces of the Digital Artifact Service. . . . . . . . . . . . . . . Interfaces of the Zones-of-Influence Service. . . . . . . . . . . . . Interfaces and classes for representing Zones-of-Influence. . . . . Interfaces and classes of the Relations Service. . . . . . . . . . . Time for relationship recognition (top) and repository update. . . . Repository update time (top) and involved number of relations. . .
124 127 129 131 133 137 142 143
7.1 7.2 7.3 7.4 7.5 7.6 7.7 7.8 7.9 7.10
Vibro-tactile belt with eight vibrator elements. . . . . . . . . . Controller of the belt. . . . . . . . . . . . . . . . . . . . . . . Person wearing the belt. . . . . . . . . . . . . . . . . . . . . . Visualization of the 2-D scene and the belt’s vibrator elements. Exemplary application scenario for focus/nimbus awareness. . Visualization and relations of the focus/nimbus application. . . Qualitative spatial relations at a point in time. . . . . . . . . . Time series of qualitative spatial relations. . . . . . . . . . . . Visualization and relations of the scenario shown in Figure 7.7. Visualization and relations of the scenario shown in Figure 7.8.
157 157 157 158 160 162 164 165 168 170
. . . .
. . . . . . . . . .
. . . . . . . . . .
List of Tables 2.1 2.2
Static characteristics of an artifact’s spatial situation. . . . . . . . Comparison of related work. . . . . . . . . . . . . . . . . . . . .
21 43
4.1 4.2
78
4.3 4.4 4.5 4.6
Extrinsic orientation and distance relations with their properties. . Comparison of approaches for reasoning about static spatial relations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Composition table for static orientation relations. . . . . . . . . . Composition table for static distance relations. . . . . . . . . . . . Composition table for combined orientation and distance. . . . . . Simulation results for the topology of Figure 4.6. . . . . . . . . .
5.1
Combinations of qualitative spatial and temporal relations. . . . . 113
6.1
Time in [s] for adding new artifacts/updating their self-descriptions. 141
7.1 7.2
Relevancy of spatial abstractions for application scenarios. . . . . 150 Framework aspects covered by implemented spatially aware applications. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171
81 85 85 86 99
Listings 3.1 3.2 3.3 3.4
Structure of an XML-based self-description. . . . . . . . . . Representation of a static and fragmented Zone-of-Influence. Representation of a dynamic Zone-of-Influence. . . . . . . . Representation of a qualitative relation. . . . . . . . . . . .
. . . .
. . . .
5.1 5.2 5.3
Maintenance rule for merging equal relations. . . . . . . . . . . . 111 Inference rule for the relation passing-by-right. . . . . . . . . . . 114 Inference rule for the relation passing-by-right-in-vicinity. . . . . 115
6.1
Specification of sensor data within a self-description. . . . . . . . 134
7.1 7.2 7.3 7.4 7.5
Zone-of-Influence for the vibro-tactile belt’s awareness shape. Rule for the vibro-tactile space awareness application. . . . . . Rule for the focus/nimbus awareness application. . . . . . . . Rules for the scenario shown in Figure 7.7. . . . . . . . . . . Rules for the scenario shown in Figure 7.8. . . . . . . . . . .
. . . . .
. . . .
. . . . .
67 69 70 72
153 155 161 167 168
List of Abbreviations API CSCW CSV DOF ECA ECEF EPC FoV GIS GML GPS GSM GUI HTTP IEEE IDE IR JEPD KML MEMS NED OSI OSGi PDA PML P2P RCC SQL TOTA URL Wi-Fi
Application Programming Interface Computer Supported Cooperative Work Comma Separated Value Degrees of Freedom Event-Condition-Action Earth-Centered Earth-Fixed Electronic Product Code Field of View Geographic Information Systems Geography Markup Language Global Positioning System Global System for Mobile Communications Graphical User Interface Hypertext Transfer Protocol Institute of Electrical and Electronics Engineers Integrated Development Environment Infrared Jointly Exhaustive and Pairwise Disjoint Keyhole Markup Language Micro-Electro-Mechanical Systems North-East-Down Open Systems Interconnection Open Services Gateway initiative Personal Digital Assistant Physical Markup Language Peer-to-Peer Region Connection Calculus Structured Query Language Tuples on the Air Uniform Resource Locator Wireless Fidelity
XXII
WGS84 XML ZoI
List of Abbreviations
World Geodetic System 1984 Extensible Markup Language Zone-of-Influence
1 Introduction Time and space are modes by which we think and not conditions in which we live. Albert Einstein, 1879-1955
At the time of this writing, we are facing a world with a huge, and ever increasing number of real-world objects with embedded computing and communication capabilities. This development is mainly driven by technological advances within the past few decades, which have made it possible to shrink sensors and actuators as well as processing and wireless communication technologies to a size that enables their integration into virtually everything. Things of everyday use in industrial settings, in manufacturing, in offices or in homes, like tools, appliances, machinery, furniture or even clothing have become the constituent “devices”, through which applications interact with the user, and there is strong evidence that this trend will continue. It can be observed that the “computer” is no longer a single desktop device, but is rather associated with services originating in the “digital world” and perceived through the “physical world”. The computer is more and more hidden in the fabric of everyday life, invisibly networked, and accessible everywhere; thus, we could say that computers become invisible, while their interfaces become omnipresent. A field of research tackling related challenges is ubiquitous computing, which was coined by Mark Weiser in a Scientific American article in 1991 [Wei91]. The vision of ubiquitous computing is the fusion of computing and communication technologies with the environment in such a way that the technology disappears. There are several terms such as calm computing, invisible computing, hidden computing, ambient intelligence, autonomous computing or pervasive computing, which all refer to the same vision but with a different focus at a time; in the context of this thesis we use the term pervasive computing to stress the pervasion of real-world objects with computing technology. Two key objectives of ubiquitous computing are identified in [Wei91]: ubiquity, namely the availability of computation and communication anytime and everywhere, and invisibility, which refers to the disappearance of computing technology. Invisibility can be achieved by embedding computational elements into real-
2
Introduction
world objects, but also by their autonomous operation [ECPS02] which makes them – due to the reduced human involvement – more or less disappear from the users’ consciousness. Pervasive computing is an inherently multidisciplinary topic, as many research fields including embedded systems, distributed systems, mobile computing, human/computer interaction and artificial intelligence [Sat01] address specific aspects for making the vision of “computing beyond the desktop” become reality. A typical service may consist of a large number of sub-services that themselves can be deployed on various devices at different – possibly changing – physical locations; the range of devices contributing to the service and their available interaction modalities may differ widely, as well. Obviously, the configuration and management of such dynamic “service landscapes” should be automated to a huge extent and not be of concern for the human user. This calls for some means of self-management and -organization, as their number, heterogeneity and complexity will sooner or later exceed the limits of human capability [HG03]. Moreover, interaction within such service environments will have to be more implicit (i.e. at the periphery of human attention) rather than explicit (i.e. at the focus of attention), so that it becomes very important that the individual devices manage themselves autonomously, with as little human intervention as possible. Clearly, the traditional approach of instructive systems [Wan04] with their deterministic and context-free nature appears less appropriate as an architecture for providing services by the interaction of embedded and networked devices; instead, a more autonomous system architecture [Hor01, KC03] is required. According to [FHdSR+ 07], two aspects of system properties contributing to their ability to operate autonomously in the background can be identified: self-management and self-organization. First, self-management relates to the individual devices. It stands for the ability of a single device to acquire information that can help to understand its situation or context, and to automatically adapt to changing contexts at runtime in a semantically meaningful way. There has been much research on this topic [CK00], which was mainly driven by technological advances in miniaturization of sensors and actuators for acquiring information about an environment and influencing it. One of the first definitions of context with regard to pervasive computing systems is given in [SAW94], where context is considered as the location of use, the collection of nearby people, hosts, and accessible devices, as well as the changes of such things over time. A more formal definition is given in [Dey00], defining context as any information than can be used to characterize the situation of an entity and denoting a system context-aware if it uses context to provide relevant information and/or services to the user. The second system property is self-organization, which re-
Introduction
3
lates to spontaneous configurations of such devices, and it stands for their ability to spontaneously (e.g. upon service requests or detection of certain spatial situations) join into ad-hoc “service ensembles” to e.g. negotiate and achieve ensemble goals – as for example the provision of a certain service to the user – through coordinated actions. Both self-management and self-organization have received much research attention in computer science over the past years [MMTZ06, SFH+ 03]. In particular, self-organization principles as inspired by nature attracted the attention of computer scientists [KE01, ZGMT04]. In the respective literature, self-organization is defined as a process in which pattern at the global level of a system emerge solely from numerous interactions among lower-level components of the system [CFS+ 01], where “pattern” refers to structure and organization in both space and time. Self-organization hence is way beyond centralized coordination, and complex collective behavior results from contextual local interactions between components [SFH+ 03]. Local interactions in turn are based on individual goals and the perception of the respective environment. The essence of self-organization is that system structure – and thus collective behavior – often appears without explicit trigger or pressure from outside the system, but is immanent to the system and results from interactions within it. System structure can evolve in time and space, may maintain a stable form or exhibit transient phenomena, or may grow or shrink in size, number or feature. An example for that is John Conway’s cellular automaton called “Game of Life”; it consists of a two-dimensional collection of cells which die, survive or become to live by simple local rules, whereas complex patterns emerge depending on an initial configuration of the automaton [Gar70]. The focus in this work is on autonomously operating real-world objects which are equipped with sensors, actuators, as well as with computing and wireless communication technology to support ad-hoc networking. These devices can have various different kinds of appearance (like shape, size, mobility, etc.) and embedded digital technology (e.g. mobile phones, smart appliances, smart rooms, etc.), and we refer to them as digital artifacts. A digital artifact has to have the ability to sense its context, and it must possess reasonable means to process and reason about the perceived context as well as the possibility to share its perceptions with others in range. Obviously, this is a fundamental requirement for such autonomous embedded systems, as sharing information about their context is a key property for collaboration and autonomous adaptation in order to reach ensemble goals based on local information gathering. As digital artifacts are by nature situated in physical space, their spatial properties (e.g. position and physical shape) – and in particular spatial relationships between them (e.g. distance and orientation) – are valuable context information
4
Introduction
and constitute a distinguishing feature of our work. Actually, most of the known phenomena of self-organization and -adaptation in nature are phenomena of selforganization in space [MZ05], and [ZM04] identifies the concept of space and the awareness of distributed components of their surrounding to play an important role for mechanisms of self-organization, as for example the coordination of activities by exploiting their spatial structure. In this regard, spatial abstractions are considered to be important means for implementing services which are distributed in physical space [Leo98, ZM04]. Self-organization is based on (direct or indirect) contextual local interactions between the components of a system; for this reason, both the inference of highlevel contextual information from spatial relationships, as well as standardized means for exchanging spatial information between the components are issues of research [HE03, MZ03]. This thesis is primarily concerned with techniques for making digital artifacts spatially aware (i.e. aware of spatial context information), which facilitates the semantically meaningful interaction among them.
1.1 Problem Statement The focus of this thesis is on mechanisms to provide digital artifacts, which are spatially distributed real-world objects with embedded digital technology, with an awareness about their spatial contexts. In this regard, an essential part of the information describing the spatial context of an artifact is its position, direction and spatial extension. In order to achieve such spatial awareness, it must have the ability to acquire its spatial context with sensors, possess means to reason about the perceived context and share this perception with others in communication range. Sharing spatial context information is a key property of autonomous artifacts, which allows them to adapt to changing spatial situations at runtime and thus contributes to their semantically meaningful, contextual interaction in space. As artifacts may have to coordinate their actions for providing certain services, we consider especially spatial relations among them as well as relationship changes to be of particular relevance. When developing context-aware applications, a tight coupling between the application and sensors is problematic due to the fact that it forces the programmer to deal with sensor details. Hence, low-level context information provided by sensors must be abstracted to be used by context-aware applications [Dey00]; for example, an application using location information may only be interested in highlevel information like rooms and buildings instead of geographical coordinates [FHO04a]. Generally speaking, abstracting context information is about separat-
1.2 Contribution
5
ing details which are not relevant for a certain application. Such symbolic abstractions of locations have been addressed by several researchers [Leo98, Sch95], with the aim to provide location information in a sensor-independent and more natural way that is closer to human concepts of space. Abstracted relations between the locations of devices are used in [HKG+ 05, KKG05], which are represented with meaningful names such as left or near. In this work, we go beyond utilizing just location information; in addition, the direction and spatial extension of mobile artifacts with respect to a global reference system and with respect to each other are taken into account. Our primary focus is on the investigation and development of concepts for recognizing and representing spatial relations between autonomous digital artifacts by using qualitative abstractions, as well as the inference of high-level context information out of it. The representation of and reasoning about qualitative spatial relations, which are concerned with the abstraction of continuous properties of the physical world and inferring knowledge from the respective qualitative representations, has already attracted much interest by researchers [CH01]. Compared with quantitative approaches, qualitative ones have clear advantages whenever the spatial cognition of humans is involved [Mus00] or systems with limited computing resources are concerned [Fre92b], among others. Summing up, the focus of this thesis can be stated as follows: What are concepts and architectures for making autonomous embedded systems aware of qualitatively abstracted spatial relations over time and using them in spatially aware applications? The thesis, however, does not address spatial representation and reasoning in general, but rather concentrates on methods for providing qualitative spatial relationship information to autonomous digital artifacts at application level. Central is the development of an architecture for enabling spatial awareness of autonomous artifacts without any kind of centralized instance, as well as its application to real-world scenarios. This includes the issues of recognizing, representing and reasoning about qualitative spatial relations both at certain points in time and by considering their changes over time.
1.2 Contribution Within the scope of this thesis, application-independent concepts that allow autonomously operating digital artifacts to become aware of and use spatial context
6
Introduction
information have been investigated and developed. In order to implement spatial awareness of artifacts among each other, quantitative and qualitative representations of spatial aspects are used. For the former, a concept referred to as Zones-of-Influence has been developed, which builds upon initial work published in [FHR+ 08]. Zones-of-Influence encode spatial properties of artifacts – in particular their absolute and relative position, direction and spatial extension – with numerical values that may be provided by sensors. They are explicitly defined twoor three-dimensional shapes of a certain size which are positioned and directed in physical space, thus representing geographic regions that are of relevance for the artifacts and their users or applications. Each digital artifact is associated with one or more of such zones at a time, together representing the relevant spatial knowledge which is distributed across them in physical space. In order to enable artifacts to autonomously recognize spatial relations to others around, they share their Zones-of-Influence through an exchange of generic, structured self-descriptions. This allows for determining spatial relations between their Zones-of-Influence, including distance, orientation or topological relations between the zones’ spatial extensions. In this regard, a data format for representing both spatial properties and determined relations within self-descriptions has been developed. A central point is the abstraction of spatial relations in a qualitative way, depending on the application domain which influences their semantics. As the relations an artifact is aware of can also be included in its self-description and exchanged with others, digital artifacts are able to reason about both self-determined and received relationships. A literature survey has been conducted in order to find suitable qualitative abstractions of relations between Zones-of-Influence on the one hand, and to identify approaches for inferring new relations on the other hand. We have developed a spatial calculus for compositional reasoning about positional relations as well as an algorithm which makes use of it for inferring and distributing spatial relations among autonomous digital artifacts over multiple hops, without the need for exchanging quantitative spatial properties between those that are out of communication range. In this regard, effects on the induced network traffic as well as the achieved accuracy have been evaluated by simulation means. A reference architecture for the spatial awareness of digital artifacts has been developed, demonstrating how the above concepts for recognizing, representing and reasoning about spatial relations among spontaneously interacting artifacts can be realized. A middleware framework has been prototypically implemented, with the aim to support the development of spatially aware applications for autonomous digital artifacts. For reasoning about spatial relations over time, in order to infer new relations or trigger appropriate application-level actions, this so-called Zonesof-Influence Framework makes use of rules. All relations are associated with time
1.3 Thesis Outline
7
intervals in which they exist, and new high-level relations are inferred by combining such intervals, for which reason inferred relations can in turn be combined with others. The qualitative relationship abstractions are encapsulated in components that can easily be exchanged at runtime, enabling artifacts to adapt the availability of spatial relations and their semantics to the current application demands. The proposed framework with its underlying concepts provides powerful means for facilitating the development of applications which take into account the contextual relations among autonomous digital artifacts. An evaluation with multiple application scenarios showing the relevance of spatial relations for developing spatially aware applications as well as the technical feasibility and quality of the Zones-ofInfluence Framework has been conducted.
1.3 Thesis Outline The thesis is structured as follows. Chapter 2 gives an introduction in contextawareness of autonomous embedded systems, with which physical objects are augmented and to which we refer to as digital artifacts then, motivates its significance for mechanisms of self-organization, discusses the role of spatial context with regard to their autonomy and semantically meaningful interaction among each other, and presents a general architecture for providing spatial awareness to artifacts. At the end of this chapter, a comprehensive survey and comparison of related work are given. Chapter 3 elaborates on quantitative and qualitative representations of spatial aspects, which are the basis for representing spatial properties of artifacts with so-called Zones-of-Influence as well as for recognizing and using spatial relations between them. In this regard, the exchange of such spatial contexts among artifacts with structured self-descriptions is discussed. Chapter 4 builds upon the qualitative abstractions of spatial relations and is concerned with the inference of knowledge from them. In this chapter, fundamentals of qualitative reasoning approaches are presented, and an algorithm for inferring and distributing qualitative spatial relationships among digital artifacts is proposed and evaluated. In Chapter 5, a rule-based approach for reasoning about qualitative spatial relations over time is presented, which is used for inferring new relations and triggering application-level actions upon observing certain patterns on the stored relations. The above concepts of exploiting spatial abstractions have been implemented in the flexible and modular Zones-of-Influence Framework presented in Chapter 6, which enables digital artifacts to maintain and use a spatial model of their environment. An overview of the architecture is given in this chapter, its components for supporting spatial awareness are described and implementation details as well as
8
Introduction
open issues are discussed. Chapter 7 eventually evaluates the implemented middleware framework by means of multiple real-world applications scenarios which cover the concepts presented in the preceding chapters. In Chapter 8, the thesis is summarized, the main contributions are discussed and an overview of open issues for possible future work is given.
2 Spatial Awareness The real power of the concept comes not from any one of these devices; it emerges from the interaction of all of them. Mark Weiser, 1952-1999
This chapter introduces spatial awareness of autonomous embedded systems, which is the main subject of this work. In Section 2.1, we first give an overview on context-awareness, which denotes a system’s property to acquire and use information about its environment, as well as on the abstraction of context for developing context-aware applications. Afterwards, Section 2.2 presents the components of autonomous embedded systems which are able to sense their context, process and reason about it and share this knowledge with others in communication range. These abilities contribute to their autonomous operation in the background and thus to a reduction of human involvement for their management and configuration. Mechanisms of self-management of individual devices and self-organization of spontaneous configurations of multiple devices are discussed in Section 2.3, which rely on the perception of context information and its local exchange among devices in range. In this regard, our focus is on the relevance of spatial information, as we are dealing with real-world objects which are distributed in physical space. The subsequent Section 2.4 thus deals with spatial context information, whereas a classification is given and our focus on spatial relations as well as their changes over time is emphasized. Afterwards, an architecture for spatial awareness is presented in Section 2.5, and Section 2.6 finally provides a comprehensive survey and comparison of related work addressing the issue of how to make autonomous embedded systems aware about abstracted spatial relations over time.
2.1 Context and Awareness Subject of the field of pervasive computing is the integration of technology into everyday objects. With computing power invisibly integrated in a huge number of networked objects it becomes necessary that they operate more autonomously,
10
2 Spatial Awareness
as the embedding of digital technology leads to constrained human-computer interfaces on the one hand, and the ubiquity of such technology-enriched physical objects makes it impossible for humans to individually manage their operation on the other hand. A contributor to the autonomy of embedded systems is contextawareness [SAW94, Sch95], which refers to the ability of a system to acquire environmental information in order to understand its situation or context – e.g. its location or spatial relations to other systems – and adapt to changing contexts at runtime. Context-awareness is considered important especially for applications where the environment changes frequently [Dey00, SAW94], which is commonly the case in mobile and pervasive computing and requires an adaptation to the current context of use. A survey on the foundations of context-aware computing and first research projects is given in [CK00]. Several definitions of context and awareness can be found in literature. In the field of computer supported cooperative work (CSCW), [DB92] considers an awareness of the others’ activities necessary for successful collaboration, and defines awareness as an understanding of the activities of others, which provides a context for your own activity. Awareness is usually supported with mechanisms for sharing information between collaborators in CSCW systems, as for example by means of document annotations or electronic conferencing systems. In the field of pervasive computing, the more specific term context-awareness refers to a system’s awareness of the context in which it is used. It has been introduced in [SAW94], where contextawareness is defined as a system’s ability to adapt according to the location of use, the collection of nearby people, hosts, and accessible devices, as well as to changes of such things over time. Three important aspects of context pointed out in [SAW94] are where you are, who you are with and what resources are nearby. A later definition of context is given in [BBC97], where it is also defined by means of examples, including the user’s location, the time of the day and the temperature. From these definitions, it can be observed that location and spatial proximity are considered important context information; in fact, [CK00] states that few contexts other than location have been used in actual applications. Our literature survey conducted in Section 2.6 leads to a similar result, namely that location and proximity are the most often used contexts in spatially aware applications (i.e. applications that are aware of spatial context information, cf. Section 2.5). Time is also considered important for context-aware applications [CK00, Dey00], and it allows for reasoning about
2.1 Context and Awareness
11
both contexts and their changes over time. However, there are many other types of context information, which [SBG99] classifies in (i) human factors (i.e. information on the user, his social environment and his tasks) and (ii) information on the physical environment (i.e. location, infrastructure and physical conditions). A more general definition of context can be found in [Dey00]: Context is any information than can be used to characterize the situation of an entity. An entity is a person, place, or object that is considered relevant to the interaction between a user and an application, including the user and application themselves. According to this – probably most often used – definition, it depends on the particular application scenario what kinds of information are context in that they have an influence on the application. We adopt this definition for our work, and consider context as any information than can be used to characterize the situation of a person or an object which is considered relevant for an application. In order to be able to use context information – i.e. information about a system’s physical environment – in an application, it has to be acquired with sensors first. Research on context-awareness is thus driven by technological advances to a certain extent, and many projects had a focus on technologies for sensing various physical contexts like location [WHFG92], direction [SBG99], light, noise, movement, skin conductance and temperature [BGS01, GSB02, HMS+ 01] or relative orientation and distance [HKG+ 05]. With emerging micro-electro-mechanical systems (MEMS) technology, smaller, more powerful and cheaper sensors are becoming available [GSB02] (e.g. MEMS-based accelerometers and gyroscopes for sensing linear and angular acceleration, respectively), which facilitates their integration into everyday objects at low cost. If the application has to have an impact on the physical world, actuators are required for changing the environment’s state. An actuator is any output to the physical world (cf. [Dey00]), as for example a mechanical switch, a light-emitting diode or a computer display. According to [Fer03], sensors and actuators can be considered as the input/output interface between the physical and the virtual world. For using sensed context information at the application level however, it must be abstracted to decouple sensor details that are not relevant for the application and therewith allow application programmers to use context at a higher level of abstraction [Dey01, Sch95], namely without having to consider low-level aspects of certain sensor technologies. This is a necessary step as sensors often do not provide context information in the form required by an application [SDA99]. With regard to the spatial focus of this thesis, sensor data has been abstracted with symbolic locations [Leo98] and qualitative spatial relations [KKG05], representing the
12
2 Spatial Awareness
respective spatial context at a high level and more naturally by using semantically meaningful names. As will be seen in Section 2.6, abstractions of sensors are actually employed in most projects which make use of context information. The acquisition, abstraction and use of context information are essential components of context-aware architectures. A more detailed view on a possible architecture of a context-aware system is given in [Fer03, FHO04a], where the following layers are in-between those for sensing and actuating: (i) context transformation (i.e. aggregation and interpretation of the low-level contexts into high-level information which is meaningful for the application), (ii) context representation (i.e. data structures representing the abstract context information of the previous layer), (iii) context prediction (i.e. inference of future from past contexts) and (iv) context triggering (i.e. rules for triggering actuators upon observing certain context conditions). Similar layered architectures are common in context-aware systems (e.g. in [Dey00, GSB02, HKG+ 05, LM98, SGKK04]), as is the use of rules for defining context-dependent application behavior (e.g. [BCFM03, GSB02, SAW94, SGKK04, TTH+ 04]). In our framework outlined in Section 2.5 and refined later in Chapter 6, sensed context goes through similar layers of abstraction until reaching the application, and a rule-based approach has been chosen for defining application-level behavior.
2.2 Autonomous Embedded Systems As stated in the previous section, context-awareness is a crucial contributor to the autonomy of embedded systems, which are considered as computer systems that are parts of larger systems and realize dedicated functions. In this section, the components of autonomous embedded systems are discussed in more detail and with respect to existing research, as well as their mechanisms for sharing contextinformation with other systems in order to allow for spontaneous contextual interaction with them. The following explanations are in large parts limited to aspects which are relevant regarding the conceptual and architectural contents in the remainder of this thesis. Since Mark Weiser’s seminal paper [Wei91], there has been a considerable amount of research on the embedding of computing technology in the physical environment. An example are wireless sensor networks [ASSC02], which typically consist of a huge number of sensor nodes deployed in an environment in order to sense certain phenomena. The nodes perform local processing on the sensed data and cooperate with other nodes in routing it back to a sink by means of multi-hop wireless communication. Another related field of research is tangible
2.2 Autonomous Embedded Systems
13
user interfaces [Fis04, IU97], which deals with user interfaces that augment the real physical world by coupling digital information to everyday physical objects and environments [IU97]. We also conducted research in this area by studying the use of physical objects with embedded direction sensing for the purpose of device control (cf. [FHR06b, FRH06, FRHR05, HRLF06]). The starting point of our work are physical artifacts augmented with autonomous embedded systems, which – in contrast to the two above examples – comprise sensing, actuating, computing and wireless communication capabilities, and allow for the autonomous provision of contextualized services by sharing information among each other without relying on an external infrastructure. The general aim of augmenting artifacts that are not computers themselves with computing technology is stated in [BGS01]: Computer-augmentation of artefacts will not be geared toward making them more computer-alike but at preserving their individual purposes and uses while enabling added value through digital information processing. The embedding of computing capabilities in everyday artifacts, and in particular their context-awareness and inter-networking capabilities, were subject of many projects surveyed in Section 2.6, as for example [BGS01, DBK+ 04, FHdSR+ 07, HMS+ 01, SGKK04]. We adopt the term digital artifact (often simply referred to as artifact in the following) for computer-augmented physical artifacts, which has been defined in [BGS01] as follows: A digital artefact is an everyday artefact augmented with computing and communication, enabling it to establish and exchange information about itself with other digital artefacts and/or computer applications. The concepts and solutions for spatial awareness developed in this thesis are based on digital artifacts. They consist of the following components also shown in Figure 2.1, which are either part of the augmented artifact’s original components or have been added by embedding computing and communication technology: • Sensors: acquisition of context information from the environment, e.g. using location, motion or optical sensors (cf. [SBG99]) • Actuators: output to the physical world, e.g. by creating mechanical motion or with light-emitting diodes
14
2 Spatial Awareness
Figure 2.1: Components of a digital artifact.
• Communication facilities: ad-hoc exchange of data with other artifacts and IT-systems in range, e.g. using IEEE 802.11 or IEEE 802.15 wireless communication • Runtime system: representation and reasoning about context information, which is either acquired with local sensors or received from other artifacts, and triggering respective application actions based on the current context • External services: provision of contextualized functionality to the artifact’s surroundings, as for example to users or other artifacts A digital artifact is typically a mobile device with a small form factor, constrained computing capabilities and limited energy resources due to battery operation, it is context-aware because of its sensors and operates autonomously (i.e. without centralized control), and it acts according to certain defined goals – like for example the provision of a certain service to the user – which may be achieved in cooperation with other artifacts. As this thesis is primarily concerned with software concepts for spatial awareness at the application level, hardware aspects regarding the embedding of miniaturized computing technology in everyday objects are beyond its scope. For evaluation purposes, notebook computers with attached sensors and actuators have been used. A possible platform for digital artifacts are Peer-its [FHdSR+ 07], which comprise the components listed above and are capa-
2.2 Autonomous Embedded Systems
15
ble of discovering and spontaneously interacting with other such artifacts (cf. Section 2.6.1 for more details).
2.2.1 Exchange of Self-Descriptions Clearly, information sharing is a key property for the collaboration among artifacts and their autonomous adaptation to changing conditions at runtime. Within the scope of this thesis, our focus is on sharing context information among digital artifacts, which we consider fundamental for a semantically meaningful, contextual interaction among them and thus beneficial to their autonomous operation in the background. According to [BGS01], context-awareness and ad-hoc information sharing are essential properties of digital artifacts, and they are enabler for emerging functionality (cf. Section 2.3). Ad-hoc context sharing has been the basis for interaction in several of the projects presented in Section 2.6.1; for example, [BGS01] employs message-based communication for broadcasting context information of artifacts via infrared transceivers mounted on the ceiling, [HMS+ 01] uses wireless communication for sharing contexts like e.g. movement patterns among artifacts in communication range, and XML-based documents containing information about the connections and spatial directions of physically connected artifacts are exchanged via a wired network in [KOY+ 05]. We employ peer-to-peer concepts for the interaction between digital artifacts in a spontaneous ad-hoc networking fashion, which appear to be more suitable than client-server architectures due to the decentralized nature of networked digital artifacts. In order to share context information, so-called self-descriptions are used for representing the contexts of artifacts and exchanged among them upon coming into communication range; each artifact then performs a pairwise comparison of its own with the received self-descriptions and triggers certain actions (cf. Figure 2.2). Hence, self-descriptions build up a standardized interface for the sharing of context information, and their ad-hoc exchange provides mutual awareness about e.g. physical properties and contexts to artifacts, which facilitates their interaction with regard to an open-world assumption (i.e. the interacting artifacts do not know each other in advance). This approach builds on previous work presented in [FHdSR+ 07, FHM+ 04, FHR+ 06a], where the authors suggest to use XML-based documents representing information about an artifact and exchanging them with other artifacts in range; upon sufficiently high similarity between two such documents at a time, the respective artifacts start to interact. Similarly, self-descriptions in this thesis are structured documents based on XML which contain information that may be relevant for the interaction among artifacts, as for example an artifact’s physical properties
16
2 Spatial Awareness
Figure 2.2: Exchange and comparison of self-descriptions.
like its size or weight, its capabilities in terms of provided services or spatial context information like position and direction. It enables artifacts to autonomously become aware of the contexts of others in a decentralized way, but has the disadvantages that just contexts of artifacts in communication range can be accessed and each artifact has to store and maintain all the information it may require. The scope of the present thesis is on the exchange of spatial information, whereas a detailed explanation of our representation of spatial contexts in self-descriptions, which we kept as simple as possible for the purpose of realizing and evaluating the proposed concepts, is given in Section 3.3. However, issues of how self-descriptions are exchanged from a technological viewpoint, under which circumstances (e.g. taking into account energy constraints) and among which artifacts (e.g. with respect to certain communication policies) are not considered.
2.3 Mechanisms of Self-Organization In the introduction, we argued that the growing number of computationally enhanced and increasingly interconnected, heterogeneous and complex everyday artifacts – be it home appliances, cars or mobile phones – calls for mechanisms of self-management and -organization which allow them to operate autonomously in the background, with no or little human interaction. The reason for that is that systems have become too complex and heterogeneous for humans to be able to use them without difficulties [HG03, KC03, MMTZ06]. According to [Fer07b, FHdSR+ 07], self-management stands for the ability of individual artifacts to operate autonomously by adapting to changing contexts at runtime, while self-organization stands for the spontaneous contextual interaction of multiple devices in order to achieve collective behavior. One approach to self-management is IBM’s autonomic computing vision [KC03], in which systems are able to self-configure, -heal, -optimize and -protect and thus function more or less independently from
2.3 Mechanisms of Self-Organization
17
human supervision, just based on high-level administrative goals and constraints. Self-organization goes further by not only autonomously controlling a system’s behavior, but creating its own structure which is not explicitly defined by a programmer. Self-organization has received much research attention in computer science over the past years, as can be seen from several recent survey papers [Mil07, MMTZ06, SFH+ 03]. In particular, mechanisms of self-organization which are inspired by nature (e.g. ants food foraging or nest building [MMTZ06]) have attracted the attention of computer scientists [KE01, MMTZ06, ZGMT04]. In the context of pattern formation in biological systems, [CFS+ 01] defines self-organization as follows: Self-organization is a process in which pattern at the global level of a system emerges solely from numerous interactions among the lowerlevel components of the system. Moreover, the rules specifying interactions among the system’s components are executed using only local information, without reference to the global pattern. According to [CFS+ 01], patterns – i.e. organized arrangements of objects in space or time – are emergent properties, which means that they are produced without external input but rather result from interactions among system components that are based on local information only. A prominent example is the cellular automaton “Game of Life” [Gar70]; it consists of a two-dimensional orthogonal grid of cells which are either white (“dead” cells) or black (“alive” cells). The cells interact with their eight neighbors at a time only, and may change – simultaneously, at discrete points in time – by using three simple rules: (i) each white cell with exactly three black neighbors becomes black, (ii) each black cell with more than three or fewer than two black neighbors becomes white and (iii) each black cell with two or three black neighbors remains black. Interestingly, although just simple rules about a cell’s local neighborhood are used, complex patterns such as the so-called “glider” shown in Figure 2.3 may emerge depending on the automaton’s initial configuration. A related approach is amorphous computing [AAC+ 00], which differs from cellular automata in that the “agents” are not regularly placed on a grid and that they do not operate synchronously [JBL+ 06]. A more general definition is given in [WH04], where the authors define selforganization as a dynamical and adaptive process where systems acquire and maintain structure themselves, without external control. In this regard, “structure” can be a spatial, temporal or functional structure, and “without external control” refers to the absence of any control instructions from outside the system, which does not exclude data inputs however. We adopt this definition for our work, and consider systems as self-organizing in some respect if they spontaneously, and de-
18
2 Spatial Awareness
Figure 2.3: Emerging “glider” pattern in “Game of Life”.
pending on their determined context, interact with other systems in order to achieve collective behavior. Following the considerations in [BGS01, MZ05], we identify two enabling properties for the self-organization of autonomous embedded systems: (i) contextawareness (e.g. using local sensors for acquiring physical contexts, cf. Section 2.1) and (ii) ad-hoc context sharing (e.g. via an exchange of self-descriptions, cf. Section 2.2). These two properties in combination allow for developing applications whose functionality not only depends on the individual contexts of autonomous embedded systems, but on the co-existence of other such systems and their contextual relations to one another. In this regard, the present thesis covers self-organization insofar as it provides enabling concepts and solutions regarding the awareness of digital artifacts about spatial properties and relations among each other, as well as the exchange of such spatial contexts with other artifacts in range. In the pervasive computing literature, a number of projects can be found which exhibit some forms of self-organization. So as to clarify what we mean by selforganization in the context of this thesis, a few examples – taken from projects which are explained in more detail in Section 2.6 – will be given in the following. In [KOY+ 05] for example, so-called u-Textures (i.e. digital artifacts equipped with displays and direction sensors) can be assembled together, whereas (i) each of them autonomously adapts its behavior according to the current shape of the overall structure and its spatial relation to it, and (ii) the assembled u-Textures as a whole cooperatively run a corresponding application. Another example are Mediacups [BGS01, HB00] (i.e. coffee cups with embedded computing, sensing and communication capabilities) that have been used for a smart doorplate application, in which an interactive display (the doorplate) adapts its content based on both the spatial co-location of Mediacups and their temperature, which are context information that is shared among those digital artifacts and thus enables the emergence of functionality by spontaneous interaction. The TOTA (“Tuples on the Air”) middleware [MZ04, MZ05] is an approach for building distributed data structures through multi-hop propagation of tuples containing both content and propagation rules, by means of which self-organizing applications e.g. for distributed motion coordina-
2.3 Mechanisms of Self-Organization
19
tion can be realized. Further projects which show mechanisms of self-organization can be found in [BIK+ 04, KKP99, LSBP02, ZGMT04], among others.
2.3.1 Self-Organization in Space We consider spatial context information valuable for mechanisms of self-organization among digital artifacts, as they are by nature situated in physical space and thus spatial properties as well as spatial relations between them, which are particularly important due to the fact that interaction is based on local information, are crucial for characterizing an artifact’s situation. From the examples above, it becomes apparent that self-organization is often related to spatial aspects like the position of artifacts or their distance- and orientation-relations to each other, and [ZM04] actually states that a vast majority of known phenomena of self-organization and self-adaptation in nature [...] are actually phenomena of self-organization in space [...]. In [Zam04], it is argued that spatial concepts play an important role in distributed computing scenarios like sensor networks, mobile ad-hoc networks and peer-to-peer (P2P) networks, and that such distributed systems require some forms of self-organization which can be based on a spatial computing model. According to [MZ05], spatial self-organization can be found at different levels in distributed systems: at (i) the physical level which enables spontaneous interaction between system components (e.g. radio broadcasts), (ii) the structure level at which spatial structures emerge (e.g. beacon-based self-localization), (iii) the navigation level which exploits available spatial structures (e.g. for geographical routing) and at (iv) the application level where the underlying navigation mechanisms are used (e.g. pattern formation). As the basis of a spatial computing model, [Zam04] proposes the exploitation of spatial abstractions, meaning that the activities of application components are abstracted as taking place in some sort of abstract metric space, and for which self-organization derives from the autonomous capability of components of sensing, acting in, and navigating that space. Thus, instead of addressing entities by their names, their positions in space are used for interaction purposes. In a similar spirit, we propose the use of spatial abstractions for the implementation of spatially aware applications in this thesis, but with a focus on physical objects and their relations among each other. We therefore distinguish the following two types of spatial abstractions, which are in detail discussed in Chapter 3: (i) quantitative abstractions by means of one or more spatial zones representing an artifact’s position, direction and spatial extension, and (ii) qualitative abstractions of relations
20
2 Spatial Awareness
between such zones representing relationships between artifacts that have a certain meaning for the application.
2.4 Spatial Context in Time The focus of this thesis is on the awareness of digital artifacts about their spatial contexts, which is important information characterizing the situation of artifacts that are by nature distributed throughout physical space. Similar to [CFH97], we classify spatial context information both in terms of the artifacts’ inherent characteristics and with respect to other artifacts, and distinguish two corresponding types of spatial context information: (i) the spatial properties of artifacts, which are independent of other artifacts, and (ii) spatial relations between artifacts – or rather relations between their spatial properties. Spatial properties and relations are valuable context information for a variety of applications, and they constitute a distinguishing feature of this work. Awareness about spatial contexts is a precondition for the autonomous adaptation of digital artifacts to changing spatial environments, and spatial relations among them – as well as relationship changes over time – are particularly relevant with regard to their spontaneous contextual local interaction [HHS+ 02, HKG+ 05, Sch02]. In fact, location information is the most prevalent type of context in context-aware applications (cf. [CK00, SBG99]), and our related work survey of Section 2.6 also shows that location and proximity are the most commonly used spatial contexts. In Table 2.4, which was inspired by [CFH97, CH01], three types of spatial properties of an artifact are distinguished: (i) its position (synonymously used with the term location in this thesis), namely where it is located in physical space, (ii) the direction in which it is aligned and (iii) the artifact’s spatial extension comprising its shape and size. It should be noted that the terms orientation and direction are not used consistently in literature, and sometimes inverse to our notion. Correspondingly, three types of spatial relations between the spatial properties of two artifacts at a time are distinguished: (i) positional relations, which include both orientation and distance relations between the positions of two artifacts, (ii) directional relations, which are relations between two artifacts’ axes of direction, and (iii) topological relations, namely relations concerning the spatial arrangement of extended objects which is defined by their extensions in addition to position and direction. A simple visualization of static properties and relations with a rectangularand a triangular-shaped artifact can be seen in Figure 2.4. The actually used relations depend on application demands, but also on the sensing and processing capabilities of the involved digital artifacts; for example, the consideration of spa-
2.4 Spatial Context in Time
21
Table 2.1: Static characteristics of an artifact’s spatial situation.
Position Direction Extension
Spatial properties
Spatial relations
geographic position intrinsic direction axis shape and size
orientation and distance relations relations between direction axes arrangement of extended objects
Figure 2.4: Static spatial properties and relations of artifacts.
tial extensions – especially when freeform shapes are involved – requires much computational power, which may exceed the capabilities of embedded systems with limited resources. Within the scope of this thesis, spatial context can be given either absolute, namely with respect to a fixed external frame of reference such as the earth reference frame, or relative, namely with respect to the reference frame of an artifact. This distinction between absolute and relative space ranges back to a historical correspondence [LCA98] between Leibniz and Clarke, who wrote Newton’s letters, in the eighteenth century. For Newton, space was an infinite and immovable three-dimensional container with its origin at the center of the universe and which exists whether or not anything is in it [Lev96]. From their correspondence, it is obvious that an absolute space was not a meaningful concept for Leibniz; instead, he considered space as something merely relative, as time is, namely as an order of coexistences, as time is an order of successions [LCA98]. Hence, relative space according to Leibnitz’ view is basically a means for describing relations among objects. We will come back to the distinction between absolute and relative space in Section 3.2.4, where reference frames for qualitative spatial relations are discussed.
22
2 Spatial Awareness
Another distinction of spatial contexts concerns its representation, where we distinguish two types: quantitative and qualitative representation. We use the former for representing spatial properties of artifacts by means of numerical values, and according to certain reference systems in order to ensure a common understanding between artifacts, which is a precondition for the recognition of spatial relations out of them. An example are WGS84 coordinates used by GPS, which represent absolute positions with numerical longitude, latitude and altitude coordinates (cf. Section 3.1). For the representation of spatial relations, qualitative abstractions are used, which have advantages compared to quantitative ones whenever the spatial cognition of humans is involved [Mus00] for example. Qualitative spatial relations are based on discrete symbols such as left or near, representing continuous properties such as orientation or distance in an abstract way which is meaningful for a certain application [CH01]. The six types of spatial context information shown in Table 2.4 are referred to as static spatial context in that they describe an artifact’s spatial situation at a particular point in time. According to [AEG94], the position, direction and extension of an artifact – i.e. its spatial properties – can be changed through translation, rotation and scaling respectively, determining how its spatial relations to other artifacts are changed. The latter are thus referred to as dynamic spatial context, as they describe how its static situation is changing at a certain point in time. We use a discrete time for the sensor observations, which meets the requirements of computing systems with discrete clock cycles [Mus00]. In addition, the spatial context of an artifact need not correspond to a certain point in time, but may also represent an artifact’s situation over a time period by comprising a time series of spatial contexts. Summing up, we distinguish four categories of spatial context information within the scope of this thesis: • Absolute (i.e. with respect to a fixed external frame of reference) vs. relative (i.e. with respect to the reference frame of an artifact) • Quantitative (i.e. using numerical values) vs. qualitative (i.e. using discrete symbols which represent intervals of quantitative values) • Static (i.e. at a certain point in time) vs. dynamic (i.e. the degree of change at a point in time) • Point in time (i.e. a snapshot at a certain time) vs. time series (i.e. a trajectory of spatial contexts over a sequence of points in time) With respect to the above classification, the focus of this thesis is on (i) quantitative spatial properties of artifacts given with respect to an absolute frame of
2.5 An Architecture for Spatial Awareness
23
reference, and on (ii) qualitative spatial relations, which are defined either absolute or relative to another artifact; in Chapter 3, these two types of spatial context will be discussed in detail. While in the case of spatial properties neither qualitative representations (e.g. symbolic locations [Leo98, Sch95]) nor dynamic contexts (e.g. velocity or acceleration with respect to the earth reference frame) are considered in the following, spatial relations are – based on the respective quantitative representations – abstracted in a qualitative way, and both static and dynamic relations between artifacts are taken into account. In contrast to spatial properties, where our scope is on spatial context given at points in time, we also consider time series of spatial relations as will be seen in Section 3.2.5.
2.5 An Architecture for Spatial Awareness The preceding sections elaborated on context-aware digital artifacts, their autonomous operation and self-organization as well as our focus on spatial context information in this regard. Based on these considerations, we have developed a multilayered architecture for providing spatial awareness to digital artifacts, which is an awareness about spatial contexts as discussed in the preceding section. We thus define spatial awareness as follows: A system is spatially aware if it is able to determine and use its spatial context, which comprises an awareness about the system’s spatial properties as well as about spatial relations to other systems. According to the focus of this thesis stated in Section 1.1, our scope is on spatial relationships between digital artifacts. Figure 2.5 shows the simplified architecture for spatial awareness, which has first been published in [HF07] and builds up the basis for the contributions in this thesis. Its comprises layers for (i) the acquisition of spatial context information, (ii) the recognition, representation and reasoning about qualitative spatial relations, and (iii) the provision of services for the development of spatially aware applications. It has a similar structure like the contextaware architecture proposed in [Fer03], except for the explicit distinction between the direct (with sensors) and indirect (via communication) way of acquiring spatial contexts as well as the focus on spatial relationships. We have implemented a middleware framework based on this general architecture, which will be presented in Chapter 6. In the following, the layers of the architecture are discussed in more detail. A precondition for recognizing spatial relations is the mutual awareness of artifacts about their spatial contexts, which requires two architectural components at
24
2 Spatial Awareness
Figure 2.5: General architecture for spatial awareness.
the lowest layer of the architecture: a sensing and a communication module. The sensing module collects sensor readings from the artifact’s spatial sensors (e.g. a GPS receiver and a digital compass for acquiring position and direction with respect to the earth reference frame), and optionally transforms them into a certain reference system such that they are comparable with the spatial contexts of other artifacts. Our focus is on sensors which provide quantitative data (i.e. numerical data, in contrast to e.g. symbolic locations [LM98]) with respect to an external reference system (e.g. the earth reference frame or the working area of a certain indoor tracking system), thus representing an artifact’s spatial properties as defined in Section 2.4. More details about reference systems and the representation of spatial properties with so-called Zones-of-Influence will be given in Chapter 3. Although out of scope, our architecture would also allow to incorporate sensors which directly provide spatial relations, like for example the ultrasonic sensor presented in [HKG+ 05] or by interpreting signal strength measurements as spatial proximity [FHO04a]. The second component contributing to mutual spatial context-awareness is the communication module, which is concerned with exchanging spatial information via self-descriptions with artifacts in communication range. Each artifact collects its own spatial context and gets aware of the others’ contexts through an exchange
2.5 An Architecture for Spatial Awareness
25
of self-descriptions as discussed in Section 2.2.1. It should be noted that an artifact’s spatial context may contain both its spatial properties and spatial relations it is aware of, which is – among other things – a precondition for our approach of inferring and distributing spatial relations presented in Section 4.3. An artifact a for example, which is aware that another artifact b is near, could include this qualitative spatial relation in its self-description. Exchanging spatial relations with self-descriptions thus enables them to become aware of relations they cannot recognize directly, as for example due to missing sensor data or because of the fact that the respective artifacts are out of range. For the above example, an artifact c, which is in a far relation to a and receives its self-description containing the distance relation between a and b, could then conclude that b is not near artifact c (given that the metric distance of far is greater than double the range in which the relation near holds). The core of the architecture is the relationship recognition layer, which is concerned with recognizing and abstracting binary spatial relations with respect to the recognizing artifact (i.e. just relations between the recognizing and one other artifact are considered at a time). Relations are recognized by a pairwise comparison of an artifact’s own with a received self-description, or – more precisely – of the quantitative spatial properties contained therein. The recognition step is performed separately for each received self-description; in this regard, the recognition of relations as well as their qualitative abstraction (i.e. by means of symbolic names like near and far) depends on the availability of sensor data and on the current application demands. Recognized relations are eventually stored in the artifact’s local repository, and provided for the purpose of spatial reasoning or directly for the use by spatial services. A comprehensive discussion on qualitative relations and their use for achieving spatial awareness of digital artifacts will be given in Section 3.2. The spatial reasoning layer is responsible for reasoning about (i.e. inferring conclusions from) the history of relations stored in the repository. In general, known qualitative relations are combined and respective actions are executed if they fulfill certain predefined conditions. We decided for a rule-based approach in order to infer new (high-level) relations or trigger application-level actions upon observing certain patterns on the stored relations. Such a pattern could be the simultaneous existence of a distance relation near and an orientation relation front to the same artifact, meaning for example that it is positioned within the recognizing artifact’s physical interaction range. Detailed information about reasoning mechanisms for qualitative spatial relations can be found in Chapter 4, our approach for rule-based reasoning will be presented in Chapter 5. At the top of the architecture, spatial services are provided to spatially aware applications. Examples are a service for
26
2 Spatial Awareness
querying the relations repository or a communication service for exchanging messages with artifacts to which a certain spatial relation exists.
2.6 Related Work This subsection finally discusses work which is related to the spatial awareness of autonomous embedded systems. According to our observations, the use of spatial relations – and particularly their qualitative abstractions – is a sparsely investigated topic in the field of pervasive computing. Although there has been much work on location systems providing symbolic (i.e. non-numerical) location information (e.g. [LM98, RAMC+ 04, WHFG92]), there are just a few projects dealing with spatial relations at the application level. In this respect, the work closest to ours is the Relate project [HKG+ 05, KKG05], which provides mobile devices with an awareness about both quantitative and qualitative spatial relations to other devices around; however, it differs in how relations are acquired and abstracted, and no rule-based approach for reasoning about them is utilized. The Cooperative Artefacts project [SGKK04, SKGK04] uses rule-based reasoning for inferring knowledge and triggering actions, but does not consider spatial relations other than proximity. Another closely related project is Mediacup [BGS01] with its spatially aware RAUM communication model [HB00] for the interaction based on topological relations. However, neither domain-specific qualitative abstractions of spatial relations nor rule-based reasoning about them by considering qualitative temporal relations seem to have been addressed in other work, and they constitute two of the novelties of this thesis. Another issue of research tackled in our work is the sharing of context information among artifacts which has been used in other projects also (e.g. [BGS01, FHR+ 06a, HMS+ 01, KOY+ 05]), but not – as proposed in this thesis – with structured documents containing both spatial properties and relations an artifact is aware of. Mutual knowledge about the artifacts’ contexts is a precondition for selforganizing behavior, as demonstrated for example in the smart doorplate application in [HB00] or the self-organization of u-Texture panels depending on their spatial configuration in [KOY+ 05]. Within this thesis, we have developed an algorithm for inferring and distributing qualitative spatial relations between artifacts over multiple hops (cf. Section 4.3), which is self-organizing in that they autonomously establish an awareness about spatial relations among each other in an ad-hoc manner. There are related approaches for localization in ad-hoc networks [BSHK06, CHH02], but ours seems to be the only one which relies on purely qualitative spatial relations.
2.6 Related Work
27
The following subsections contain a detailed discussion of related work. In Section 2.6.1, projects which contribute to making autonomous embedded systems spatially aware are surveyed and discussed with respect to their relation to this thesis. As there are many publications on related topics available already, just those which are closely related to the focus of our work are considered. This includes projects dealing with context-aware frameworks and middleware in general, systems and technologies for making spatial information available to applications, and projects which address the acquisition and use of spatial relations between objects. A comparison of selected projects with regard to the contributions of this thesis is given in Section 2.6.2. Related work that is specific to the representation of space – in particular the qualitative representation of spatial relations – as well as reasoning about qualitative spatial relations can be found in the Chapters 3 and 4, respectively.
2.6.1 Projects University of Toronto (US) One of the first projects about spatial awareness beyond pure positional information is presented in [Fit93]. The Chameleon prototype has been developed, a portable spatially aware palmtop computer for visualizing and manipulating electronic information which is associated with physical objects in the real world; it can thus be seen as a window to the three-dimensional information space which is situated in a physical context. The palmtop computer is aware of its position and direction in physical space by the help of a 6-D input device attached to its display, which allows for example to display additional information to a map depending on the palmtop’s position and orientation relative to the map. A graspable user interface for manipulating electronic objects on a horizontal display by means of one or more physical artifacts, which are referred to as bricks, is presented in [FIB95]. As in [Fit93], position and orientation of the bricks are again sensed with a 6-D input device mounted on the table. Context Toolkit The Context Toolkit (http://www.cc.gatech.edu/fce/contexttoolkit), which has been presented in [Dey00, SDA99], facilitates the development of context-aware applications by building upon the widget concept from GUI toolkits. It provides abstractions of low-level context information acquired from sensors according to the application requirements by means of several components: (i) “widgets” which encapsulate sensors and make context available to applications and components, (ii)
28
2 Spatial Awareness
“aggregators” or “servers” which are special widgets that collect context information about certain entities like users from multiple widgets, and (iii) “interpreters” which can be used by widgets and applications to abstract context information into higher-level contexts. The Context Toolkit is based on a distributed architecture which uses XML and HTTP for communication between the components. It is related to the framework presented in this thesis in that it provides context abstractions to applications. In contrast, this thesis specifically addresses the abstraction of spatial relations and the development of spatially aware applications thereof; in this regard, our relation recognizers (cf. Section 6.2.3) are comparable to context widgets which also provide abstract context information and allow applications to register in order to be notified about context changes. Pin&Play and Pushpin Computing In the Pin&Play project (http://eis.comp.lancs.ac.uk/pin&play), which was a joint project of the Lancaster University (UK) and the Victoria Institute (SE), a new ad-hoc networking approach for everyday objects that can be spatially arranged on surfaces has been developed [LSG02]. The surfaces are augmented with layered conductive sheets, in order to serve both as a power supply for the objects as well as a communication medium which is based on a bus-topology. The objects are miniaturized computing platforms that can simply be attached to the surface with pin connectors. The Pin&Play approach thus corresponds to how people are used to attach objects to surfaces and freely arrange them in a spatially meaningful way. A similar infrastructure has been used in the Pushpin Computing project [LSBP02], which however uses the layered sheets for power supply only; Pushpins take advantage of capacitive coupling or infrared to communicate with other Pushpins in spatial proximity. The Pushpin platform is designed to explore principles of self-organization in distributed peer-to-peer sensor networks, by incorporating real-world sensor data instead of simulation. The programming model is inspired by the Paintable Computing work of [But02], and it uses process fragments containing executable code and state information which are executed on Pushpins and may be exchanged among them. Smart Dust The Smart Dust project [KKP99, WLLP01] at the University of California explored the integration of autonomous sensing, processing and communication capabilities in small devices of an intended cubic-millimeter scale, together with a power supply consisting of batteries and solar cells. These so-called Smart Dust
2.6 Related Work
29
motes use the optical medium for communicating with other motes or base stations, either passively using reflectors or actively with a laser diode and steerable mirrors, and they are based on micro-electro-mechanical systems (MEMS) technology for sensing and optical communication. [KKP99] points out the usefulness of self-organizing mote ensembles, which enable more sophisticated behavior by coordinating the individual motes’ actions, by means of an application scenario. Similar to Pushpin Computing [LSBP02], the project aimed at providing a platform for investigating distributed sensor networks in the real world. The TinyOS operating system (http://www.tinyos.net), which is already supported by numerous platforms and sensor boards, emerged from the Smart Dust project and provides an interesting future runtime system for our framework. Cooltown The goal of the Cooltown project [KBM+ 02] at the HP laboratories in Palo Alto was to associate people, places and things with a digital presence on the web, in order to support nomadic users, which are moving from place to place, with a ubiquitous, location-aware system. Location-awareness is provided by locationdependent services, which are based on URLs for addressing information, wireless beacons for broadcasting them in a limited area and client devices for discovering and using them. This approach allows to associate physical objects with links to information and services, and thus bridges the physical with the virtual world. Several applications have been implemented, as for example the Cooltown museum in which infrared transceivers broadcast URLs to additional information about nearby exhibitions, which is displayed on the PDAs of visitors passing by. In the present thesis, we do not focus on location as a context for service provision, but rather exploit spatial relationships between objects for influencing the application-level behavior. TEA The European Technology for Enabling Awareness (TEA) project (http://www. teco.edu/tea) investigated the development of a self-contained module which provides context-awareness as add-on component to mobile devices such as mobile phones and PDAs [GSB02]. The focus of this project was on the use of multiple low-cost sensors for resource-constrained devices, with the aim to acquire context information of their environment and situation which cannot be derived from location, and allow mobile devices to improve their services by adapting to the context of use. TEA uses a layered architecture which separates the context-layer from the
30
2 Spatial Awareness
sensor-layer, and thus provides means for modelling contexts as abstractions of real-world situations and independent of certain sensor technologies. The developed context-awareness module has been evaluated as add-on device for mobile phones, where sensors such as microphones, light-sensors and accelerometers are used to identify situations like in-hand, on-table or in-pocket, and the corresponding user-defined setting are automatically activated. With regard to this thesis, an interesting aspect mentioned in [GSB02] is the sharing of contexts between two mobile phones, by which the caller may be enabled to decide if it is a good time to call for example.
Mediacup The Mediacup project [BGS01] was like TEA concerned with the integration of multiple sensors for context acquisition, but it addressed the issue of how noncomputational everyday physical objects can be augmented with hardware and software for sensing, processing and communicating context information, without limiting their original appearance and functionality [GSB02]. According to the present thesis, the authors of [BGS01] refer to such objects as “digital artifacts”. The Mediacup has been developed, which is an ordinary coffee cup with invisibly integrated sensors for recognizing abstract contexts such as if the cup is stationary or if somebody is drinking out of it. The main research interest was on the emergence of new functionality from a spontaneous exchange of context information among networked artifacts, which are thus equipped with infrared diodes for communicating with transceivers mounted on the ceiling. The spatially aware RAUM communication model [HB00] has been used, which makes interaction based on spatial relations in the physical world instead of identities possible. It allows an artifact to define a geometric space – either stationary or relative to its own position – as a scope for communicating its context, which is only processed by other artifacts that are physically inside that space. This communication concept is demonstrated with a smart doorplate in [HB00], which defines the respective room as context communication space and displays the message that a meeting takes place whenever some hot Mediacups are co-located. A valuable extension of the RAUM model would be to consider not just topological “inside” relations, but to exploit other spatial relations such as distance, orientation and direction between artifacts for the definition of communication scopes, as for example to send messages to artifacts in front or accept messages from similarly oriented artifacts in spatial proximity only.
2.6 Related Work
31
eSeal Like the Mediacup, eSeal [DBK+ 04] is an example of a small digital artifact which autonomously, and without infrastructure support, acquires context information from embedded sensors and abstracts it to domain-specific context. The eSeal system is an electronic seal that can be attached to physical goods in order to monitor their integrity and notify about possible violations as well as the time and location of their occurrence. Different types of integrity are distinguished, including conditional integrity (i.e. the maintenance of given internal properties like temperature and light) and relational integrity (i.e. the maintenance of certain relations such as spatial proximity to other goods). Our work contributes to the latter by providing means for recognizing application-specific spatial relations and triggering certain actions when they are no longer fulfilled.
Smart-Its The project Smart-Its (http://www.smart-its.org) has been inspired by the TEA and Mediacup project, but changed the research focus from individual context-aware devices to ad-hoc networking and context sharing [GSB02]. Smart-Its are smallscale, embedded devices with sensing, processing and wireless communication capabilities, which are not intended as add-on components for mobile devices, but as flexible autonomous devices for their own. They consist of two modules, one for sensors and perception and a second one for application-specific processing and communication. With Smart-Its Friends [HMS+ 01], a demonstrator for evaluating the value of context sharing has been built. The underlying idea is to make connections between artifacts based on proximity-relations between their contexts; artifacts are considered to be in context-proximity if they experience similar situations. In this application, Smart-Its devices broadcast their movement pattern acquired from accelerometer data, and receiving devices compare it with their own movement and establish a connection at the application level if they are similar (i.e. if they have been manipulated the same way). Spatial proximity is thus not acquired by a direct relation of positions, but rather by a comparison of movements. Smart-Its Friends is similar to this thesis in that context information is exchanged between artifacts and used for processing qualitative relationships; however, just similarities between contexts – and thus only a small subset of the possible relations – are considered.
32
2 Spatial Awareness
Aura Aim of the Aura project (http://www.cs.cmu.edu/aura) at the Carnegie Mellon University (US) was the creation of distraction-free pervasive computing environments which adapt to the users’ contexts and needs [GSSS02, SG02]. Central are the support of user mobility as well as a transparency of dynamic resource variability by automatically adapting the environment or ongoing tasks to the user’s context. Location information – which is acquired from IEEE 802.11 access points – is a key context in this regard, as for example in the “Portable Help Desk” [GSSS02] application which provides users with an awareness about nearby people and resources. A hybrid location model [JS02] has been developed within the Aura project, which uses a combination of hierarchical and geometric location representations by allowing each space in this hierarchy to define geometric points or areas. A space service based on this model has been implemented, answering SQL queries about distance, containment and intersection relationships, among others.
Easy Living The project Easy Living at Microsoft Research aimed at the development of a prototypical architecture and technologies for intelligent environments in which stationary and mobile devices cooperate to provide user access to information and services [BMK+ 00]. The geometry of the world plays a central role in this project, as knowledge about physical relationships between people, devices, places and things increases the shared understanding between user and system and thus contributes to a more natural and implicit interaction (e.g. the selection of a display which is in the user’s field of view) [BKMS00]. A geometric model has been developed which represents the measured quantitative position and direction of one entity’s reference frame with respect to that of another entity, and optionally represents their physical extensions with polygons. In this model, reference frames of the entities are vertices and geometric measurements between them are edges in an undirected graph, which allows to process relationships between entities – including the intersection of regions (e.g. to select all devices within a particular radius around the user) – which are connected with a path in the graph. The project focus concerning the perception area is on stereo computer vision for distinguishing and localizing people [KHM+ 00]. Devices are networked without a central server using asynchronous message passing. Although the geometric model explicitly represents spatial relationships between extended entities, it is maintained on a central server and not distributed among the entities as in our work. Moreover, no
2.6 Related Work
33
qualitative representation of and reasoning about qualitative spatial relations are addressed. ParcTab The ParcTab system (http://sandbox.xerox.com/parctab) was developed at Xerox PARC for investigating mobile and context-aware computing in office settings, and it pioneered the use of location information for a variety of context-aware applications. ParcTabs are small hand-held devices with a touch-sensitive screen, which are able to communicate with each other and with desktop computers using an infrared (IR)-based cellular network [WSA+ 95]. A room provided with an IR transceiver becomes a cell in this network, which also allows for sensing locations at the granularity of rooms. Infrared transmissions are used for determining location the same way as in the Active Badge system [WHFG92], which is one of the first indoor location sensing systems. It provides symbolic location information about people who wear badges that are equipped with a microprocessor and an IR transmitter. The badges periodically emit unique user-identifiers, which are collected by IR receivers placed in the environment and sent to a central location server. The proximity of ParcTabs as a context for interaction was explored in several application scenarios [SAW94], e.g. in the form of “proximate selection” which is a user interface technique for emphasizing objects that are located nearby. The ParcTab system and applications are related to this thesis in that location and proximity information are used for building context-aware applications. Place Lab Place Lab (http://www.placelab.org) is an open-source software solution developed at Intel Research, which works both indoors and outdoors and allows clients like notebooks, PDAs or mobile phones to estimate their own positions by listening for the identifiers of stationary radio beacons like IEEE 802.11 access points or GSM cell towers and looking up the beacons’ positions in a database [LCC+ 05, SLB+ 03]. According to [LCC+ 05], clients have access to the positions of more than 2 million beacons worldwide (primarily IEEE 802.11 access points) and localize themselves with an accuracy of about 20 (802.11) to 100 (GSM) meters within an urban area. Sentient Computing The idea behind the Sentient Computing project is to give applications the ability to adapt their behavior based on a model of the environment, which allows them
34
2 Spatial Awareness
to act as if they shared the user’s perception [ACH+ 01]. This world model is implemented with a set of software objects containing up-to-date information about the states and locations of real-world objects. To achieve a level of accuracy close to human’s perception, a building-wide ultrasonic location system has been developed; a prototype of this system has been presented in [HHS+ 02]. Each user wears a mobile device (i.e. referred to as bat), which emits ultrasonic pulses to a sensor network mounted on the ceiling and can be localized with an accuracy of up to 3cm. The system is also able to estimate a user’s direction, assuming that the bat is mounted on the body’s front side. Similar to the Zones-of-Influence approach presented in Section 3.1.1, spatial relations between objects are processed in terms of geometric containment and overlaps relationships between 2-D spaces associated with those objects. They have been used for “follow-me” applications, for example to display a user’s desktop on the nearest screen by defining two spatial zones, one from where the machine can be used and another one from where the screen can be seen [HHS+ 02]. In contrast to this thesis, neither relations beyond topology nor spatial reasoning have been considered. NearMe The NearMe Wireless Proximtiy Server [KH04], which has been developed at Microsoft Research, allows to discover people and things in physical proximity using Wi-Fi. Clients register with the server, scan for Wi-Fi access points and report observed access points and their signal strengths to the server, who compares the data with that of other clients. The server can then send back a list of nearby users and things that have been registered. NearMe provides no absolute location but only proximity information with a resolution in the tens of meters; an a-priori calibration of the environment is not required. Implemented applications include the discovery of resources in proximity and sending e-mails to people nearby. NearMe inspired recent work [VPKdL08] for calibration-free localization, which uses relative distances between pairs of mobile phones that are acquired from the measurement of signals from observed GSM cell towers. Imperial College of Science (UK) In [Leo98, LM98], a location service is proposed which collects data from multiple sensors, abstracts the sensor readings to a uniform representation and performs fusion by exploiting overlaps between locations – which are represented in a semi-symbolic (i.e. containing both symbolic and geometric representations) hierarchical model – in order to improve accuracy. Spatial containment, overlaps
2.6 Related Work
35
and equivalence relations can be acquired from the location model. The location service thus provides an abstraction from location sensor details and allows for developing location-aware applications independently of certain sensor technologies, which corresponds to the application-level provision of abstract spatial relations in this thesis.
Location Stack The Location Stack [HBB02] is an abstract layered software engineering model for the use of location in ubiquitous computing applications, which is inspired by the seven-layer OSI model for networking. Similar to [Leo98], it is designed to support multiple sensor technologies and multiple applications with generic abstraction layers in-between. These include a fusion layer for merging readings from multiple location technologies, and an arrangements layer for reasoning about relationships between objects. The Location Stack has been applied to the Labscape project [AGH+ 02] and the Easy Living system [BMK+ 00] in order to illustrate the value of separating concerns with a layered architecture.
GISS At the Johannes Kepler University Linz (AT) we have developed a Group Interaction Support System (GISS) [FHO04a, FHO04b] for the support of interaction within groups in mobile distributed computing environments, for example by visualizing the locations of group-members in an instant messenger or by automatically forming a temporary group with people staying at the same location. The implemented system uses the context-framework presented in [BCFM03], which is based on a peer-to-peer communication architecture and uses Event-ConditionAction (ECA) rules for defining the behavior of the system. A central part of GISS is its location sensing subsystem, which acquires location and proximity information from multiple different sensors, abstracts the spatial sensor readings to standardized symbolic values, performs fusion algorithms on them to increase precision and accuracy, and provides the spatial information in a sensor-independent way at the application level. In contrast to this thesis, the abstraction of proximity relations depends on the used sensors and cannot be adapted to application demands, just proximity relations are considered and no reasoning mechanisms are included in GISS.
36
2 Spatial Awareness
Peer-It In recent work also conducted at the Johannes Kepler University Linz (AT), a software/hardware platform called Peer-it [FHdSR+ 07, FHR+ 08] has been developed which comprises sensors, actuators and wireless communication facilities on the hardware level as well as a component-based framework at the software level. Peer-its are intended to be used as stick-on computer solutions for physical objects such as machines or appliances, thus enabling their autonomous operation and spontaneous interaction with other peers. Interaction among peers is based on their discovery once coming into proximity, an exchange of XML-based “profiles” representing their properties and interests as well as a similarity analysis of such profiles [FHR+ 06a]. It could thus serve as an implementation of the Digital Artifact Service of our Zone-of-Influence Framework described in Section 6.2, which is responsible for discovering artifacts and exchanging self-descriptions – which correspond to the “profiles” in [FHR+ 06a] – with them. The Peer-it system emerged from previous work on spontaneous interaction [FHM+ 04], in which autonomous artifacts exchange and compare their profiles once they come into spatial proximity. Also the concept of Zones-of-Influence, which constitutes the basis for recognizing relations in the present thesis (cf. Section 3.1.1), has originally been introduced in [FHR+ 08] for limiting interaction to peers whose Zones-of-Influence overlap; however, just topological relations and no reasoning mechanisms for inferring conclusions from spatial relations are considered. u-Texture The u-Texture project [KOY+ 05] at the Keio University (JP) addresses the issue of creating smart environments in an intuitive way by physically assembling smart building blocks with embedded computing, sensing and networking capabilities into shapes like tables or shelves. A u-Texture prototype including a touch screen, serial interfaces at the four sides to detect a connection with other u-Textures, a 3D direction sensor as well as both wired and wireless network interfaces has been developed. For assembling u-Textures horizontally or vertically to one another, a connection structure called u-Joint is used in this prototype. Assembled u-Textures have self-organizing capabilities by autonomously adapting their behavior to the current shape and cooperatively running a corresponding application. By exchanging information on their connections and spatial directions with XML-based documents similar to our self-descriptions, each one of the u-Textures recognizes the assembled shape as well as its location and direction in it. One of the realized applications is the “Collaboration Table” which allows for exchanging drawings
2.6 Related Work
37
among connected u-Textures by means of simple drag-and-drop operations, similar to the “Gateways” concept [GSG07] of the Relate project. Relate The Relate project (http://eis.comp.lancs.ac.uk/relate) was funded by the European Commission in the Sixth Framework Programme (FP6) and coordinated by the Lancaster University (UK). Its aim was to investigate relative positioning of mobile objects without a surrounding infrastructure, including research issues such as the measurement of spatial relationships and software platforms for relative positioning systems. A hardware/software system is presented in [HKG+ 05], which is based on an ultrasonic peer-to-peer sensing hardware for distance and orientation measurements. It allows mobile devices to independently establish and maintain a model of fine-grained quantitative and qualitative spatial relationships to other devices in two dimensions. The Relate system architecture consists of several layers, including a processing pipeline for computing quantitative relations, performing abstractions to qualitative relations and inferring information over time. A spatial model maintains a list of directed labelled graphs where each one represents the spatial arrangement of devices at a certain point in time. As an extension of the Relate system, a set of “spatialized widgets” – i.e. visual interaction elements whose appearance and behavior adapts depending on the spatial situation – for incorporating spatial relationships in the user interface is proposed in [KKG05]. [BSHK06] uses the Relate system for constraint-based distance estimation, as well as a graph in which the nodes represent mobile devices and the edges are labelled with distance constraints (i.e. intervals for modelling uncertainty). With transitive inference steps (cf. Section 4.1), more specific distance constraints between devices are inferred. Further work within the Relate project deals with securing wireless connections by spatial relationships [May06] and visualizing “Gateways” to discovered services in a GUI according to their spatial orientation [GSG07]. The architecture of the Relate system is similar to ours insofar as it allows mobile devices to recognize and abstract spatial relations, infer new relations from existing ones and provide applications with access to the spatial model. Within this thesis however, spatial relationships are not directly recognized with sensors like the one presented in [HKG+ 05], but on the basis of explicitly defined spatial context information exchanged among artifacts. A main focus of our work is on the qualitative abstraction of relations, which has not been considered for spatial aspects other than position in the Relate project, and qualitative reasoning using first-order-logic is part of the Relate architecture presented in [HKG+ 05] but not addressed in more detail.
38
2 Spatial Awareness
DOLPHIN DOLPHIN [MFH+ 04] is a distributed 3-D indoor positioning system that uses devices for bidirectional ultrasonic ranging similar to the ones used for the Relate system presented above. The aim of the project is to reduce the high initial configuration costs of the system, which are inherent to other ultrasonic positioning systems [HHS+ 02, PMBT01] in that they require a huge number of precisely positioned references. DOLPHIN tries to overcome this problem by using iterative multilateration, which is based on the idea that sensor nodes estimate their position using a small number of a-priori configured reference nodes, and hence become references by themselves. In contrast to existing ultrasonic positioning systems, just the location of objects and not their direction in space is detected. Cooperative Artifacts In [SGKK04, SKGK04], the instrumentation of everyday physical objects with embedded computing, wireless communication, sensors and actuators is described, which allows them to autonomously perceive and cooperatively assess their situation without the need for an instrumented environment; such instrumented objects are referred to as Cooperative Artifacts. A platform-independent architecture is depicted, which consists of (i) a sensor component for acquiring context information of the real world, (ii) a perception component for associating sensor data with meaning, (iii) a knowledge base containing domain knowledge and observational or inferred knowledge about its situation, (iv) an inference component for reasoning about its own knowledge and those provided by other artifacts, and (v) an actuator layer for effecting the real world. Rule-based reasoning is used for both inferring further based on existing knowledge and for determining the behavior depending on the environmental context, and artifacts cooperate by sharing their knowledge using some kind of query/reply protocol. The concept has been demonstrated with chemical containers [SGKK04], which are augmented with Cooperative Artifacts and are thus able to detect hazardous situations, as for example that incompatible chemicals are stored too close together. The ultrasonic range sensors developed in the Relate project presented above detect distance relations between containers, and infrared light sensors are used for recognizing if they are stored inside or outside an approved area. The authors of [SGKK04] consider their approach novel with regard to embedding generic reasoning capabilities into otherwise non-computational artifacts. The Cooperative Artifacts work overlaps with ours regarding the cooperation between autonomous artifacts by sharing knowledge, the inference of new knowledge and triggering of actions with rule-based
2.6 Related Work
39
reasoning as well as the use of spatial distance relations in [SGKK04] for detecting hazardous situations. The focus of this thesis however is on the spatial awareness of artifacts beyond location and distance estimations, including the abstraction of spatial relations and qualitative reasoning about them over time, which is not covered in [SGKK04]. Ubiquitous Chip The Ubiquitous Chip [TTH+ 04] is a flexible and autonomous prototype hardware/ software platform which can be connected with devices such as sensors, actuators and other ubiquitous chips, and allows for describing its behavior by means of Event-Condition-Action (ECA) rules. Inputs from sensors and packets received via the serial port are evaluated with respect to the rules’ conditions, and the actions of a rule – as for example to control a connected actuator – are executed when all its conditions are fulfilled. Although this rule-based approach is similar to ours in that the system functionality can be programmed easier and changed dynamically at runtime, the rule conditions are formulated on low-level sensor data and not on high-level contexts as in our work. MiddleWhere A distributed middleware called “MiddleWhere” for location-awareness is presented in [RAMC+ 04], which is based on a hierarchically organized hybrid location model using both geometric and symbolic representations, and enables probabilistic fusion of location data from different sensor technologies. A spatial database is used for representing and reasoning about location information and defining triggers that generate events when a certain location-based condition is fulfilled. Among other things, the middleware is able to infer spatial relationships, where three types are distinguished: (i) topological RCC-8 relations [Ege89] and additional relations for representing connectedness between regions, (ii) containment, Euclidean distance and usage (i.e. if a person is within the usage area of an object, similar to [ACH+ 01]) relations between an object and a region, as well as (iii) proximity, co-location and distance relations between two objects. TOTA “Tuples on the Air” (TOTA) [MZ03, MZ04] is a middleware for developing distributed context-aware and self-organizing applications. It is based on spatially distributed tuples, which are injected in the network from artifacts running the TOTA
40
2 Spatial Awareness
middleware, autonomously propagate hop-by-hop according to tuple-specific patterns and can be queried by artifacts within their one-hop neighborhood. The propagation pattern of a tuple is defined by its spatial scope (i.e. distance and spatial direction of propagation), the presence or absence of other tuples and how its content has to be changed during propagation. TOTA tuples can thus be used for creating and exploiting distributed spatial structures of the network, including application examples such as routing in mobile ad-hoc networks, motion coordination and gathering of context information [MZ05, MZL03]. As an example for the latter, an artifact may broadcast query-tuples with the information it is looking for; these tuples create gradients to their source by incrementing a counter at every hop, and receiving artifacts which possess the requested information can inject answer-tuples following the path in the direction where the query-tuples’ gradients are decreasing. With respect to our work, the TOTA middleware could be used for implementing our spatial relationship distribution algorithm of Section 4.3. Gator Tech Smart House The Gator Tech Smart House [HMEZ+ 05] is a laboratory for investigating pervasive computing systems with the aim to create assistive environments specifically for the elderly and disabled people. A multi-layered OSGi-based middleware and numerous applications based on it have been realized, as for example a smart floor for tracking the locations of people as well as ongoing work on a smart projector which uses an ultrasonic tracking system for detecting where people are facing in order to decide on which wall to display the contents. In the spirit of our approach for developing applications based on qualitative spatial relations, the Gator Tech Smart House allows domain-experts to model application behavior by associating actions with abstract high-level contexts. Rutgers University (US) A spatial programming model similar to the TOTA approach [MZ04] is presented in [BIK+ 04], which allows for implementing distributed applications in a networktransparent way. It is based on “smart messages” containing data and code, which migrate to nodes of interest in the network using geographical routing algorithms and are executed there. The core idea is to identify spatial regions which are relevant for a certain application (either static or dynamically defined by a spatial proximity or orientation relation with respect to a certain node), and access network resources through spatial references by specifying their expected spatial regions and properties.
2.6 Related Work
41
Spatial Cognition: Reasoning, Action, Interaction Spatial Cognition: Reasoning, Action, Interaction is an interdisciplinary research center at the Universities of Bremen and Freiburg, which covers various aspects of knowledge about spatial environments (http://www.sfbtr8.uni-bremen.de). It comprises tens of research projects and spans a huge number of topics, ranging from the investigation of human spatial cognition to mobile robot navigation. One of the projects related to this thesis is “Q-Shape”, which deals with reasoning about paths, shapes and configurations. Several qualitative spatial calculi [MDF05, MRW00] have been developed in this project, which are defined by a set of relations between objects from a certain domain and operations on these relations (cf. Section 4). Of particular interest is the SparQ toolbox [DFWW06, WFDW07] which makes existing calculi available for integration in own applications, and it can be run in server mode and accessed via a TCP/IP connection. It provides services for the qualification of quantitative geometric descriptions, performing computations with relations based on the operations defined in a certain spatial calculus and constraint-based reasoning for checking consistency or determining if particular relations between certain objects hold, among other things. Another interesting project is “OntoSpace” which deals with the linguistic aspects of spatial terms [Ten05] and investigates human-robot interaction through dialogue [TM03], among others.
2.6.2 Comparison Table 2.2 compares selected projects of Section 2.6.1 by means of a set of aspects which concern software frameworks and architectures using spatial context information, and in particular spatial relations, at the application level. First, the abstraction of sensor technology – namely a separation from low-level sensor details which are not relevant at the application level – is an important property of context-aware systems (cf. Section 2.1), and local context exchange refers to their ability to share perceived context information in a peer-to-peer manner with other systems in communication range. In this thesis, we distinguish three types of spatial properties describing an artifact’s state, which are its position, direction and physical extension, as well as four types of spatial relations between two artifacts at a time, which are positional (distance and orientation), directional and topological relations. They can be recognized from a single or multiple sensor technologies, and represented quantitatively and/or with qualitative symbols. In this regard, adaptivity describes the ability of a framework to dynamically adapt the consideration of certain qualitative relations as well as their semantics to the
42
2 Spatial Awareness
application demands. The maintenance of a relations history means that relations are stored together with points in time or with time intervals in which they have been recognized or exist, which is a precondition for reasoning about relations over time. Qualitative reasoning is a central issue in this thesis, where our focus is on rule-based reasoning about qualitative spatial relations and about temporal relations between the spatial ones (i.e. spatiotemporal relations). Several observations can be made from Table 2.2. First, in most projects sensor data is abstracted for their use at the application level, which can thus be considered state-of-the-art and also plays an important role in our work with respect to relationship abstractions. Second, most approaches for spatial awareness exploit the position of artifacts, which is actually the most often used context in existing context-aware applications [CK00] and is usually the basis for calculating positional orientation and distance relations. However, proximity can be determined by other means also, as for example in the Smart-Its Friends application [HMS+ 01] where two devices are considered to be in proximity if their movement-patterns are similar. Third, although context sharing is used in several projects for cooperation between autonomously operating artifacts, none of them uses structured XMLbased self-descriptions for the representation of spatial properties and relations. Fourth, directional relations – as for example that two objects are aligned in the same direction – are hardly used in the field of pervasive computing, which may be due to sensing difficulties. Fifth, qualitative representations of spatial relations are utilized by almost all projects compared in Table 2.2; in most cases however, just proximity or containment relations are used. The contributions of this thesis, which distinguish it from the related work discussed above, include (i) the adaptive recognition and qualitative abstraction of relations from the explicitly defined spatial properties position, direction and physical extension, (ii) a standardized representation for the exchange of spatial contexts between autonomous entities and (iii) the proposed approach for rule-based reasoning about qualitative spatial and temporal relations. This together facilitates the development of spatially aware applications in that it allows for simply specifying qualitative abstractions of spatial relations which are relevant for the application, and defining actions to be executed once a certain pattern of spatial relations over time has been recognized.
2.6 Related Work
43
a determined
xa x
x
x x x
x x
x x
x
x x x
x
x x
x x
x
x x
x
x
x x
x
x x x
xa x
x x
x
x
x
x
xe
x
xe
x
x x x
x
x x
by the sensor range which defines a geographical area ad-hoc exchange of structured profiles with discovered peers in range c via a wired backbone infrastructure d supported by the framework, but not readily available at the time of this writing e implicitly by initiating a connection upon recognizing the respective qualitative relation f given by tuple gradients of neighboring nodes g depends on the plugged-in relation recognizers b wireless
x xd
x
x x x
x xf
x x x x
x
x x
x x x
x
x
x x
x
ZoI Framework
x x
x xc
TOTA
x x x
x xb
MiddleWhere
Sentient Computing
Relate
Easy Living
x
u-Texture
x
x x
Smart-Its Friends
x
x
Mediacup
x x
Peer-it
Software/architecture sensor abstraction local context exchange Spatial properties position direction extension Spatial relations positional (distance) positional (orientation) directional topological Relations recognition multiple sensors quantitative represent. qualitative represent. runtime adaptivity relations history Qualitative reasoning rule-based inference spatial relations spatiotemp. relations
ParcTab
Cooperative Artifacts
Table 2.2: Comparison of related work.
x
x xg xg x x x x x
3 Representation of Space Simple things should be simple. Complex things should be possible. Alan Kay, 1940-
In order to deal with space computationally, it has to be represented in a standardized way such that it can be processed by a computer. In this regard, we distinguish between the quantitative and qualitative representation of space, which are discussed in detail in the Sections 3.1 and 3.2, respectively. While the former deals with concrete facts about the spatial properties of an artifact using numerical values (as provided by sensors for example), the latter is concerned with symbolic and human-readable abstractions of spatial relations (i.e. symbols are used for the representation rather than numeric values) which are acquired by a pairwise comparison of quantitative spatial properties. For the representation of an artifact’s spatial properties we propose to use so-called Zones-of-Influence, which are explicitly defined geographical regions that are relevant for the application. Our focus is on qualitative spatial relationships as well as their changes over time and the application-dependent semantics. We consider qualitative relationship abstractions to be particularly valuable for implementing services that are distributed among multiple artifacts in physical space. Quantitative spatial properties and qualitative spatial relations – which together constitute an artifact’s spatial context – are exchanged between artifacts in range by means of XML-based self-descriptions, which enables them to reason about both self-determined and received relations; an example structure of self-descriptions, which we used for evaluation purposes in this thesis, is presented in Section 3.3. Section 3.4 eventually sums up findings concerning the quantitative and qualitative representation of spatial properties and relations with regard to their representation and exchange by autonomous embedded systems.
46
3 Representation of Space
3.1 Quantitative Representation 3.1.1 Spatial Abstraction with Zones-of-Influence For providing spatial awareness to digital artifacts, we propose a concept referred to as Zones-of-Influence (ZoI). It builds on initial work published in [FHR+ 08], where ZoIs are three-dimensional spatial regions associated with digital artifacts; in this work, they serve as an explicit proximity model in that they are used for limiting the interaction between artifacts to those whose ZoI-geometries overlap. In contrast to proximity sensors like Bluetooth signal-strength measurements of nearby artifacts as used in [FHM+ 04, FHO04a] for example, a Zone-of-Influence provides added value by allowing to explicitly define the “proximity range” by its shape, the size of the shape as well as its position and direction in space relative to that of the artifact for which it is defined. A similar concept has also been used in [BMK+ 00, HB00, HHS+ 02], where geometric containment relations between regions of interest that are associated with physical objects (e.g. a person and the area in front of a screen) are used for location-aware applications (e.g. to display the person’s desktop on that screen). In [FHR+ 08], ZoIs are specified with XMLbased documents similar to the self-descriptions used in this thesis and exchanged among artifacts, which enables them to decide on the initiation of an interaction with other artifacts in communication range based on a comparison of their own with received Zones-of-Influence. Applications for this Zones-of-Influence concept are manifold, as it becomes useful whenever digital artifacts, namely technology-enriched objects which are situated in physical space, have to interact with one another and their spatial context has an influence in this regard. For example, ZoIs can be used for modelling dangerous zones of a machine (e.g. its area of operation in which goods are processed) or the physical extension of a parking place, in order to detect if workers are in a dangerous situation (i.e. within the machine’s operational area) or if the parking place is occupied (i.e. a car is within it), respectively [FHR+ 08]. Another automotive example taken from [FHR+ 08] is the representation of the car’s braking distance by means of a dynamically changing ZoI, which is a simple bounding box around the car whose length in the car’s front corresponds to its velocity and thus to the respective braking distance; once an overlap between the ZoIs is recognized, the involved drivers are warned that the space between them is too small (cf. Figure 3.1). The work conducted within this thesis builds on the above Zones-of-Influence concept as the basic building block for determining spatial relations between digital artifacts. In contrast to [FHR+ 08] however, Zones-of-Influence are not utilized
3.1 Quantitative Representation
47
Figure 3.1: Zone-of-Influence for a car’s braking distance [FHR+ 08].
for the mere purpose of initializing the interaction among artifacts whose ZoIs overlap; instead, different types of static and dynamic spatial relations are recognized and provided for the use by arbitrary spatially aware applications. We define a Zone-of-Influence as an explicitly defined spatial region, which is given by its extension as well as its position and direction in space, and represents spatial properties of an artifact that are relevant for its application-level behavior with respect to the spatial properties of other artifacts. Hence, Zones-of-Influence are spatial two- or three-dimensional shapes of a certain size which are positioned and directed in physical space, and they encode spatial properties of artifacts that are relevant for and have an influence at those applications in which the artifacts are involved. In the following, we use the term shape synonymously for the spatial extension of a ZoI, namely including also its size. Special types of Zones-of-Influence are physical ZoIs representing an artifact’s natural physical position, direction and extension, as well point-shaped ZoIs, which are point abstractions for zones whose spatial extension is not relevant for the application or cannot be considered due to limited computational power. As stated in Section 2.4, our focus is on the quantitative (i.e. numerical) representation of spatial properties with respect to an absolute (i.e. fixed, external) frame of reference. In particular, WGS84 coordinates for representing absolute position and Euler angles for absolute direction can be used, which are explained in Section 3.1.2. Position and direction of a Zone-of-Influence are related to a single point, to which we refer to as the ZoI’s anchor point; they are visualized as black dots in Figure 3.1. Moreover, just static spatial properties are used for representing a Zone-of-Influence, meaning that dynamic contexts which characterize the degree of change at a point in time (e.g. acceleration) are not considered in this thesis –
48
3 Representation of Space
Figure 3.2: Types of Zones-of-Influence in 2-D.
although in principle supported by the Zones-of-Influence Framework presented in Chapter 6. Each digital artifact is associated with a single or multiple Zones-of-Influence at a time, together representing the spatial application-relevant knowledge which is distributed across them in physical space. A ZoI can be dynamic, which means that its spatial properties depend on sensor values (e.g. if its position and direction depends on that of a digital artifact), or static for ZoIs whose position, direction and shape do not change over time. Moreover, a single Zone-of-Influence can be fragmented, namely split in multiple – relatively positioned and directed – shapes, which is meaningful for representing ZoIs with complex or disconnected spatial extensions. The following list sums up these four types of ZoIs, which have been distinguished in [FHR+ 08] already and are clarified by means of two-dimensional examples in Figure 3.2: • Static: does not change over time (i.e. no dependence on sensor values) • Dynamic: one or more spatial properties change over time • Multiple: more than one ZoI is assigned to the same artifact • Fragmented: a single ZoI is split in multiple shapes Due to the exchange of Zones-of-Influence by means of self-descriptions (cf. Section 3.3), they can be discovered by artifacts in communication range, which are thus able to determine spatial relationships between their own and received Zones-of-Influence – including positional, directional and topological relations. Therewith, application-level behavior of artifacts can not only be determined by
3.1 Quantitative Representation
49
proximity relations to other artifacts as in most projects compared in Section 2.6, but rather by arbitrary spatial relations among them that are acquired by a pairwise comparison of Zones-of-Influence, which constitutes the core of this thesis.
3.1.2 Reference Systems for Position and Direction The spatial properties of a Zone-of-Influence are either static or dynamic in that they depend on sensor values which change over time. Two spatial properties of a digital artifact, which are acquired from respective sensors, are considered in this work: position and direction. The shape of a ZoI is assumed to be statically defined and independent of any sensor values for now. We consider positionand direction-sensors that provide quantitative readings given for two- or threedimensional space, and with respect to an absolute reference system. Knowledge about the used reference system is required for a common understanding of artifacts about exchanged – static or dynamic – Zones-of-Influence, which in turn is a precondition for recognizing spatial relations by their pairwise comparison. In the case that digital artifacts have to deal with different reference systems at the same time, e.g. if two location systems – one for outdoor and another one for indoor – are used together, there must be means for transforming to a common reference system (cf. [JS02]). In the following, reference systems for position and direction are surveyed in short. For the property position, numerous sensing techniques and technologies with different reference systems have been developed (cf. [HB01]). An indoor location system may provide three-dimensional Cartesian coordinates relative to a certain point in the building for example, and a GPS receiver provides spherical polar coordinates in a global reference system; a comprehensive overview of coordinate systems for georeferencing can be found in [Dan99]. The most common global system, which is also used by GPS, is the World Geodetic System 1984 (WGS84) [IoD00]. In this system, points are typically represented as spherical polar coordinates in latitude ϕ (i.e. the angle between any point on the surface of the earth and the plane of the equator), longitude λ (i.e. the angle between the prime meridian and a plane through a point on the earth’s surface, where both planes are perpendicular to the equator) and altitude h (i.e. the height of a point’s location above a reference ellipsoid which approximates the earth’s surface). Also, they can be represented as Cartesian coordinates (x,y,z) using Earth-Centered (i.e. the origin is the mass center of the earth) and Earth-Fixed (i.e. coordinates of a point on the earth’s surface do not change with its rotation) coordinates. In such a Cartesian ECEF system, positional relations between two points on the surface of the earth require special treatment; for example, the shortest Euclidean distance be-
50
3 Representation of Space
Figure 3.3: WGS84 spherical (ϕ,λ ,h) and Cartesian (x,y,z) coordinates.
tween them differs from the actual distance on the surface. A visualization of the WGS84 spherical and Cartesian coordinates can be seen in Figure 3.3. There are also different reference systems for the spatial property direction. As mentioned in Section 2.4, we do not use the term orientation in order not to confuse the spatial property direction with spatial orientation relations. A common direction system is the North-East-Down (NED) system defined by the x-axis pointing to the north, the y-axis pointing to the east and the z-axis heading toward the center of the earth. One way of representing directions in three-dimensional Euclidean space is the use of Euler angles, which represent spatial directions with a sequence of rotations around the three axes by the following values: yaw (i.e. a rotation around the z-axis), pitch (i.e. a rotation around the y-axis) and roll (i.e. a rotation around the x-axis). An important point is that the order of rotations matters, and different sequences lead to different resulting directions. For evaluation purposes, the IS-900 6 degrees of freedom (DOF) motion tracking system from Intersense Inc. [Int08] has been used within this thesis (cf. Chapters 5-7). It has a local reference system with the z-axis pointing to the earth’s center and the x- and y-axis pointing in two orthogonal horizontal directions according to the boundaries of the room in which the tracking system is installed. Direction is provided with Euler angles, where the sequence starts with a rotation about yaw (around the z-axis), followed by pitch (around the y-axis) and roll (around the x-axis). In this regard, we distinguish between two-dimensional (2-D)
3.2 Qualitative Representation
51
representations, as well as three-dimensional (3-D) ones where the z-axis – and thus both pitch and roll – are disregarded.
3.2 Qualitative Representation Zones-of-Influence as discussed in the previous section provide means for representing spatial properties of digital artifacts, and they are shared among them using XML-based self-descriptions. As stated before already, our main interest is on spatial relations acquired from a comparison of Zones-of-Influence as well as their qualitative abstraction for the application-level use; however, until now we have not explained what qualitative exactly means and why our focus is on qualitative spatial relations. These points will be clarified in Section 3.2.1, and more details about how spatial relations can be represented in a qualitative way as well as the importance of reference frames in this regard will be given in the subsequent sections. Section 3.2.5 afterwards introduces a combined consideration of qualitative spatial and temporal relations, which allows for qualitative reasoning about spatial relations over time.
3.2.1 Qualitative Abstractions of Space In order to process spatial relations computationally, the mathematics of Euclidean space probably comes to mind first, which is actually the most precise system for spatial representation [HN02]. For example, if we consider the two artifacts shown in Figure 2.4, it could be determined that artifact B is located +93,2° with regard to the direction axis of A, and in a distance of e.g. 3, 52m. Although these quantitative orientation and distance relations are correct and precise, the qualitative information that B is right and near could be sufficiently precise and more appropriate for a particular application – especially if the spatial cognition of humans is involved [CH01, Mus00]. It can be seen from these examples that qualitative abstraction is about naming quantitative values, which means to represent continuous properties of the world by discrete systems of symbols [CH01]. According to [CH01, HN02], the actual quantization, namely the mapping of continuous to a finite number of discrete values, must be relevant with regard to their intended use. This means that only as many distinctions as necessary are made and thus details not relevant for a certain application are hidden, which is a key motivation for using spatial abstractions. With regard to our work, both the types of spatial relations (e.g. static topological and distance relations as classified in Section 2.4) and the semantics of their qual-
52
3 Representation of Space
itative abstraction – including the represented dimensionality of physical space, the granularity of abstraction, the naming of the relations and the used reference system (cf. [FR93]) – depend on the respective application domain, which has to be taken into account for the design of the middleware framework presented in Chapter 6. As stated above, qualitative abstractions are often the preferred choice when the spatial cognition of humans is involved, as for example in human computer interaction. Reasons for that are that humans are used to think in coarse abstractions and draw conclusions from them [For97, Fre92b, Mus00], and that qualitative approaches are similar to natural language terms and thus considered cognitively more adequate [RM04, Ten05]. With regard to this thesis, it should be easier for a programmer to develop spatially aware applications if he is able to use high-level qualitative spatial relations such as right and near, whose semantics are adapted to the respective application domain, rather than having to deal with quantitative distances and angles for example. A related disadvantage however is that applicationspecific qualitative abstractions of quantitative data constitute an additional layer of indirection which needs to be constructed manually (cf. [Leo98]). Further advantages of qualitative representations are that they allow for dealing with coarse and imprecise information [CFH97, HN02] and that reasoning about qualitative knowledge can be computationally less expensive than processing complex quantitative data, which is due to the separation from unnecessary details that are not relevant for a certain application or in a particular context [CFH97, FR93, GGH06]. These two aspects are important with regard to resource-constrained embedded systems and whenever precise sensor information is not available or simply not required by the application. A considerable amount of research is dedicated to qualitative spatial representation and reasoning, which is mainly concerned with (i) abstracting continuous spatial relations of the physical world and (ii) inferring knowledge from the respective qualitative representations. Application areas include robot navigation [BF04, MF00, MSER99], geographic information systems [AEG94, Sha96], highlevel computer vision [Aie02, CH01] and recently also context-aware applications in the field of pervasive computing [BSHK06, KKG05]; our work is related to the latter. Although the present thesis covers many aspects of qualitative spatial representation and reasoning, that is as far as they are relevant to facilitate the development of spatially aware applications for autonomous embedded systems, just a few of them are discussed in detail; comprehensive surveys with further and advanced information are given in [CH01, FR93, Her94, HN02]. Cognitive and linguistic aspects are not addressed in this thesis, detailed discussions about that can be found in [CBGG97, Her94, Ten05, TM03].
3.2 Qualitative Representation
53
3.2.2 Recognition and Representation of Spatial Relations In the following, we will describe how different types of relations can be recognized by comparing Zones-of-Influence and represented in a qualitative way. This builds up the basis for qualitative spatial reasoning, which is covered in the subsequent Chapters 4 and 5. The recognition of spatial relationships is based on a pairwise comparison of Zones-of-Influence, namely their quantitative spatial properties position, direction and shape, which are exchanged between digital artifacts using XML-based self-descriptions; the exchange of self-descriptions has been discussed in Section 2.2.1 already. We consider just sensors with absolute quantitative spatial information, others which directly provide spatial relations are out of scope. In general, we are convinced that it is much easier for an artifact to determine its own absolute spatial properties and share them with other artifacts for the purpose of relationship recognition, for which a multitude of location systems [HB01] and wireless communication technologies are available, rather than directly acquiring relations to surrounding artifacts without using their spatial properties. Although the latter would allow for an autonomous solution without the need for an external infrastructure, only few technological solutions like the ultrasonic peer-to-peer sensor presented in [HKG+ 05] or the infrared beacons of Polulu Robotics and Electronics [Pol08] are available by now, and they suffer from multiple severe constraints including the required line-of-sight, the limited sensor range and the comparatively low accuracy of the sensor readings. As soon as an artifact receives a self-description, it compares the Zones-ofInfluence of the transmitting artifact which are contained therein with its own ZoIs, and calculates binary relations from them. The latter artifact is commonly referred to as primary object in literature [CH01], the former one – i.e. the artifact which recognizes the spatial relation – is referred to as reference object. As we relate Zones-of-Influence and an artifact may be associated with more than one ZoI at a time (cf. Section 3.1.1), we refer to them as primary ZoI and reference ZoI, respectively. The scope of our work is on binary relations only (i.e. relations between two ZoIs), unary relations (e.g. if and how fast a ZoI is moving) and ternary relations (e.g. that a ZoI is located between two other ZoIs) are not considered. A binary relation is often denoted as R(x, y) and it is read as x is in relation R to y, where x and y denote the primary and reference ZoI; more details on binary relations and their properties will be given in Section 4.1. From an architectural viewpoint, it depends both on the application demands and on the availability of the required spatial information in the compared self-descriptions which relations are being recognized.
54
3 Representation of Space
Figure 3.4: Recognition of spatial relations by comparing self-descriptions.
The relationship recognition is shown in Figure 3.4 by means of a simple example, where the quantitative distance between the physical Zones-of-Influence (i.e. their natural physical position, direction an extension) of two artifacts A and B is determined. In this example, each one of the artifacts is aware of its 2-D Cartesian coordinates from an indoor tracking system, that is (xA , yA ) = (2m, 3m) and (xB , yB ) = (5m, 7m), encodes this information in the description of a ZoI which is exchanged with the respective other artifact using a structured self-description, and eventually calculates the quantitative Euclidean distance by comparing the ZoI-description of its local with that contained in the received self-description (i.e. d = 5m). Detailed information about the structure of self-descriptions will be given in Section 3.3. The recognition of quantitative relations is often a first step before performing a qualitative abstraction, and in this respect especially considered in the Relations Service of the Zones-of-Influence Framework (cf. Section 6.2.3). For relations between physical ZoIs, we often refer to as relations between the respective artifacts. Before having a closer look at different types of qualitative spatial relations that can be recognized by comparing two Zones-of-Influence at a time, there are some issues which have to be considered when representing spatial relations in a qualitative way. According to [FR93], these include (among others): • the dimensionality of the represented space (2-D or 3-D) • the types of relations to be represented (e.g. distance or orientation changes) • the spatial primitives among which relations are defined (e.g. points or intervals)
3.2 Qualitative Representation
55
Figure 3.5: Orientation relations of nearby extended objects, after [Her94].
• the granularity and semantics of the qualitative representation (i.e. how is the space partitioned and which relations are represented) • the reference frame with respect to which a relation is defined (cf. Section 3.2.4) First, the dimensionality of space is determined mainly by application requirements or sensor limitations; in this thesis, 2-D projections of the physical space are used, which are sufficient for many application scenarios and conceptually similar to 3-D. For the representation of spatial relations, it is necessary to decide on a certain kind of spatial primitives first among which they are defined. With regard to positional and directional relations, mainly points [Fre92b, Mor04] and line segments [AEG94, MRW00] have been used in literature, the representation of extended objects for topological relations is discussed in [CBGG97, RCC92]. Similarly, we use spatial extensions for topological relations, as well as points for reasoning about positional relations between Zones-of-Influence (i.e. their anchor points) in Chapter 4, which have shown to be suitable abstractions for positional relations between distant objects during evaluation (cf. Chapter 7). However, once objects are placed in spatial proximity and do not have a uniform spatial extension, points may no longer be appropriate. Instead, it will be necessary to incorporate also the objects’ spatial extensions in order to recognize meaningful relations; respective approaches for orientation and distance relations can be found in [Her94] and [CFH97], respectively. Besides topological relations, for which of course spatial extensions are required, we also used distance relations between the anchor points and shapes of ZoIs for the evaluation scenario presented in Section 7.2.1. The problem of extended objects is exemplarily shown with orientation relations in Figure 3.5. In both cases, the primary object p is considered to be in the relation right to the reference object r if just their center points are used, although the relation front would be more appropriate in the left case. Possible solutions for dealing with orientation relations between rectangular shaped objects are presented in [Her94], which are based on an adaptation of the partitions of space in which
56
3 Representation of Space
Figure 3.6: Definition of static qualitative spatial relations in 2-D.
certain orientation relations hold – i.e. illustrated with equally sized sectors for the relations front and right in Figure 3.5 – to the shape of the reference object.
3.2.3 Static and Dynamic Spatial Relations As stated in Section 2.4, we distinguish between static and dynamic qualitative spatial relations; in this subsection, an overview and a respective classification of spatial relations will be given, which has been published in [Hol07a, Hol07b] already. With respect to the qualitative spatial reasoning literature we use the term object in the following, meaning Zones-of-Influence that are associated with digital artifacts. First, qualitative static spatial relations represent spatial relations between objects at certain points in time. We distinguish four types which are shown in Figure 3.6 by means of an exemplary spatial configuration, whereas two objects p (the primary object) and r (the reference object) are placed in two-dimensional Euclidean space; however, the relations can be defined correspondingly in three dimensions. In this figure, arrows represent the objects’ direction axes, and the resulting qualitative spatial relations for the given configuration are written boldface. For topological relations, we distinguish five possible relations between regions as defined in the RCC-5 Region Connection Calculus [CBGG97]; these are disjoint, overlaps, inside, contains and equal. A more precise qualitative representation is provided with the RCC-8 relations [CBGG97], which cover three additional cases where the regions have contact at their boundaries but do not overlap (cf. Figure 3.6); for real-world scenarios involving physical sensors however, the detection of touching boundaries has shown to be problematic due to their limited precision and accuracy. As tests for spatial relations between complex shaped regions can be
3.2 Qualitative Representation
57
Figure 3.7: Cone-/projection-based orientation and qualitative distance [CFH97, RM04].
computationally expensive (cf. [Leo98]), it should be noted that topological relations may not be feasible for the use with resource-constrained embedded systems. For orientation and directional relations, a cone-based qualitative representation which partitions the 360° range in four equally sized sectors is used, where each one of the respective regions is associated with a certain relation. In this regard, orientation relations describe where the primary object is placed relative to the reference object [CH01, Her94]; therefore, the space around the reference object is partitioned, and the relation is denoted by the region in which the primary object is located. Similarly, directional relations relate the direction of the primary object (as given by its intrinsic direction axis) with that of the reference object, such that the space around the primary object is partitioned according to the direction axis of the reference object, and the region in which the direction axis of the primary object points denotes the directional relation. The most common representation systems are cone- and projection-based [RM04] (cf. Figure 3.7), where it again depends on the application domain which one is used. Concerning applications for autonomous embedded systems, a cone-based representation seems to be most suitable as it easily allows to change the granularity by adding or removing axes and thus allows to cope with sensors of different accuracy. Moreover, sectors of different size can be defined (cf. [Mus99, Mus00] in order to take into account the domain-specific semantics of relations. Distance relations are a measure for the effort which is needed to go from a certain primary to the reference object [HCF95]. In this regard, our focus is on the Euclidean spatial distance; other measures like the time of travel are out of scope. We consider distance relations which partition the space around the reference object r in circular ranges, and assume an isotropic space in which points at the same distance are connected with concentric circles. Each one of the qualitative distances conforms to an interval of quantitative ones [CFH97], defining the qualitative relation between the reference and the primary object; the num-
58
3 Representation of Space
Figure 3.8: Definition of dynamic qualitative spatial relations in 2-D.
ber of intervals again determines the granularity of the relations. The distance relations shown in Figure 3.6 partition the space around the reference object r in circular ranges of the same size, except the outer range which is open; in contrast, [CFH97] defines the range of an outer distance relation as bigger than that of an inner one. For the example of Figure 3.4, the resulting relation of B to A would be medium-dist if the distance ranges have a size of e.g. 4m. Figure 3.7 shows common qualitative representations of orientation and distance relations, which partition the space in the eight cardinal directions north, east, south, west, northeast, south-east, south-west and north-west, and in the five distance levels very close, close, commensurate, far and very far, respectively [CFH97, RM04]. Correspondingly, four types of qualitative dynamic spatial relations shown in Figure 3.8 represent how an object’s spatial relation is changing in terms of metric values at a certain point in time. However, such changes need not result in changes of the corresponding static spatial relations; for example, two objects may be moving away from each other at a certain point in time, but the static spatial relation near may still exist. Dynamic topological relations represent a change in the scale of the primary with respect to the reference object, as for example that it is growing. For dynamic orientation relations, the space is partitioned in the same way as for static ones, but around the primary object; the region in which the thick arrow points denotes the dynamic relation. As opposed to dynamic orientation, dynamic distance and directional relations are one-dimensional properties which can just change through an increase or decrease of the metric distance (moving away or toward) and the angular difference of their direction axes (rotating clockwise or counterclockwise), respectively. From an implementation viewpoint, dynamic spatial relations are estimated by comparing the corresponding quantitative static spatial properties at two or more successive points in time. Additionally, there must
3.2 Qualitative Representation
59
Figure 3.9: Epsilon-neighborhood of a static qualitative orientation relation.
be means for representing the absence of relative motion, for which four further relations – stable-size (for topology), stable-orient (for orientation), stable-dist (for distance) and stable-dir (for direction) – are introduced; in this regard, it should be noted that just relative changes between the primary object and the reference object are represented, for which reason stable dynamic relations do not necessarily mean that there is no motion at all. The relations shown in the Figures 3.6 and 3.8 are so-called jointly exhaustive and pairwise disjoint (JEPD) relations, which means that the relations together cover all possible spatial situations and do not overlap (i.e. just one relation holds at a time); qualitative spatial reasoning (cf. Chapter 4) is commonly realized over such JEPD relations [WFDW07]. An important issue in real-world scenarios concerns the recognition of spatial relations from sensor data, which usually fluctuate although the sensed condition does not change; for example, the Cartesian coordinates reported by a location tracking system typically change within its specified accuracy. Such unstable sensor data may lead to frequent relation changes at the boundaries of two qualitative ranges, where e.g. a small change in the relative position of two objects may cause different qualitative distance or orientation relations. In order to avoid such fluctuations, the quantitative range of the currently existing static relation between two objects can be extended by a certain epsilon value as shown with the relation right in Figure 3.9, and dynamic relations are considered stable unless the quantitative change between two successive sensor values is significantly high. An alternative would be to use fuzzy relations as discussed e.g. in [CFH97, Dut88, Mus00], which is, however, not considered in this thesis.
60
3 Representation of Space
Figure 3.10: Intrinsic and extrinsic orientation relations.
3.2.4 Frames of Reference Another issue related to qualitative spatial representations are frames of reference, which influence the semantics of spatial relationships in that the same spatial situation can be expressed in different ways depending on the used reference frame. For example, the relation that a person p stands in front of another person r can have completely different meanings depending on whether it is related to the viewing direction of r, the shape of the room in which they are situated or the perspective of a third person. As for the spatial properties position and direction, which are given according to a certain reference system (cf. Section 3.1.2), spatial relations have to be represented with respect to a reference frame in order to ensure a common understanding across artifacts and their applications. Several classifications of reference frames can be found in literature, which mostly concern the representation of orientation relations [Mus00]. We adopt the classification given in [AEG94, CFH97], where orientation relations of a primary object p with respect to a reference object r are determined by one of the following three reference frames: • Intrinsic: the orientation is given by inherent properties of the reference object (e.g. the reference object’s front side is given by its axis of direction) • Extrinsic: the orientation is determined by external factors (e.g. the North pole or the object’s motion serves as an external reference direction) • Deictic: the orientation is represented from an external viewpoint (e.g. the viewpoint of an external observer from which the reference object is seen) The distinction between intrinsic and extrinsic reference frames for orientation relations is visualized in Figure 3.10. This classification has also been applied to
3.2 Qualitative Representation
61
distance relations in [CFH97, HCF95], whose scale can analogously be defined with respect to an intrinsic (e.g. depending on the size and shape of the reference object), extrinsic (determined by external factors like the travelling time) or deictic reference frame (i.e. the observer’s relative position influences the perceived distance). Another classification is given in [Kla98, Mus99], where (i) egocentric (i.e. relations between objects are represented with respect to the perspective of an observer, either an external observer in the deictic or the reference object in the intrinsic case) and (ii) allocentric frames of reference (i.e. relations between objects are given with respect to a fixed external coordinate system as in the extrinsic reference frame) are distinguished. Furthermore, [Lev96, Ten05] distinguish between intrinsic, relative and absolute reference frames, where the latter two are comparable to deictic and extrinsic reference frames, respectively. Our focus is on intrinsic and extrinsic frames of reference, as we only consider binary relations between the Zones-of-Influence of two artifacts at a time; relations between artifacts with respect to the viewpoint of a third one, as it is the case with a deictic reference frame [Ten05], are not considered. The reason for this is that our focus is on artifacts that are able to autonomously assess their own spatial situation with respect to their environment, namely by comparing local with received Zones-of-Influences as proposed in Section 3.2.2, for which reason deictic relations between surrounding artifacts are out of scope. As different reference frames are mainly relevant for orientation we consider extrinsic distance relations only, meaning that they are defined by the Euclidean distance between two artifacts – more precisely, between the anchor points or shapes of two Zones-of-Influence at a time.
3.2.5 Spatiotemporal Relations As mentioned in Section 2.4, we also consider time series of spatial relations in addition to their representation at certain points in time, which is simply due to the fact that actions take place in both space and time. Although it is obviously valuable to consider space and time in a combined fashion, there is only little existing work with regard to qualitative representation and reasoning. An approach for qualitative spatiotemporal reasoning about motion sequences is presented in [ZF96] for example, whereas more accurate inference results are derived by interpreting static representations as motions. A representation of the course of motion in the context of robotics using qualitative motion vectors is proposed in [MSE+ 98, Mus99], and [GGH06] qualitatively distinguishes movement patterns in order to recognize activities in a smart home setting. However, to the best of our knowledge, none of the existing approaches addresses the issue of application-
62
3 Representation of Space
specific combinations of spatial and temporal relations to high-level spatiotemporal relations. The consideration of time series of relations allows artifacts to recognize not only the current spatial relations to their surroundings, but also how they are changing over time like for example that another artifact is ”passing by on the right hand side”. Such a single spatiotemporal relation results from a logical and temporal combination of spatial relations, which distinguishes ours from existing work and will be explained in the following. In this regard, our focus is again on qualitative abstractions, which are a basis for qualitative reasoning about spatial relations over time; this section however deals just with the representation of spatiotemporal relations. In Chapter 5, a rule-based approach for inferring new spatial or spatiotemporal relations and triggering appropriate actions from the history of known relations will be presented, which has been published in [Hol07b] and constitutes a core component of the Zones-of-Influence Framework presented in Chapter 6. For representing and reasoning about qualitative spatial relationships among artifacts over time, a representation for the dimension time is required. A discrete time model is used for it, which is motivated by the fact that processing in computing systems is based on discrete clock cycles [Mus00], and thus also the recognition of relations – for which sensor readings and spatial data contained in received self-descriptions are used – happens at discrete points in time, either event-driven or at a fixed rate. Moreover, similar to spatial relations which are not always required with a high precision, it may be meaningful for an application that relations which occur “almost simultaneously” are assigned the same points in time – again depending on the application domain. As we are just interested in temporal order relations, we use a discrete time T with whole-numbered points ti ∈ T , and assume the time difference Δt between two consecutive points in time to be constant (i.e. ti+1 = ti + Δt). In the Zones-of-Influence Framework, discrete time is determined by an update thread with an application-dependent temporal resolution, which will be explained in detail in Section 6.2.3. This time model allows for associating spatial relations with discrete points in time, which is the basis for representing and reasoning about spatiotemporal relations. As with spatial relations, which are defined on spatial primitives like points or line segments (cf. Section 3.2.2), temporal relations have to be defined between certain temporal primitives. We use time intervals on the discrete time T in this thesis, where an interval [s, e] is defined by an ordered pair of a start-point s and an end-point e, and it equals a set of successive points in time: [s, e] = {t ∈ T |s ≤ t ≤ e}
(3.1)
3.2 Qualitative Representation
63
Figure 3.11: Qualitative temporal interval relations, cf. [All83].
We agree with the argumentations in [All83, For97, Fre92a], that the use of intervals corresponds to our intuitive notion of time, and – similar to qualitative spatial relations – intervals allow for qualitative abstractions of temporal relations such as “a occurred before b” or “a happened during b”, which is helpful whenever exact temporal relationships are unavailable or simply not of relevance. Spatiotemporal relations in this thesis are based on the qualitative temporal interval relations proposed in [All83], with the difference that we do not use a continuous but a discrete time. Alternative approaches which use time points rather than intervals can be found in [Fre92a, VK86]. According to Allen’s interval algebra [All83], 13 qualitative temporal relations between two intervals can be distinguished at a time, like for example before, equals and during. Figure 3.11 illustrates these relations between the intervals a and b with pictographic examples, whereas the markings on the time axes represent the discrete points in time t0 , . . .tn−1 at which events like e.g. the recognition of spatial relationships may occur, and exemplary time intervals a and b in which certain relations exist are shown with rectangles encompassing consecutive sequences of discrete points in time. Each line in the figure contains a circle representing the earliest discrete point in time at which the respective interval relation can be detected. As we have proposed in [Hol07b], time intervals in which certain spatial relations exist can be temporally related with such interval relations, like for example that near(y, x)[t,t+5Δt] overlaps with the relation le f t(y, x)[t+2Δt,t+11Δt] , where x and y denote Zones-of-Influence of artifacts and the subscript intervals
64
3 Representation of Space
represent the times in which the respective relations between them exist. Thus, the consideration of such temporal relations between time intervals in which spatial relations exist allows digital artifacts to adapt to spatiotemporal situations, which are characterized by corresponding spatial changes over time. As we do not consider a continuous but a discrete time, the semantics of the relationship definitions given in [All83] are different for the relations before, after, meets and met-by; with regard to Figure 3.11, before(a,b) and after(b,a) are defined with ea < sb in the continuous and ea + Δt < sb in the discrete case, and the relations meets(a,b) and met-by(b,a) are defined with ea = sb in the discrete and ea + Δt = sb in the continuous case (cf. [Köh03]). With the above definition of intervals it is also possible to represent points as intervals with the start- and endpoint being the same. Clearly, if points are related with other points or intervals, some of the qualitative relations shown in Figure 3.11 are not meaningful; these are overlaps, during, starts and finishes as well as their inverse counterparts. As stated above, time intervals in which certain relations exist can be related with qualitative temporal relations. To clarify this idea, Figure 3.12 shows exemplary time series of spatial distance and orientation relations between two artifacts x and y, which are illustrated as points and change over time due to changing spatial properties from which they are calculated. Some intervals are shown with boxes like in Figure 3.11, containing discrete points in time in with certain spatial relations or combinations of them exist. Arrows indicate temporal relations between such intervals; for example, the relation near(y,x)[1,2] overlaps with the relation left(y,x)[2,3] , and near(y,x)[6,6] is in a temporal meets relation to mediumdist(y,x)[7,8] . As can be seen in the figure, intervals of the same or different types of spatial relations can be temporally related. A distinct feature of our work is that temporally combined spatial relations can be interpreted as high-level spatiotemporal relations which exist in certain time intervals, for which reason they can in turn be temporally combined with other relations. Hence, the idea is that logical and temporal combinations of relations, which are relevant for an application, are explicitly represented with a new semantically meaningful name and can eventually be used like any other relation. We just consider combinations of relations between the same two artifacts at a time, which is necessary for using a combined relation like any other binary relation that is also defined between two artifacts. An example of temporally combined relations is given in Figure 3.12, where the three relations near(y,x)[6,6] , medium-dist(y,x)[7,8] and far(y,x)[9,9] – with the temporal meets relation between their intervals – form a new spatiotemporal relation leaving(y,x)[6,9] , which can again be temporally com-
3.2 Qualitative Representation
65
Figure 3.12: Qualitative spatiotemporal relations.
bined with other relations. In the example above, the resulting interval [s, e] of the temporal relation between two intervals a and b is defined as follows: [s, e] = [min(sa , sb ), max(ea , eb )]
(3.2)
However, although using the minimum and maximum of the start- and endpoints in time of the combined intervals makes sense for the given example, it is just one possibility as the resulting interval in general depends on the semantics of the combined relations and is thus up to the application programmer. In addition, time intervals can also be logically combined using and, or and not; as with temporally combined relations, the resulting relation of a logical and or an or combination of two relations is assigned a name reflecting the semantics of the new relation, and the interval in which the logically combined relation is fulfilled is given by an intersection or union of the two intervals, respectively. Thus, the logical combination of relations also results in a new relation and can be used like any other spatial or spatiotemporal relation. In the case of a logical not, the interval of the resulting relation equals the repository’s whole history, except the interval in which this relation holds. For the example of Figure 3.12, the relation combination back(y,x)[8,9] ∧ leaving(y,x)[6,9] can be interpreted as a new relation leaving-backward(y,x)[8,9] . Such combinations of relations lead to semantic problems in the case that disjunctions of alternatively possible relations (i.e. compound relations, cf. Section 4.1) are involved, which is thus disregarded with respect to spatiotemporal representations in the following.
66
3 Representation of Space
In Section 2.5, we stated that relations are stored in the artifact’s repository for further use, in particular for the purpose of rule-based reasoning which will be explained later on in Chapter 5. Due to the additional consideration of time, we have proposed in [HFH+ 08, Hol07b] that each relation is not only stored with a name which identifies the semantics of the relation by its type (e.g. static orientation) including its dimension and the frame of reference, a value which represents a certain relation of that type (e.g. right) and an identification of the reference and primary Zone-of-Influence between which the relation exists (i.e. just binary relations are considered), but also with its time interval which is defined by two discrete points in time. As with spatial properties, qualitative relations can be exchanged with other artifacts via self-descriptions, which is explained in more detail in Section 3.3.2.
3.3 Structure of Self-Descriptions In the Sections 2.2.1 and 2.3, we pointed out the importance of context-sharing by means of self-descriptions for the contextual interaction of digital artifacts and hence for their autonomous operation in the background. A self-description contains information which is of relevance for the application-level interaction among artifacts, as for example an artifact’s physical size, its provided services or the spatial context it is aware of. Based on the concept of spontaneous interaction between artifacts upon coming into spatial proximity which has been proposed in [FHM+ 04, FHR+ 06a], digital artifacts in this thesis exchange their self-descriptions within communication range and compare them with respect to certain application demands, in particular to compute spatial relations that are relevant for a running application from the context information contained therein. As mentioned in Section 2.2.1, the present thesis does not address how and under which circumstances the actual exchange of self-descriptions is performed, which depends on the implementation of the communication component of the framework described in Section 2.5. In order to support the open-world assumption mentioned in Section 2.2.1, namely to enable the interaction between artifacts which do not know each other in advance, we decided for a non-fully specified XML structure of the self-description which enables applications to include arbitrary relevant structured or non-structured information. XML has been chosen because it provides readability for humans and machines on the one hand, and allows to include spatial information represented with different XML dialects – such as those discussed in Section 3.3.1 for example – on the other hand. A notable disadvantage of using XML is its redun-
3.3 Structure of Self-Descriptions
67
dancy, which leads to higher costs concerning storage, processing and communication. The XML-structure of a self-description, which has first been presented in [HFH+ 08], is fairly simple: it consists of an XML root-element DADescription, which contains DADescriptionElement elements with information about the artifact. Different types of self-describing information are distinguished with unique type attributes, the structure of the data contained in a certain XML element however is application-specific and not further specified. The XML fragment of Listing 3.1 shows the structure of a self-description. According to the focus of this thesis, it can contain elements of type ZoI, where each one represents a single Zone-of-Influence of the respective digital artifact (cf. Section 3.1), and of type QualitativeRelation, each one representing the qualitative spatial or spatiotemporal relation relation between two Zones-of-Influence (cf. Section 3.2). The self-description is open for including arbitrary further information like an artifact’s unique identifier or information about its provided services, just by including the corresponding description elements. Details about how they are used in the Zones-of-Influence Framework will be given in Chapter 6. Listing 3.1: Structure of an XML-based self-description. < DADescription > < D A D e s c r i p t i o n E l e m e n t type = " ZoI " > < D A D e s c r i p t i o n E l e m e n t type = " Qu a l i t a t i v e R e l a t i o n " >
3.3.1 Quantitative Spatial Properties There are many ways for representing quantitative spatial properties like position, direction and shape in an artifact’s self-description. A large number of different XML-based markup languages emerged over the past years, which often focus on a certain application domain. Examples are the Keyhole Markup Language (KML) [Goo08], which is designed for geographic annotations with e.g. images, polygons or three-dimensional models in earth browsers or maps, as well as the Geography Markup Language (GML) [CDL+ 04], which is a comprehensive modelling language and interchange format for geographic systems. Another proposed standard is the Physical Markup Language (PML) [FAOH03], an XML-based language for
68
3 Representation of Space
representing information about physical objects with a focus on Electronic Product Code (EPC) network enabled objects. In general, depending on the application requirements, self-descriptions may contain spatial information of arbitrary XML-based markup languages; in particular, all of the above mentioned languages could be used for representing Zonesof-Influence. This flexibility is supported by the software framework presented in Chapter 6 insofar as it allows for an easy exchange of the components that are responsible for parsing self-descriptions. However, the focus of our work is not on the investigation of markup languages for representing spatial information but rather on concepts for making autonomous embedded systems spatially aware, for which reason we have developed a simple data format that is tailored to the description of spatial properties and qualitative relations within the scope of this thesis. For the sake of simplicity, just as many information as required for evaluating the proposed concepts for spatial awareness is represented; it has been derived straightforward from the specification of Zones-of-Influence, and will be explained in the following. As described in Section 3.1, a Zone-of-Influence is specified by its position, direction and shape; thus, an XML-element DADescriptionElement of type ZoI in an artifact’s self-description contains the following information: • Name: Serves as an identification of the ZoI as well as a semantic identifier for applications. The name (element Name) must be unique per artifact. • Position: Defines the position (element Position) of the so-called anchor point of the ZoI (e.g. using WGS84 ECEF coordinates), which specifies a reference point that is important for recognizing relations like distance and orientation to other Zones-of-Influence. It is typically the value sensed by some position sensor; however, it can also be determined by other static or dynamic contexts. Details about the integration of dynamic positions provided by sensors will be given further below. It should be noted that this point need not be the physical position of the artifact or the center of the ZoI’s shape. In order to be able to compare values from different sensors with different reference systems, the used reference system has to be specified using the XML attribute refSystem – similarly to the GML markup language [CDL+ 04]. • Direction: Defines the direction (element Direction) of the anchor point – and thus of the whole Zone-of-Influence – in space, which determines how the overall ZoI is oriented in space (e.g. using Euler angles). As for
3.3 Structure of Self-Descriptions
69
position, the direction can be static or dynamic, and the reference system has to be specified using the attribute refSystem. • Shapes: Each ZoI consists of one or more shapes (element Shape) defining its spatial extension. As stated in Section 3.1.1, we use the term shape synonymously with the spatial extension of a ZoI. Like in PML [FAOH03] also, a shape can be a predefined basic shape like a sphere or a cuboid, but it can also be an arbitrary freeform shape that is defined by a sequence of points. Each shape requires specific parameters describing its characteristics, as for example the width, height and depth of a cuboid. The position and direction of a single shape are given relative – and according to the respective reference system – to the position and direction of the anchor point by using the XML elements RelativePosition and RelativeDirection, respectively. In the following, examples for representing Zones-of-Influence in self-descriptions are given, which should provide a better understanding of the framework and evaluation details presented in the Chapters 6 and 7. The XML fragment in Listing 3.2 describes the static Zone-of-Influence ExampleStaticZone, which means that its position and direction are fixed (i.e. not given by sensor values) and thus do not change over time. Moreover, the XML code shows a fragmentation of the ZoI with two shapes: a cuboid and a point. However, other basic and freeform shapes can also be represented in the description of a Zone-of-Influence. Position and direction in this example are given in Cartesian ECEF coordinates according to the WGS84 reference system and in NED Euler angles, respectively. In this regard, we assume positions to be given in meters and angles in degrees. Listing 3.2: Representation of a static and fragmented Zone-of-Influence. < D A D e s c r i p t i o n E l e m e n t type = " ZoI " > < Name > ExampleStaticZone < Position refSystem = " WGS84ECEF " > <X > 4094824
1189656 4727278 < Direction refSystem = " NEDEuler " > < Yaw > 0.4 < Pitch > -88.5 < Roll > 1.2 < Shape type = " Cuboid " > < RelativePosition > <X > 0.2 1.4 42 < RelativeDirection > < Yaw > -3.7 < Pitch > 4.5 < Roll > 0.0 < Width >2 < Height >2 < Depth >2 < Shape type = " Point " >
70
3 Representation of Space
< RelativePosition > <X >2 0 0
An important aspect of a Zone-of-Influence is that its spatial properties (i.e. position, direction and shape) can be dynamic, since the properties of moving artifacts are subject to changes. A ZoI is called dynamic if at least one spatial property depends on a dynamic context, as for example when its position is determined by a physical sensor like the IS-900 tracking system [Int08]. In order to allow for modelling dynamic Zones-of-Influence, a dedicated DADescriptionElement of type SensorData is used, which specifies both where the data can be acquired from and how the values have to be referenced for their use in the description of a Zone-of-Influence. An exemplary dynamic ZoI is shown in Listing 3.3, where the position values are given by the name of the sensor data – which is PosDir in this example – together with an @ prefix (e.g. @PosDir.x). Detailed explanations of the specification of sensor data accessors in self-descriptions (cf. Listing 6.1) and the corresponding implementations in the Zones-of-Influence Framework can be found in Section 6.2.2. Listing 3.3: Representation of a dynamic Zone-of-Influence. < D A D e s c r i p t i o n E l e m e n t type = " ZoI " > < Name > ExampleDyna m i c Z o n e < Position refSystem = " IS900AtJKU " > <X > @PosDir . x @PosDir . y @PosDir . z < Direction refSystem = " IS900AtJKU " > < Yaw >0 < Pitch >0 < Roll >0 < Shape type = " Sphere " > < Radius >5
The above examples showed how static, dynamic and fragmented Zones-ofInfluence can be defined. As mentioned in Section 3.1.1, an artifact can also be associated with multiple ZoIs; this is implicitly supported by the general structure of self-descriptions, which may contain multiple DADescriptionElement elements of the same type.
3.3.2 Qualitative Spatial Relations A self-description can not only contain quantitative descriptions of Zones-of-Influence, but also qualitative spatial relations between them. Thus, artifacts are able to exchange relations they are aware of among each other, which is a precondition for inferring relations to other artifacts to which none could be determined
3.3 Structure of Self-Descriptions
71
by a comparison of Zones-of-Influence. This is the case if the respective artifacts are out of communication range (and therefore cannot directly exchange their self-descriptions) or when the Zones-of-Influence contained in exchanged selfdescriptions are based on sensor-data which cannot be interpreted by the receiving artifact (e.g. for sensors with unknown or incompatible reference systems), among other reasons. In Section 4.3, an approach for inferring relations among artifacts over multiple hops by using transitive inference is presented, which relies on a local exchange of qualitative relations they are aware of. In contrast to the representation of ZoIs in self-descriptions, the exchange of qualitative relations has not been integrated in the Zones-of-Influence Framework (cf. Chapter 6) at the time of writing this thesis yet, for which reason simulation is used for the purpose of evaluation in the next chapter. As described at the end of Section 3.2.5, qualitative relations between Zones-ofInfluence are stored in the artifact’s repository with a name and a value as well as an identification of the two ZoIs between which the relation exists and the respective time interval. They are thus – by using DADescriptionElement XML elements of type QualitativeRelation as shown by means of an example in Listing 3.4 – represented with the following information in self-descriptions: • Name: Identifies the type of the relation (e.g. StaticOrientation as defined for 2-D in Figure 3.6), which must be unique for the group of artifacts among which self-descriptions are exchanged. With this name, the relation semantics are fully specified, including the dimensionality, the used spatial primitives, its granularity and the frame of reference (cf. Section 3.2.2). • Value: Represents a certain relation in a qualitative way (e.g. right), whose meaning is defined by the corresponding relation type and which must be unique for that type. • Reference ZoI: Specifies the reference Zone-of-Influence of the relation. Since a ZoI must be unique per artifact (cf. Section 3.3.1), it is given by the schema <artifact identification>:. • Primary ZoI: Specifies the primary ZoI of the relation in the same way like the reference ZoI, which is usually a Zone-of-Influence of the artifact from which the self-description has been received. The time interval in which a qualitative relation exists is another important information which has to be stored in the relations repository in order to enable spatiotemporal reasoning. However, just the currently existing relations without any temporal information are exchanged with self-descriptions, which is mainly for
72
3 Representation of Space
two reasons. First, in order to take advantage of other spontaneously networked artifacts’ relations which are associated with certain intervals in time, a synchronization of the clocks among them – which requires additional computing and communication resources – would be necessary. Secondly, and most importantly, a synchronization of clocks is contrary to our general focus on autonomously operating digital artifacts, which act independently of other artifacts and on the basis of locally perceived information only. Listing 3.4: Representation of a qualitative relation. < D A D e s c r i p t i o n e E l e m e n t type = " Qu a l i t a t i v e R e l a t i o n " > < Name > StaticOrientation < Value > right < ReferenceZoI > 00 -00 -00 -70 -05 -75 - D3 -49 :PhysicalShape < PrimaryZoI > 00 -00 -00 -70 -05 -75 -67 -1 B:PhysicalShape
3.4 Summary In this chapter we have discussed representations of spatial aspects, which are the basis for providing spatial awareness to spontaneously interacting artifacts. Zones-of-Influence have been introduced as an approach for representing quantitative spatial properties which are relevant for an application. Each artifact can be associated with one or more of such Zones-of-Influence, which are geographic regions of a certain shape and size that can be defined relative to an artifact’s position and direction in space. Just static spatial properties are taken into account, and their position and direction is represented with respect to a fixed, external frame of reference. Extensions of this concept by additionally including dynamic spatial properties such as the artifact’s angular acceleration or linear velocity are of course possible, but not considered within this thesis. For the sake of simplicity, we are mainly concerned with two-dimensional Zones-of-Influence; however, the presented concepts are conceptually similar for spatial properties in three dimensions. Zones-of-Influence are shared among artifacts with so-called self-descriptions, which enables them to recognize spatial relations by a pairwise comparison of their local with received Zones-of-Influence. Our focus is on qualitative relationship abstractions, which have advantages whenever the spatial cognition of humans or resource-constrained systems are involved, among others. A crucial point is the observation that the semantics of qualitative relations – including the used spatial primitives, the way how the space is partitioned and the reference frame
3.4 Summary
73
with respect to which they are defined – depends on the spatially aware application that makes use of them. There are often multiple ways for determining a certain relation; for example, distance relations can be determined by relating point abstractions of two artifacts, or by defining Zones-of-Influence whose spatial extensions correspond to qualitative distances and recognizing topological overlaps relations between them. We have discussed a classification of spatial relations with respect to the topology, position and direction of two Zones-of-Influence at a time by means of examples. In this regard, static relations such as overlaps and left as well as dynamic relations such as moving-away or growing, which we consider valuable for characterizing the spatial situation of real-world objects with respect to their surroundings, have been distinguished; however, cognitive aspects concerning the semantics of relations are disregarded in the present thesis. We use just binary relations (i.e. relations between two Zones-of-Influence), and thus only intrinsic and extrinsic reference frames for the representation of orientation. For the consideration of time-series of qualitative spatial relations, we propose their logical and temporal combination to spatiotemporal relations, which allows for dealing with trajectories of spatial relations that have a certain meaning for the application in a qualitative way. Therefore, relations are associated with discrete time intervals in which they exists, which can be related with qualitative temporal relations such as before or during. Each relation, which results from a combination of other relations, is explicitly represented with a semantically meaningful name as well as a time interval in which it exists; a rule-based approach for inferring high-level relations by a combination of known relations will be presented in Chapter 5. Thus, an inferred relation can in turn be combined with other relations, which constitutes a distinguishing feature of our work. Compound relations – i.e. disjunctions of alternative relations which are commonly the result of composition operations (cf. Chapter 4) – are disregarded with respect to relationship combinations. XML-based self-descriptions are used for the exchange of information which is relevant for the application-level interaction between artifacts, whereas our focus is on quantitative spatial properties as well as on qualitative spatial relations. We have proposed and exemplarily discussed a structured data format for their representation within self-descriptions in this chapter, which is kept as simple as possible so as to be able to realize and evaluate the developed concepts for spatial awareness in Chapter 7. The semantics of exchanged spatial information (e.g. given by the reference system of a Zone-of-Influence or the type of a qualitative relation) must be known by all interacting artifacts to be able to use them. For the exchange of qualitative relations, just currently existing relations without any temporal information
74
3 Representation of Space
are considered in order to cope with the autonomy requirement of spontaneously interacting digital artifacts.
4 Distributed Spatial Reasoning It is quality rather than quantity that matters. Lucius Annaeus Seneca, c. 4 BC - 65 AD
Reasoning about perceived context information is fundamental for autonomous systems to draw conclusions about their spatial contexts and adapt to changing situations at runtime. According to our focus on qualitative spatial relations, this chapter builds on the qualitative abstractions presented in Chapter 3 and is concerned with inferring new from known relations. In Section 4.1, we first give an overview of approaches for qualitative spatial reasoning, with a focus on binary relations and their properties as well as compositional reasoning about them. In Section 4.2, we discuss reasoning about positional and directional relations in more detail and present composition tables for distance and orientation relations. The main part of this chapter is about an algorithm for inferring and distributing spatial relationships among autonomous digital artifacts over multiple hops, which is presented in Section 4.3 and evaluated by simulation means in Section 4.4. The general idea is that artifacts recognize spatial relations to others in their vicinity by sharing spatial context information, and infer relations to artifacts out of communication range (i.e. data cannot be exchanged directly, but just via other artifacts) by means of qualitative spatial reasoning techniques which are discussed in this chapter. In particular, the algorithm makes use of composition tables or the properties of binary relations in order to provide artifacts, which are out of communication range, with an awareness about their spatial relationships to each other. Section 4.5 eventually discusses benefits and drawbacks of the presented qualitative spatial reasoning approaches and elaborates on their use for autonomous embedded systems.
4.1 Overview of Qualitative Approaches Qualitative spatial reasoning in general builds upon spatial relations as discussed in Section 3.2, and aims at providing computers with the ability to draw conclusions in a qualitative manner. With regard to our focus on autonomous embedded
76
4 Distributed Spatial Reasoning
systems, qualitative spatial reasoning allows digital artifacts to infer new relations based on known ones, and thus improves their spatial awareness in that implicitly available knowledge is made explicit and can be directly used at the application level. It is typically realized with calculi over sets of qualitative spatial relations, from which a considerable number has been developed within the past two decades. They focus on different spatial properties, where most of the research efforts have been concerned with topological reasoning about regions as well as reasoning about distance and orientation relations on points or line segments [DFWW06, KB99]. In the following, the foundations of qualitative spatial reasoning will be discussed as far as they are relevant to understand the proposed approach for relationship distribution presented in Section 4.3; starting points for further information on this research field can be found in several survey papers [CH01, FR93, HN02] and books [For97, Her94]. As stated in Section 3.2.2, this thesis is concerned with binary relations between artifacts – or, more precisely, between Zones-of-Influence which are associated with them; we use object synonymously for Zone-of-Influence in the following, as it is the common term in the research field of qualitative spatial representation and reasoning. A relation R between two objects x and y (i.e. (x, y) ∈ R) is often denoted as R(x, y), where x is referred to as the primary and y as the reference object. According to [DFWW06], a binary spatial calculus consists of (i) a domain D containing the spatial primitives (e.g. points in two-dimensional space representing physical objects), (ii) a finite set BR of typically jointly exhaustive and pairwise disjoint 2-ary base relations on that domain, as well as (iii) a set of operations which are closed over the powerset R of these base relations. Thus, the result of an operation may be a compound relation (i.e. the union of multiple base relations), which allows for expressing uncertainty (e.g. that an object x is either left or front of y); for this reason, the operations of a calculus have to be defined for all possible unions of base relations (cf. Section 4.2.2). A special case in this regard is the universal relation, which is a disjunction of all base relations and represents the complete lack of knowledge about the relation between the respective two objects [NS02, WFDW07]. The following operations union (∪), intersection (∩) and composition (◦) on binary relations are relevant for our approach, where R, S ∈ R [DFWW06]: R∪S
= {x|(x ∈ R) ∨ (x ∈ S)}
(4.1)
R∩S
= {x|(x ∈ R) ∧ (x ∈ S)}
(4.2)
R◦S
= {(x, z)|∃y ∈ D : (x, y) ∈ R ∧ (y, z) ∈ S}
(4.3)
4.1 Overview of Qualitative Approaches
77
In the Sections 4.1.1 and 4.1.2, closures of binary relations and compositional reasoning are discussed, which build up the basis for the relationship distribution algorithm presented in Section 4.3. In this regard, our focus is on static positional and directional relations, topological and temporal relations will not be considered in the remainder of this chapter.
4.1.1 Properties and Closures of Binary Relations Once an artifact has become aware of certain qualitative relations to and between other artifacts, it may be able to infer new relations from them, which is the subject of qualitative spatial reasoning. One possibility is to exploit the properties reflexivity, symmetry and transitivity of binary relations R over a set D, where any of them may but need not be fulfilled for a given relation: ∀x ∈ D : ∀x, y ∈ D : ∀x, y, z ∈ D :
(x, x) ∈ R (reflexive)
(4.4)
(x, y) ∈ R ⇒ (y, x) ∈ R (symmetric)
(4.5)
(x, y) ∈ R ∧ (y, z) ∈ R ⇒ (x, z) ∈ R (transitive)
(4.6)
Table 4.1 shows a selection of spatial orientation and distance relations as presented in Section 3.2.3, together with their properties. These relations are extrinsic relations, i.e. referring to an external reference like the earth or a building (cf. Section 3.2.4). For example, the relation left is transitive, as an artifact x, which is left of y, is also left of z if y is left of z, but it is neither symmetric nor reflexive. On the other hand, the relation near is both symmetric (if x is near y, then y is also near x) and reflexive (as x is inherently near to itself), but not transitive as the distance between x and z not only depends on the distance between x and y as well as between y and z, but also on the orientation relation between those artifacts [CFH97]. The properties of intrinsic orientation relations however are not transitive, which is for the following reason (cf. [Fra98]). In the above example that x is left of y, the orientation relation is defined with respect to the walls of a room for example, but independent of the artifact’s direction in space. In this extrinsic case, it does not matter which one of its sides is directed upwards for example. On the other hand, if the artifact’s intrinsic front side is of relevance for the application – e.g. given by an embedded sensor which determines the artifact’s axis of direction, which we have utilized e.g. in [FRH06, FRHR05] – the relation left is no longer transitive as it depends on the reference artifact’s spatial direction (i.e. the artifacts y and z in the above case). The properties of binary relations can be exploited for inferring relations between artifacts without reverting to quantitative spatial information, which we use
78
4 Distributed Spatial Reasoning Table 4.1: Extrinsic orientation and distance relations with their properties.
reflexive
symmetric
transitive x x x x
front right back left near medium-dist far
x
x x x
for our relationship distribution algorithm that has first been published in [HF07]. The general idea is to process the transitive, symmetric and reflexive closure for all relations over a group of artifacts, which are defined as follows: r(R) = R ∪ Δ with Δ = {(x, x)|x ∈ D} (reflexive closure)
(4.7)
s(R) = R ∪ R−1 with R−1 = {(y, x)|(x, y) ∈ R} (symmetric closure) (4.8) t(R) =
∞
Ri with Ri+1 = Ri ◦ R (transitive closure)
(4.9)
i=1
From an implementation viewpoint, the closure with respect to a certain relation property is processed iteratively for every relation R over a group of artifacts. As the artifacts are distributed in space, each artifact x shares both quantitative spatial properties and qualitative spatial relations it is aware of via self-descriptions with others in communication range, and infers additional relations R as defined by the respective closure. Like those relations which have been recognized by a comparison of quantitative spatial properties, inferred relations are eventually added to the artifact’s relations repository and included in its self-description. To find the reflexive closure, x infers the relation (x, x) ∈ R. For the symmetric closure, x infers (x, y) ∈ R for each relation (y, x) ∈ R it is aware of. In both cases, inferred relations concern the direct neighbors of an artifact only. For the transitive closure, artifact x infers the relation (z, x) ∈ R if it is already aware of (y, x) ∈ R and acquires the qualitative relation (z, y) ∈ R through a received self-description of artifact y. The transitive closure provides efficient means for inferring relations between artifacts out of range, namely without routing self-descriptions over multiple hops; a detailed explanation will be given in Section 4.3.1. A precondition for process-
4.1 Overview of Qualitative Approaches
79
ing closures is that the involved artifacts have knowledge about the properties of exchanged relations.
4.1.2 Compositional Reasoning Compositional reasoning has received much attention in the research community, as for example in [All83, Fre92b, Her94, Sha96]. Basically, the fundamental question involved is the following: “Given the relation between two objects x and y, as well as between y and z, what is the relation between x and z?”. The result of a composition operation depends on the meaning of the relation, that is it has to be developed from the relation’s geometric semantics [MDF05], and it is often a compound relation (i.e. a set of alternatively possible base relations). The results are commonly stored in composition tables, which define the resulting relations for all possible compositions of base relations; compound relations can be computed as the union of the compositions of base relations. Usually, an iconic notation is used for it, which represents the neighborhood structure of relations; two relations are referred to as conceptual neighbors if and only if they can be directly transformed into each other (i.e. without passing other relations) [Fre91]. A detailed explanation of the composition of positional relations on the basis of composition tables will be given in Section 4.2. The transitivity of relations can be seen as a special case of the composition operation, where all three relations – the two composed ones as well as the resulting relation – are the same [Sha96]. This is obvious from the explanations in Section 4.1.1, where the transitive closure is defined as the union of repeated composition operations on pairs of objects between which the same relation R holds. Although this approach is universal in the sense that it can be applied to arbitrary relations whose properties are known, it is quite limited as many relations like distance and intrinsic orientation are not transitive. Based on the exploitation of transitivity, we thus proposed an approach for distributing relationship information throughout a network of artifacts in [Hol07a] by using compositional reasoning, which will be presented in Section 4.3.2. To the best of our knowledge, the distribution of qualitative spatial relations among digital artifacts using transitive inference as described above is a novelty of this thesis. Figure 4.1 should clarify the idea of compositional reasoning among digital artifacts by means of a simple theoretical example. It shows a distribution of physical objects o1, . . . o8 in space, as well as a person p which is interested in its orientation and distance relations to them. We assume that the person and all objects are equipped with digital artifacts that are able to localize themselves according to an extrinsic frame of reference (i.e. represented with arrows) and communicate
80
4 Distributed Spatial Reasoning
Figure 4.1: An example for compositional reasoning.
with each other within a short range (i.e. represented with dotted lines between them). However, although artifact p is able to communicate with o3 only, it should acquire the location of all other objects relative to its own position. This can be solved trivially by letting each artifact exchange its position information with all other artifacts. In the following sections, we will tackle the issue of how compositional reasoning can be exploited to answer such questions with purely qualitative relations and by exchanging spatial information with direct neighbors only. The idea is that artifacts determine relations to their direct neighbors (e.g. o4 recognizes that o6 is in the relations front and medium-dist, and o3 recognizes that o4 is front and near to itself), exchange this knowledge with them (e.g. o4 tells o3 about its relation to o6) and thus allow their neighbors to infer relations to artifacts out of range using appropriate composition tables (e.g. o3 infers that o6 is front and either in a medium-dist or far distance relation according to Table 4.5).
4.2 Reasoning about Positional and Directional Relations In this section, we will have a closer look at compositional reasoning, where the focus is on positional (i.e. orientation and distance) and directional relations as previously shown in Figure 3.6. First, a literature survey on related work is presented, which focuses on popular spatial calculi that are closely related to our work; due to the huge number of calculi however, the survey does not claim to be exhaustive. Afterwards, a new spatial calculus that has been published in [Hol07a] is presented, which allows for reasoning about static orientation and distance rela-
4.2 Reasoning about Positional and Directional Relations
81
Table 4.2: Comparison of approaches for reasoning about static spatial relations.
orientation
a no
distance
Freksa Ligozat et al. Hernández et al. Abdelmoty et al. Moratz et al. Renz et al. Zimmermann et al.a Clementini et al.a
x x x x x x x
x x
Proposed approach
x
x
direction
x x x x
xb
calculus is specified considered for the relationship distribution algorithm of Section 4.3
b not
tions in a combined fashion. We consider the proposed approach valuable for the use with resource-constrained autonomous embedded systems, and it is eventually exploited for our relationship distribution algorithm presented in Section 4.3.
4.2.1 Related Approaches Table 4.2 shows a comparison of popular qualitative spatial calculi which are classified according their consideration of (i) orientation, (ii) distance and (iii) directional relations. As stated above already, topological and temporal relations are not taken into account in this regard. Many spatial calculi incorporate orientation only, like for example Freksa’s ternary double-cross calculus [Fre92b] and the flip-flop calculus of Ligozat et al. [SN04], which describe the deictic orientation of a point in the plane with respect to a vector that is given by two further points. However, they can represent intrinsic or extrinsic orientation relations also by viewing them as positional binary relations between a dipole and an isolated point at a time. On the other hand, there are only a few calculi dealing with spatial distance only, as for example that of Hernández et al. [HCF95] who particularly addressed the composition of distances depending on intrinsic orientation relations. Early work that combines orientation and directional relations is that of Abdelmoty et al. [AEG94], which represents them by computing the intersections of orientation lines. The binary dipole relation algebra of Moratz et al. [MRW00] uses
82
4 Distributed Spatial Reasoning
straight line segments, which are each determined by a pair of points in order to represent orientation relations between objects. A continuative calculus of Moratz et al. is the oriented point relation algebra [MDF05], where oriented points are used instead of dipoles, and the granularity is adjustable with a single parameter; the exact set of base relations thus depends on the chosen level of granularity. A similar approach with arbitrary granularity is presented by Renz et al. in [RM04], who developed the star calculus for relating two points in a plane with respect to a reference point or direction. However, there is only few existing work about the combination of orientation and distance relations. Zimmermann et al. [ZF96] add distance to their ternary calculus for representing orientation relations, and show how distance information restricts the possible orientations; this dependency is also discussed in [Sha96]. Clementini et al. [CFH97] show the interplay between orientation and distance relations, but also do not present composition tables for reasoning about them in combination. To the best of our knowledge, there is no existing work which deals with compositional reasoning about combined binary orientation and distance relations as presented in Section 4.2.2, neither taking into account directional relations nor without considering them. Moreover, in Section 4.3 we use it for distributing relationship information among autonomous artifacts, which also seems to be new.
4.2.2 Composition of Positional Relations Building on the fundamentals of compositional reasoning explained in Section 4.1.2, we specifically address the composition of static positional relations in the following. We consider them the most valuable ones with respect to relating spatially distributed digital artifacts, as they can be processed with comparatively little computing efforts and are thus suitable for resource-constrained embedded systems. Moreover, most topological relations are not meaningful for physical objects (in contrast to Zones-of-Influence which may be associated with them), and directional relations – as for example that two objects are directed similarly – are often not relevant for spatially aware applications. This estimation is also reflected by the related work survey in Section 2.6, where only few projects take advantage of relations between the directions of artifacts. We have developed composition tables for the four orientation base relations front, right, back and left as well as for the three distance base relations near, medium-dist and far, according to their definitions in Section 3.2.3. Composition tables specify the possible relations RS(z, x) of a certain relation type between two objects x and z by taking into account their relations R(y, x) and S(z, y) to a com-
4.2 Reasoning about Positional and Directional Relations
83
mon third object y, separately for all possible combinations of base relations. As mentioned already, the composition of base relations has to be computed depending on their semantics, which means that we have to consider how exactly they partition the space; in the following, this is in four equally sized sectors for orientation and three circular ranges of the same size (except the outer range which is open) for distance relations as shown in Figure 3.6. In general, it can be said that the construction of composition tables is an error-prone and time-consuming process [DFWW06], but they make efficient reasoning by simple table lookups possible once they are available. Although not considered in this thesis, composition tables can also be used for composing relations of different types, for example to infer topological from directional relations as has been done in [Sha96]. An important issue regarding orientation relations is the considered reference frame, as the result of a composition operation depends on the directional relation between the objects x and y. For an extrinsic frame of reference (cf. Section 3.2.4), an external reference direction is used for all objects involved in the reasoning process, which means that they have exactly the same direction in space. In the case of an intrinsic frame of reference, the orientation relations depend on the direction of the respective reference objects x for the relation R(y, x) and y for the relation S(z, y) (cf. [Her94]). For qualitative compositional reasoning about intrinsic orientation relations, it is thus necessary to integrate qualitative directional information in the reasoning process also. We therefore distinguish the relations similar-dir, right-dir, opposite-dir and left-dir, which have been defined in Section 3.2.3. Consequently, dedicated composition tables for intrinsic and extrinsic positional relations are required, with 12 base relations in the former (due to the four orientation and three distance relations) and 48 base relations in the latter case (due to the four additional directional relations). A finer granularity by distinguishing more relations is of course possible; however, changes in the number and semantics of relations require a reconstruction of the composition table. How the composition of positional relations among three artifacts x, y and z is acquired can be seen in Figure 4.2, exemplarily for the two positional relations front(y,x) ∧ near(y,x) and right(z,y) ∧ near(z,y), and for an extrinsic reference frame at the top as well as for a similar direction between the objects x and y – i.e. similar-dir(y,x) – at the bottom of the figure. Due to the abstraction of space in discrete partitions, a composition operation usually results in a compound relation, which means that not a single but multiple alternatively possible relations are provided. While the former composition results in a disjunction of just four relations – these are front(z,x) ∧ near(z,x), front(z,x) ∧ medium-dist(z,x), right(z,x) ∧ near(z,x) as well as right(z,x) ∧ medium-dist(z,x) – the latter results in a set of even six relations as a consequence of the additional range of possible alternative
84
4 Distributed Spatial Reasoning
Figure 4.2: Composition of positional relations with same (top) and similar directions.
directions of object y with respect to x. It can be seen that intrinsic orientation relations lead to many disjunctions and are thus practically less relevant, for which reason our focus is on extrinsic orientation and distance in the following. The results of intrinsic composition will become more specific if the number of distinguished qualitative directional relations is increased, but at the same time lead to bigger tables due to the higher number of base relations. The Tables 4.3 and 4.4 show the separate composition for extrinsic orientation with four and distance with three base relations, which means that orientation and distance are composed independently. An iconic notation is used, which represents the neighborhood structure among the positional relations (cf. Section 4.1.2). The four orientation base relations are visualized with black dots indicating their orientation, and the three distance base relations are visualized with filled areas indicating their spatial ranges; disjunctions of base relations, which are often the result of a composition operation and represent possible alternative relations, are visualized by superimposing their icons like for example in [Fre92b]. The results in the main diagonal of Table 4.3 equal the corresponding orientation base relations, which implies that static orientation is a transitive relation. As stated in Section 4.1, the composition of compound relations can be computed by a composition of the respective base relations and a subsequent union of the resulting – possibly compound – relations. From these tables, it can be seen that the composition operation often leads to coarse results, which is also depicted in the literature
4.2 Reasoning about Positional and Directional Relations
85
Table 4.3: Composition table for static orientation relations.
Table 4.4: Composition table for static distance relations.
[Her94, Sha96]; for example, the compositions of front and back or of near and medium-dist result in the union of all orientation or distance base relations respectively, which are referred to as universal relations. The composition table for combined orientation and distance relations that has been published in [Hol07a] is shown in Table 4.5, partially for six base relations at a time; all others can be determined by simply rotating the table. The composition table has been created by moving an object y within the range of a particular quali-
86
4 Distributed Spatial Reasoning Table 4.5: Composition table for combined orientation and distance.
tative relation R(y, x) to x as shown in Figure 4.2, and determining which positions are possible for a third object z in order to fulfill a second base relation S(z, y) to y; these possible positions eventually define the resulting compound relation RS(z, x). The upper example of Figure 4.2, namely static orientation for an extrinsic reference frame, is highlighted in Table 4.5. As can be seen, more accurate results are achieved by a combined consideration of orientation and distance; for example, composing the extrinsic distance relations medium-dist(z,y) and far(y,x) only yields the universal relation, but additionally taking into account e.g. the orientation relations front(z,y) and front(y,x) results in just one distance relation, namely far(z,x). Another example are the orientation relations front(z,y) and back(y,x), whose composition again results in the universal orientation relation, but much more specific results can be inferred in combination with the according distance relations.
4.3 Spatial Relationship Inference and Distribution
87
4.3 Spatial Relationship Inference and Distribution This section presents an approach for inferring and distributing qualitative spatial relations between digital artifacts, with the aim of providing a group of artifacts with an awareness about spatial relations among each other. Artifacts hence cooperate by sharing their spatial contexts, which allows them to recognize relations to artifacts in vicinity and infer further relations – even to others that are out of range – using qualitative spatial reasoning techniques as discussed in Section 4.1. After presenting the general concept by taking advantage of the properties of binary spatial relations, an algorithm is proposed which is based on compositional inference and thus allows for inferring more significant conclusions. An evaluation of the algorithm with different topologies of networked groups of artifacts by simulation means will be given afterwards in Section 4.4.
4.3.1 General Concept by Exploiting Relation Properties We distinguish two general approaches for providing a group of artifacts, which are either connected directly (i.e. they are within communication range) or via other artifacts, with a mutual spatial relationship awareness: (i) exchanging quantitative spatial properties between each pair of artifacts or (ii) sharing spatial properties with directly connected artifacts only, together with their knowledge about qualitative spatial relations to other artifacts. While the former can be done by simply broadcasting spatial properties throughout the network, which means that selfdescriptions are exchanged between distant artifacts over multiple hops, we developed a new approach for the latter which is based on locally perceived information only. The proposed approach exhibits self-organizing behavior in that artifacts autonomously, and in an ad-hoc manner, establish a collective spatial awareness of each other (cf. Section 2.3). Although there exist several approaches for localizing nodes in ad-hoc networks (cf. [LR03]), as for example [BSHK06] and [CHH02] which are both based on quantitative distance estimates between nodes, ours seems to be the only one which is solely based on qualitative spatial relations. The exchange of self-descriptions is the basis for relationship inference and distribution among digital artifacts. As explained in Section 2.2.1, the self-description of a particular artifact contains both quantitative spatial properties represented with Zones-of-Influence and qualitative spatial relations it is aware of (e.g. acquired by a pairwise comparison of its local with received self-descriptions), as well as any additional information which is required by a particular application. In the case of our relationship distribution algorithm presented below, this is an identification of the artifact as well as a sequence number which is unique for a particular process
88
4 Distributed Spatial Reasoning
of inferring and distributing relations among a group of artifacts. Each qualitative relation is given by (i) a unique name identifying its type, (ii) a value which represents a certain relation of that type as well as (iii) the reference and primary artifact between which the relation exists; in contrast to Section 3.2.5 however, no time intervals are stored as temporal relations are not considered in the following. Due to our focus on positional relations, digital artifacts are abstracted using point-shaped Zones-of-Influence, to which we refer to as artifacts in the following as they simply represent the artifacts’ positions. Moreover, we assume that their qualitative relations to each other do not change during the process of relationship inference and distribution. The simplest way for achieving mutual spatial relationship awareness is a flooding of self-descriptions, which enables each artifact to recognize relations by a pairwise comparison of its own spatial properties with those contained in received self-descriptions. We consider it the most feasible approach for the case that the self-description of each artifact has to be sent to all other artifacts of a group. We are aware of alternative broadcasting techniques surveyed in [WC02], which are based on (i) an assignment of probabilities for rebroadcasting messages, (ii) the assumption that nodes have common transmission distances or (iii) the maintenance of knowledge about its neighborhood by each node, and therewith aim at reducing the number of redundant broadcasts; however, as they either do not guarantee that all nodes are reached or require topology information obtained from periodic "hello" packets, we decided for using simple flooding. Nevertheless, flooding (i) induces a high amount of traffic as will be seen in Section 4.4, which is because each artifact broadcasts all self-descriptions it receives for the first time, and (ii) it requires much processing power due to the fact that all relations are calculated from the quantitative spatial properties. With regard to battery-powered embedded systems with constrained computing resources, flooding may thus simply exceed their computing and communication capabilities. An alternative we have proposed in [HF07] first is to exploit the relation properties reflexivity, symmetry and transitivity as defined in Section 4.1.1 in order to infer certain relations with purely qualitative means. Assuming that an artifact also shares its qualitative spatial relations and is aware of those relations’ properties, it is able to acquire relations without having to process them from quantitative spatial properties (which are possibly unavailable). By sharing inferred relations with their own self-descriptions, in particular the transitive closure with respect to a certain relation can be processed, which allows for acquiring relations between distant artifacts without an exchange of spatial contexts between them.
4.3 Spatial Relationship Inference and Distribution
89
In the remainder of this section, the general concept of our approach for inferring and distributing relations is explained, which comprises the following four steps: 1. A certain artifact y broadcasts its self-description, e.g. in regular intervals or due to significantly changed sensor values as a result of movement, to surrounding artifacts in communication range. 2. Each artifact x, which receives the self-description of the broadcasting artifact y for the first time (i.e. determined by the sequence number mentioned above), recognizes spatial relations R(y, x) to y according to the application demands (cf. Section 3.2.2) from the quantitative spatial properties contained therein, and adds them to its own self-description. 3. It then infers further relations from the already known and newly recognized or received ones (i.e. contained in the self-description of y) by taking advantage of qualitative spatial reasoning techniques – either based on relation properties (cf. Section 4.1.1) or on compositional inference (cf. Section 4.1.2) – and adds them to its self-description, too. 4. If either the relations in the self-description of x changed (i.e. new relations have been added) or it is the first time a self-description has been received, x broadcasts its self-description containing current quantitative spatial properties – in the case of limited communication bandwidth they can be included just for the first broadcast – and the qualitative spatial relations it is aware of, together with the sequence number of the received self-description. More detailed information about the algorithmic implementation of the above approach will be given in Section 4.3.2. The Figures 4.3 and 4.4 demonstrate how the algorithm works for a two-dimensional Euclidean space. Although possible, we do not consider a hop-count in the following, which would allow for restricting the spatial range in which relations are distributed. The vertices of the graphs represent a group of five wirelessly communicating artifacts a, . . . e, and edges between them show that the respective artifacts are within communication range. Broadcasts of self-descriptions are illustrated with curved arrows from one vertex to another, and the relations below them show the spatial relationship awareness of the respective artifacts at certain points in time during the relationship distribution. A projection-based qualitative abstraction has been used for the transitive orientation relations, which partitions the space in the four half-planes left, right, front and back using two orthogonal axes. They are aligned according to an external reference frame that is indicated by the x- and y-axes at the top left of the figures.
90
4 Distributed Spatial Reasoning
Figure 4.3: Distribution of the transitive relation left.
The reflexive and symmetric distance relations are defined with concentric circles as described in Section 3.2.3, where the four distances near, medium-dist and far are distinguished. To keep the examples short and clear, we will consider just the relations left and near. They are defined as follows, where d is a certain distance which equals the range of near and xi , yi ∈ R indicate the Euclidean position of artifact i: left(a,b)
⇔
xa < xb
near(a,b)
⇔
(xa − xb ) + (ya − yb ) < d
(4.10) 2
2
2
(4.11)
In step (1) of Figure 4.3, artifact d first broadcasts its self-description with a unique sequence number, which is received by the artifacts c and e. They thereupon perform the recognition, inference and possible broadcast as described above. While artifact e recognizes that artifact d is on the left hand side, i.e. e has become aware of the relation left(d,e), artifact c clearly does not recognize this relation as it is to the right of d. Both artifacts c and e then broadcast their own self-description in step (2) as they have received it for the first time, whereby the self-description of e already contains the newly recognized relationship to d. This distribution of relations continues in step (3), where more and more relations are being recognized.
4.3 Spatial Relationship Inference and Distribution
91
Figure 4.4: Distribution of the symmetric and reflexive relation near.
In step (4) of Figure 4.3, artifact d infers the two relations left(a,d) and left(b,d) from the relations left(a,c) and left(b,c) it received from artifact c as well as the knowledge about the relation left(c,d) and that left is a transitive relation. Due to the changed relations, artifact d again broadcasts its self-description, now containing all three relations it is aware of. This in turn also makes artifact e aware about its relations to a and b in step (5). Finally, this distribution process terminates in step (6), as no further relations are inferred. Figure 4.4 shows a similar picture for the reflexive and symmetric relation near, again with artifact d broadcasting its self-description first. Artifact c, which receives the self-description of d, recognizes the relation near(d,c) by a comparison of the spatial properties of c and d, and then infers the relations near(c,d) due to the symmetry of the relation near as well as near(c,c) due to its reflexivity. According to the distribution approach described above, we assume that the relationship inference always takes place upon receiving a self-description; for this reason, also the relation near(d,d) is not recognized until step (2), although the overall reflexive closure could be processed by each artifact without requiring information contained in received self-descriptions. For both examples, the result is a group of artifacts in which every one is aware of its left and near relations to each other, as far as they could be recognized and inferred by taking advantage of their properties. From a mathematical viewpoint, the transitive closure of the relation left as
92
4 Distributed Spatial Reasoning
well as the reflexive closure and the symmetric closure of the relation near have been calculated (cf. Section 4.1.1). It should be noted that the figures are simplified in that all nodes broadcast at the same time; in fact, artifacts broadcast their self-descriptions independently from each other. In contrast to the transitive orientation example, no relations are inferred over multiple hops in case of the reflexive and symmetric distance relation. However, as we are interested in inferring relations between distant artifacts which cannot be acquired directly, like for example if they are out of sensor- and communicationrange, our focus will be on transitive inference in the following. Exploiting the transitivity property for reasoning purposes has the advantage that it can be applied to arbitrary binary relations and is not restricted to spatial ones; for example, all relations with the semantics “equals”, “greater than” or “implies” are transitive. Moreover, it reduces the communication required between artifacts in comparison to flooding, as no self-descriptions are routed through the network of artifacts and local self-descriptions are broadcast upon changed relations only (cf. Section 4.4). On the other hand, a major disadvantage is that just a subset of the relations which hold between artifacts can be detected compared with an availability of quantitative sensor data between all pairs of artifacts, which is for two reasons. First, many relations – such as intrinsic orientation and static distance (cf. Section 4.1.1) – are not transitive, and thus no transitive closure can be processed for them. Second, processing the transitive closure requires that all three relations – i.e. the two composed as well as the resulting one – have the same value, for which reason e.g. no front or back relation can be inferred between the artifacts b and d in our example.
4.3.2 Distribution Algorithm Using Compositional Inference Although the distribution approach described above allows for reasoning about qualitative spatial relations under certain circumstances, we argued that it is quite limited as it works for the same relation values at a time only and many relations are not transitive. As stated in Section 4.1.2, transitivity can be seen as a special case of composition where the two composed as well as the resulting relation are the same. As a logical next step, we thus replaced the reasoning step with compositional inference using pre-defined composition tables such as those shown in Section 4.2.2, which often allow for inferring more accurate results. Given appropriate tables, it is possible to infer new from two existing relations which (i) need not have the same value (e.g., the orientation relations front and right can be composed) and (ii) need not be transitive (e.g. a composition of distance relations is possible). Moreover, composition allows for reasoning about compound relations,
4.3 Spatial Relationship Inference and Distribution
93
which means that uncertainty can be taken into account which is also not the case if just the transitivity property is exploited. Algorithm 1 Operations of artifact x upon receiving the self-description of y. 1: if self-description of y received the first time then 2: recognize qualitative relation of y to x and put it to self-description of x 3: end if 4: R ← get qualitative relation of y to x from self-description of x 5: for all artifacts z which are in relation to y do 6: if z = x then 7: S ← get qualitative relation of z to y from self-description of y 8: RS ← perform composition R ◦ S 9: if self-description of x already contains relation of z to x then 10: intersect known relation with composition result RS 11: else 12: put composition result RS to self-description of x 13: end if 14: end if 15: end for 16: if first self-description received or relations in self-description of x changed then 17: broadcast self-description of x 18: end if According to the approach for inferring and distributing relations described in Section 4.3.1, we have developed an algorithm and implemented it for the purpose of evaluation (cf. Section 4.4). Algorithm 1, which has been published in [Hol07a], describes the operations an artifact x performs upon receiving the self-description of artifact y. At the first time artifact x receives the self-description of y during the same process of inferring and distributing relations, which is determined by a unique sequence number contained in the self-description of y, it recognizes the relation R(y, x) and puts it to its own self-description. Afterwards, artifact x infers relations RS by composing the relation R with each relation S contained in the received self-description, whereas R and S can be compound relations. The composition operation is thus repeatedly applied to triples of artifacts, and the results are again added to the self-description of x. If the composed relations R and S are compound ones, the composition result RS is the union of the compositions of base relations (cf. Section 4.2.2). In order to retain the most accurate result, RS is eventually intersected with relations to
94
4 Distributed Spatial Reasoning
Figure 4.5: Recognition, composition and intersection of qualitative positional relations.
the respective artifact that are possibly known by x already (i.e. contained in its self-description); however, it should be noted that intersection leads to meaningful results for JEPD relations only (cf. Section 3.2.3). In the case that no further relations can be inferred, no more self-descriptions are being broadcast and the distribution process terminates. The broadcast can be delayed for a certain amount of time such that it is not performed separately for each received self-description, but only once for several changes, which reduces the induced traffic as will be seen in Section 4.4. It should be noted that local communication policies of artifacts, which concern privacy and security aspects as discussed in [MOFH03] for example, may have an impact on the recognition and inference of relations but are not considered in this work. Figure 4.5 shows the algorithmic steps recognition, composition and intersection by means of a simple topology with four artifacts a, . . . d which are placed in
4.3 Spatial Relationship Inference and Distribution
95
two-dimensional space. Each vertex represents a digital artifact, edges between two vertices indicate that the respective artifacts are within communication range, and arrows represent broadcasts of self-descriptions. For the composition operation, a combined consideration of orientation and distance relations using the composition table of Table 4.5 is used. First, after an initial broadcast of artifact d, the artifacts a and c receive the self-description of d and recognize qualitative positional relations by comparing their own position with that contained in the received self-description. For example, artifact c recognizes that d is front and near, which results in the relation front(d,c) ∧ near(d,c). Subsequent broadcasts of the self-descriptions of a and c enable b and d to recognize their relations to the broadcasting artifacts also, as for example that c is right and near to b. Afterwards, in the second step of the algorithm, artifact b infers additional relations from the received and previously recognized ones. For example, b composes the relations front(d,c) ∧ near(d,c) (i.e. received from artifact c) and right(c,b) ∧ near(c,b) (i.e. recognized by a comparison with the position of artifact c), which results in a disjunction of the four relations front(d,b) ∧ near(d,b), front(d,b) ∧ medium-dist(d,b), right(d,b) ∧ near(d,b) and right(d,b) ∧ medium-dist(d,b). Another composition artifact b performs concerns the positional relations between d and a as well as those between a and b, which results in a further composed relation between d and b (cf. Figure 4.5). Artifact b finally intersects the two composition results, which leads to a disjunction of just two relations, namely front(d,b) ∧ near(d,b) and front(d,b) ∧ medium-dist(d,b). We distinguish three variants of the distribution algorithm, which differ in how the composition operation is implemented. The first one, which corresponds to the pseudocode of Algorithm 1 and has been used for the example in Figure 4.5, is a combined composition of positional relations. It produces the best results of all variants, namely those which constrain the resulting compound relations most, as will be shown by simulation in the next section. In the second variant, which is a small modification of the above algorithm, the composition of orientation and distance relations is performed separately by using dedicated composition tables for orientation (cf. Table 4.3) and distance (cf. Table 4.4). As we have discussed in Section 4.2.2 already, using separate composition tables for orientation and distance often leads to comparatively coarse results, in particular with regard to distance relations. On the other hand, the composition tables are much easier to create and smaller than in the combined case. The third and last variant processes the transitive closure of orientation relations, which means with respect to Algorithm 1 that just orientation relations are considered for the inference, and that the composed relations R and S are not compound relations and do have the same relation value at a time. This variant requires the least administrative and computing
96
4 Distributed Spatial Reasoning
efforts, but – for reasons mentioned in the previous section – provides the coarsest results. A comparison of these three variants and the flooding approach with respect to their accuracy and the induced traffic by means of different network topologies will be given in the following section. An early algorithm from the research field of qualitative spatial representation and reasoning, which has been introduced in [All83], computes the closure of a set of temporal interval relations among nodes every time a new relation is inserted; a detailed explanation of a corrected version of this algorithm can be found in [Her94]. It first intersects a possibly already existing with the new relation between two nodes i and j, and then computes the closure by propagating the effect of the new relation in the case it has changed. For this purpose, it checks whether the new relation can be used to constrain (i.e. to make them more specific) the relations between i and each other “comparable” node k as well as between each other node k and j by using composition table lookups, which in turn may lead to changed relations that are to be propagated; in this regard, propagation can be controlled by defining which nodes are “comparable” and which ones are not. The main difference to the algorithm proposed in this thesis is that the maintenance and inference of relations is centralized and not distributed across spontaneously interacting artifacts, which however is fundamental for their autonomous operation as we have argued in Section 2.2. Therefore, the recognition and maintenance of relations as well as the inference of new ones is conducted by the artifacts independently from each other, and it is solely based on their local perceptions and the exchange of them with other artifacts in communication range. According to the analysis of the closure algorithm presented in [All83], the upper bound of the operation time of our algorithm is also O(n2 ) for a fixed number of n artifacts, as at most n ∗ (n − 1) compound relations can hold between them, and each one consists of up to 12 positional base relations which cause a constant maximum number of 12 self-description changes per artifact.
4.4 Simulation Results The algorithm presented in Section 4.3 is an efficient means for providing spatially distributed digital artifacts with an awareness about qualitative spatial relations among each other. The key contribution in this regard is the inference of relations over multiple hops without the need for numeric calculations, given the availability of appropriate composition tables which allow for resource-efficient reasoning with simple table-lookups. In order to evaluate the algorithm and its implementation variants, we simulated it with different network topologies and
4.4 Simulation Results
97
varying numbers of artifacts by using the J-Sim simulation environment [Uni04]. In J-Sim, artifacts are implemented in Java as individual software components which communicate with each other via ports. This suits our work on autonomous digital artifacts well, which interact with their local neighborhood by exchanging self-descriptions. The Tcl script language is used in J-Sim for composing the components and thus defining the overall system behavior. The evaluation was done without taking into account certain wireless communication technologies or transmission protocols, just the logics of the algorithm have been simulated. The aim was to compare the implementation variants of our algorithm and the flooding approach both with regard to (i) their accuracy in terms of acquired qualitative relations after termination of the distribution process and (ii) the induced traffic which is generated by broadcasts of self-descriptions. We have implemented (i) the flooding variant, as well as the proposed algorithm with its three ways for composing spatial relations (cf. Section 4.3): (ii) transitive closure, (iii) separate composition and (iv) combined composition. Each one of the composed J-Sim components, which represents a digital artifact executing one of these algorithms, is able to recognize and infer qualitative relations to others, put the results to its self-description and exchange it with others via the communication ports. In this regard, the three composition tables have been fully implemented in order to allow for simulating arbitrary topologies. We only used positional (i.e. orientation and distance) relations, the algorithm is however applicable for other relation types also as long as composition tables are available or the relation properties are known. First, we investigated the spatial awareness which is achieved by artifacts during a whole relationship distribution process. We used the example topology with n = 5 artifacts shown in Figure 4.6 for it, and visualized the whole process step-by-step for the case that artifact b initially broadcasts its self-description and that combined composition is used. In the simulation, the five artifacts are independently executed components, which are connected via their ports as indicated with edges between vertices in the figure. The achieved spatial awareness for this topology after termination of the relationship distribution can be seen in Table 4.6, depending on the algorithm used in each case. In the following, we will have a closer look at the simulation results. With the flooding approach, each artifact gets to know the self-descriptions of all others in the network, and it is possible to compute exactly one base relation from the sensor data of each artifact. Flooding is thus the most accurate variant, resulting in n ∗ (n − 1) = 20 relations (i.e. 4 base relations per artifact), and it causes a total number of 50 received self-descriptions over all n artifacts. The transitive closure algorithm on the other hand is the least accurate one, mainly for two reasons. First, all three relations have to be the same for processing the transitive closure,
98
4 Distributed Spatial Reasoning
Figure 4.6: Compositional inference and distribution started at artifact b.
Artifact b R(a,b): medium-dist ∧ left R(c,b): near ∧ back R(d,b): near ∧ right R(e,b): medium-dist ∧ right R(a,b): medium-dist ∧ left R(c,b): near ∧ back R(d,b): near ∧ right R(e,b): right R(a,b): medium-dist ∧ left R(c,b): near ∧ back R(d,b): near ∧ right R(e,b): (near ∨ medium-dist) ∧ right R(a,b): medium-dist ∧ left R(c,b): near ∧ back R(d,b): near ∧ right R(e,b): (near∧right) ∨ (medium-dist∧right)
Artifact a
R(b,a): medium-dist ∧ right R(c,a): medium-dist ∧ right R(d,a): far ∧ right R(e,a): far ∧ right
R(b,a): medium-dist ∧ right R(c,a): – R(d,a): right R(e,a): right
R(b,a): medium-dist ∧ right R(c,a): right ∨ back R(d,a): right R(e,a): right
R(b,a): medium-dist ∧ right R(c,a): (near∧right) ∨ (medium-dist∧right) ∨ (far∧right) ∨ (near ∧ back) ∨ (medium-dist ∧ back) ∨ (far∧back) R(d,a): (medium-dist ∧ right) ∨ (far∧right) R(e,a): (medium-dist ∧ right) ∨ (far∧right)
Algorithm
Flooding
Transitive Closure
Separate Composition
Combined Composition R(a,c): (near∧left) ∨ (medium-dist∧left) ∨ (far∧left) ∨ (near ∧ front) ∨ (medium-dist ∧ front) ∨ (far ∧ front) R(b,c): near ∧ front R(d,c): near ∧ right R(e,c): (near∧right) ∨ (medium-dist∧right)
R(a,c): left ∨ front R(b,c): near ∧ front R(d,c): near ∧ right R(e,c): (near ∨ medium-dist) ∧ right
R(a,c): – R(b,c): near ∧ front R(d,c): near ∧ right R(e,c): right
R(a,c): medium-dist ∧ left R(b,c): near ∧ front R(d,c): near ∧ right R(e,c): medium-dist ∧ right
Artifact c
R(a,d): (medium-dist ∧ left) ∨ (far∧left) R(b,d): near ∧ left R(c,d): near ∧ left R(e,d): near ∧ right
R(a,d): left R(b,d): near ∧ left R(c,d): near ∧ left R(e,d): near ∧ right
R(a,d): left R(b,d): near ∧ left R(c,d): near ∧ left R(e,d): near ∧ right
R(a,d): far ∧ left R(b,d): near ∧ left R(c,d): near ∧ left R(e,d): near ∧ right
Artifact d
R(a,e): (medium-dist ∧ left) ∨ (far∧left) R(b,e): (near∧left) ∨ (medium-dist∧left) R(c,e): (near∧left) ∨ (medium-dist∧left) R(d,e): near ∧ left
R(a,e): left R(b,e): (near ∨ medium-dist) ∧ left R(c,e): (near ∨ medium-dist) ∧ left R(d,e): near ∧ left
R(a,e): left R(b,e): left R(c,e): left R(d,e): near ∧ left
R(a,e): far ∧ left R(b,e): medium-dist ∧ left R(c,e): medium-dist ∧ left R(d,e): near ∧ left
Artifact e
4.4 Simulation Results 99
Table 4.6: Simulation results for the topology of Figure 4.6.
100
4 Distributed Spatial Reasoning
for which reason e.g. artifact a is not able to infer any relation to artifact c and vice versa, as the orientation on the path from a to c changes (i.e. right(b,a) and back(c,b)). Second, the spatial relations have to be transitive, which is not the case for distances; thus, no distance relation can be inferred from the artifacts c, d and e to a as well as from e to b and c. It results in 58 base relations, as a missing distance or orientation relation corresponds to 3 or 4 base relations (i.e. the respective universal relations, cf. Section 4.1), and no positional relation at all (e.g. from c to a) corresponds to their product with 12 alternative positional relations. The simulation also showed that a traffic of 36 received self-descriptions is induced over all artifacts. With delayed broadcasts, which avoids multiple broadcasts due to successively received self-descriptions by performing them in short fixed intervals only, it could be reduced to just 25 self-descriptions, which is 50% compared with flooding. In both cases, the simulation was started once from each artifact, and the average of the resulting traffic was taken. Additionally, we investigated the relative accuracy achieved by a certain algorithm, which we define as the complement of the ratio between the (i) difference of the actual (i.e. 58) and the minimum (i.e. 5 ∗ 4 = 20) number of relations, and (ii) the difference of the maximum (i.e. 5 ∗ 4 ∗ 12 = 240) and the minimum number of relations. For the transitive closure algorithm, this is 1 − (58 − 20)/(240 − 20) = 83% compared with flooding. Taking into account the separate composition of orientation and distance using the composition Tables 4.3 and 4.4, more accurate conclusions can be drawn. For example, artifact a is now able to infer the two alternatively possible base relations right(c,a) and back(c,a), and artifact c is able to narrow down the possible distance relations to artifact e to near(e,c) and medium-dist(e,c). The implementation variant with separate composition results in a total number of 42 relations and induces a traffic of 38 self-descriptions without and 26 by using delayed broadcasts. The relative accuracy raises to 90% and the traffic for delayed broadcasts to 52%. Nevertheless, the composition of distances often results in the universal relation, which does not provide any information about the spatial distance between artifacts. With a combined composition of orientation and distance relations as shown in Table 4.5 however, the accuracy of the relations between some artifacts can be increased further. For example, while artifact a is only able to infer the relation right(e,a) due to the resulting universal relation by composing the distance relations mediumdist(b,a), near(d,b) and near(e,d) without taking into account orientations, their combined consideration allows to exclude the distance relation near(e,a). The total number of relations can eventually be refined to 38, the induced traffic is the same with 38 and 26 self-descriptions, respectively. The accuracy thus raises to
4.4 Simulation Results
101
92% and the traffic remains 52%, which means that a higher accuracy is achieved with the same traffic. We simulated the four implementation variants from above with the common network topologies full binary tree, line, mesh and ring, and for varying numbers of artifacts which are positioned as shown in Figure 4.7. The induced traffic for the different topologies and depending on the number of nodes can be seen in Figure 4.8, whereas several observations can be made. First, it can be seen that it is in any case reduced by using qualitative composition in comparison to flooding. Second, the transitive closure algorithm is always equal or below the traffic induced by a separate or combined composition of positional relations, which is due to the smaller number of inferred relations and thus the fewer broadcasts. Third, the induced traffic for separate and combined composition is quite different for the binary tree topology, whereas it is virtually the same for line, mesh and ring topologies. Fourth, using delayed broadcasts significantly reduces the traffic with an increasing number of nodes, for example to less than 40% for the mesh topology with 15 and the binary tree topology with 16 nodes. We also experimented with complete graphs and star topologies which lead to similar results.
Figure 4.7: Simulated topologies full binary tree, line, mesh and ring.
From the above evaluation it can be seen how many broadcasts are caused by the different algorithm implementations, especially in comparison with the flooding approach. In order to provide a group of connected artifacts with an awareness about their qualitative spatial relations among each other, flooding requires the highest amount of communication. However, by relying on qualitative inference between artifacts which are not directly connected, not all relations between them can be stated with certainty; instead, usually disjunctions of spatial relations are acquired between distant artifacts. In the following, the relative accuracies – i.e. with respect to the flooding approach – for the four topologies are finally shown in Figure 4.9, from which several things can be seen. As in the above example, 100% accuracy means that each artifact knows the exact base relations to all other artifacts, 0% means that just universal relations are known (i.e. all 12 positional
102
4 Distributed Spatial Reasoning
Figure 4.8: Traffic induced by different algorithms depending on the topology.
4.5 Findings and Discussion
103
Figure 4.9: Relative accuracy of different algorithms compared with flooding.
relations are possible between any pair of artifacts) and 50% means that just half of the alternative base relations which may exist between artifacts are possible and the others can be excluded. First, the transitive closure approach leads to the least accuracy, which is clear as it only supports a subset of the possible compositions. Second, the accuracy decreases with an increasing number of nodes, which is due to the coarser composition results coming along with the higher number of hops between artifacts. Third, the percentage-wise reduction of traffic is in all simulated scenarios higher than the percentage-wise loss of accuracy; for example, the induced traffic for the mesh-topology with 16 nodes drops to 27% in the case of separate composition, whereas the accuracy is reduced to 73% only.
4.5 Findings and Discussion In this chapter, we first discussed fundamentals of qualitative reasoning about binary spatial relations, and presented means for inferring new from known positional relations by (i) exploiting the properties reflexivity, symmetry and transitivity, as well as by using composition tables for (ii) a separate or (iii) a combined consideration of orientation and distance relations. As for many relation types none of the three properties is fulfilled and closures can be processed for equal relation values only, our focus was on compositional reasoning which allows for overcoming these restrictions. We proposed an algorithm for inferring and distributing relations among autonomously operating digital artifacts over multiple hops by taking advantage of those qualitative reasoning techniques, namely without numeric calculations or the need for exchanging spatial contexts between artifacts that are out of communication range. The aim was to provide a group
104
4 Distributed Spatial Reasoning
of artifacts – which are either connected directly or via other artifacts – with an awareness about their relations to each other; therefore, self-descriptions containing both quantitative spatial properties and qualitative spatial relations are shared between neighboring artifacts. In this regard, our focus was on static positional relations in 2-D between two stationary artifacts at a time, which are given with respect to an extrinsic reference frame. It should be noted that also other than positional relations could be used, given that the respective composition tables are available. The presented approach allows for resource-efficient reasoning with simple table lookups, and is thus – among other reasons for using qualitative relations mentioned in the previous chapter – valuable whenever relations between artifacts out of sensor- and communication-range have to be acquired, or the artifacts’ resources – in particular their computing and communication capabilities – are limited. However, a mentionable disadvantage is the time-consuming construction of composition tables as well as the necessity of their reconstruction upon changes in the number or semantics of the corresponding relations. We have implemented the algorithm variants depicted at the beginning of this subsection, and evaluated them in a Java-based simulation environment for different topologies and numbers of networked artifacts. Wireless communication aspects such as delays or packet losses as well as conflicting or inaccurate sensor data have been disregarded in the simulation, which we however consider interesting with respect to future work. Another issue of future work would be to extend the presented approach to moving artifacts with constantly changing relations between them, which is not covered in this thesis but can be important with regard to dynamic real-world scenarios. The simulation of our algorithm showed that the traffic induced by broadcast self-descriptions is significantly reduced compared to an exchange of self-descriptions between each pair of artifacts by means of flooding. However, especially between distant artifacts which are several hops away from each other, often just coarse results in the sense of alternatively possible relations can be acquired, which is also reflected by a decrease of the overall accuracy with an increasing number of artifacts in the simulation. In this regard, a combined composition of distance and orientation leads to more accurate results than their separate composition, which in turn provides more accurate results than exploiting just the transitivity property of binary relations. It should be noted that the proposed approach of inferring relations by utilizing qualitative spatial reasoning techniques is not only meaningful for acquiring relations among a group of connected artifacts as presented in this chapter, but also for “en passant” inference, meaning that an artifact which is already aware of its relations to others around shares this information with artifacts passing by.
5 Rule-Based Spatial Awareness All our knowledge begins with the senses, proceeds then to the understanding, and ends with reason. There is nothing higher than reason. Immanuel Kant, 1724-1804
In the preceding chapters, qualitative abstractions of spatial and spatiotemporal relations, as well as the inference of new relations from them by means of compositional reasoning, have been discussed within the scope of spontaneously interacting digital artifacts. They are able to acquire spatial properties characterizing their situation on the one hand and recognize relations by a pairwise comparison of their own with the spatial properties of artifacts in range on the other hand, which are – together with recognized and inferred relations – shared among them by using ad-hoc communication. However, what we have not considered by now are programmatic means for achieving spatially aware behavior of artifacts. Essential in this respect is their ability to reason about the history of qualitative relations they are aware of, for which purpose a rule-based solution is proposed in this chapter. First, in Section 5.1, a motivation for and an introduction in the use of rules for spatially aware digital artifacts are given. Afterwards, Section 5.2 presents our approach for rule-based reasoning about qualitative spatial relations, in particular the inference of new relations and the triggering of appropriate application-level actions as a consequence of observing certain defined patterns on the known relations. It builds on qualitative relations that have been recognized and abstracted depending on application demands, and allows for achieving spatially aware behavior using declarative programming. As will be seen in Chapter 6, rule-based reasoning constitutes an integral component of the Zones-of-Influence Framework; in this regard, it serves as a layer of abstraction between the low-level relationship recognition and the application programming layer. Finally, a proof-of-concept evaluation with a motion tracking system is provided in Section 5.3, and Section 5.4 summarizes the proposed approach and points out open issues for future work.
106
5 Rule-Based Spatial Awareness
5.1 Achieving Spatially Aware Behavior The focus of the present thesis is on the spatial awareness of autonomous embedded systems that are integrated into everyday physical artifacts, so as to be able to develop spatially aware applications which rely on the artifacts’ spatial relations to one another as well as on relationship changes over time. We have presented an architecture for spatial awareness in Section 2.5, whose core responsibilities are the recognition of spatial relations to artifacts in communication range and their qualitative abstraction for the application-level use, depending on application demands and the availability of required spatial information. The representation of qualitative static and dynamic spatial relations on the one hand, and their combined consideration with qualitative temporal relations between intervals in which they hold on the other hand, have been extensively discussed in Section 3.2. For achieving spatially aware behavior of digital artifacts, which means in particular that they are able to determine and use spatial relations among them, an important issue is their ability to reason – i.e. to draw conclusions – about those relations and their changes over time. It allows to make knowledge explicit for applications which is implicitly contained in the known relations and their semantics only, and thus contributes to the autonomous adaptation of artifacts to their spatial environment. In Chapter 4, we were concerned with compositional reasoning, which commonly uses predefined composition tables for inferring new from known relations. Although such a reasoning technique contributes to the development of spatially aware applications in that the number of relations which can be used is extended, it does not provide programmatic means for achieving spatially aware behavior. Hence, the upcoming issue is to find an appropriate software solution which supports programmers in defining the actions to be executed for certain combinations of relations. In this regard, our scope is on reactive systems whose output depends on the current and past states; proactive behavior as e.g. addressed in [May04] is not considered.
5.1.1 Reasoning with Rules In general, two programming paradigms can be distinguished: imperative and declarative programming [BIK+ 04]. While the former is concerned with how to do something, the latter allows for describing what to do. We decided for a declarative approach using rules in this thesis, which is for several reasons. First, a rulebased approach provides a higher degree of flexibility with regard to customizing the behavior of spontaneously interacting digital artifacts [BCFM03, TTH+ 04], as both the rules and the data on which they operate can change at runtime. The sec-
5.1 Achieving Spatially Aware Behavior
107
ond reason is that the utilization of rules allows for a more natural and intuitive modelling of the behavior of networked mobile devices, and is thus also suitable for non-technical users and domain experts [BCFM03, HR85]. A third reason for choosing a rule-based approach is the comparatively high speed and scalability of some rule engines, especially when the data does not change entirely between rule executions [PNF+ 08]; due to their abstraction from the more fluctuating lowlevel sensor readings, this is usually the case for qualitatively represented spatial relations. In the field of pervasive computing, rules have been used for behavior modelling in many projects. For example, [SAW94] distinguishes context-triggered actions as one of four categories of context-aware applications, and uses if-then rules to specify in the “contextual reminders” application which message should appear depending on different contexts such as time, location and the identity of people. Another example is given in [BGS01], where rules are applied to a history of contexts acquired from temperature and movement sensors in order to detect high-level situations such as “drinking out of the cup”. [BCFM03] and [TTH+ 04] use a rule-based approach for programming the behavior of mobile systems in a more natural way, and [SGKK04] takes advantage of rules both for inferring further based on existing knowledge – which can be built-in domain knowledge, observed or previously inferred knowledge – as well as for determining an artifact’s behavior. According to [BBB+ 07], two types of rules for programming reactive systems can be distinguished. The first one are ECA (Event-Condition-Action) rules, which have the structure on <event> if then and thus define which action (also referred to as consequence) is executed when the specified event occurs and the condition holds. The second type are production rules, which have the structure when then and specify the action to be executed when the condition becomes true due to an update on the database. Instead of events in the case of ECA rules, which makes them particularly suitable for developing distributed applications (like e.g. in [BCFM03, TTH+ 04]), production rule languages have the concept of a working memory containing data and an update statement for explicitly notifying about changes in data, which makes them the preferred choice for managing the state of a system and defining its behavior based on domain-specific knowledge [BBB+ 07]. As the focus of this work depicted in Section 1.1 is on making digital artifacts spatially aware about their environment, which means that they have to be able to maintain a spatial model, draw conclusions about it and adapt their behavior accordingly, we have chosen to use production rules and refer to them simply as rules in the following.
108
5 Rule-Based Spatial Awareness
Figure 5.1: Rule-based reasoning [PNF+ 08].
With regard to this thesis, the data are qualitative spatial or spatiotemporal relations, and actions which result from fulfilled conditions on that data are primarily the inference of new high-level relations or the triggering of application-level actions. This allows applications to adapt to their spatial environment and thus exhibit spatially aware behavior, which is presented and evaluated in the Sections 5.2 and 5.3, respectively. In order to be able to use production rules however, a rule engine software is required, which matches data against rules and infers conclusions that result in corresponding pre-defined actions. For a better understanding of rule-based spatial awareness discussed in detail afterwards, the next subsection gives a short overview on the fundamentals of rule engines.
5.1.2 Using a Rule Engine Due to our choice for production rules, we use a so-called production rule system for reasoning about spatial relations, which is often simply referred to as rule engine. As can be seen in Figure 5.1, it basically consists of a knowledge base which contains both rules (i.e. the logic) and facts (i.e. the data), and an inference engine that infers conclusions from them which result in certain actions [HR85]. The rules are constructed by a human domain expert and stored in the production memory of the rule engine, and facts are inserted in the working memory where they can be retracted (i.e. removed) or modified [PNF+ 08]. Rules consist of a conditional part which specifies patterns on the data using first order logic, as well as a consequence part which defines one or more actions that are executed when the condition is fulfilled; in that case, the rule is said to be fired.
5.2 Rule-Based Qualitative Spatial Reasoning
109
The process of matching facts against rules is called pattern matching, and it is performed by the inference engine which can be regarded as the core component of the rule engine. Facts are evaluated for matches against rules upon insertion in the working memory, and added to the so-called agenda if they match; the actual execution of the rules however, which is performed for each combination of facts for which they are fulfilled, has to be triggered explicitly. The agenda is responsible for resolving conflicts in the case that multiple rules are scheduled for execution; for this purpose, it defines an execution order of the rules, which may modify the working memory and thus possibly cause some other rules to be removed from the agenda for example. An important aspect of rule-based systems is that rules can be changed without recompilation of program code, which allows for adapting the artifact behavior at runtime. Within the scope of this thesis we decided for using the Java-based rule engine JBoss Drools 4.0 [PNF+ 08], whereby the facts stored in the working memory are Java objects representing qualitative relations with their intervals. It uses an extension of the Rete algorithm for pattern matching, which minimizes the number of patterns to be evaluated by compiling the rule conditions into a network [BBB+ 07], and thus scales to large numbers of rules and facts.
5.2 Rule-Based Qualitative Spatial Reasoning In Section 5.1, we have pointed out the importance of reasoning about qualitative spatial relations over time in order to achieve spatially aware behavior, and argued for the use of production rules which provide declarative means for defining the actions to be executed for certain combinations of relations. The proposed rule-based approach we have published in [Hol07b] builds on a repository of qualitatively abstracted relations (i.e. the rule engine’s working memory), which have been – according to the architecture for spatial awareness presented in Section 2.5 – either (i) recognized by a pairwise comparison of Zones-of-Influence, (ii) received with the self-description of a digital artifact in range or (iii) inferred from already known relations, and it serves as a layer of abstraction for the development of spatially aware applications. As rules are applied to the relations repository for detecting arbitrary patterns that can be formulated with first order logic, and executing any actions when the specified patterns have been found, they can serve a number of different purposes. For achieving spatial awareness, we distinguish the following two types of rules: • Maintenance rules: perform maintenance operations on the repository, which can result in modifications or retractions of relations
110
5 Rule-Based Spatial Awareness
• Inference rules: infer relationships or application-level actions depending on specified patterns of relations observed in the repository In the following, these two types of rules and their use with the rule engine are presented, whereas a detailed discussion on inference rules – which actually define an artifact’s spatially aware behavior – will be given in Section 5.2.1. An important aspect that has to be considered first however is the timing concerning (i) the recognition of relations, (ii) an update of the repository therewith and (iii) the subsequent execution of rules. The starting point is the recognition of relations, for which we distinguish two variants: • Periodic recognition of all relations, for which reason each Zone-of-Influence of an artifact has to be compared with all other ZoIs it is aware of • Event-based recognition for each changed or new Zone-of-Influence, which has to be done just for these ZoIs upon notification Afterwards, the repository is updated with the recognized relations, which can be a modification of the time intervals of existing or an insertion of newly recognized relations. The actual execution of rules – i.e. of the actions in the consequence parts of rules that match – has to be triggered explicitly, which is done immediately after updating the repository. For the purpose of a first feasibility study of our rule-based approach by using two Zones-of-Influence in Section 5.3, all relations are periodically recognized and inserted in the repository, whereas a relation that is recognized at time t is inserted with the interval [t,t] – meaning that it holds at time t – in the repository; all rules are fired afterwards. With this approach, previously received Zones-of-Influence or recognized relations do not have to be considered for the insertion. In order to allow for temporal reasoning, namely to relate the time intervals of relations – e.g. in order to recognize that one relation holds during another one – as discussed in Section 3.2.5, it is necessary that equal relations at successive discrete points in time (e.g. three relations with the intervals [t,t], [t + 1,t + 2] and [t + 3,t + 3]) are merged to a single relation with a correspondingly extended time interval (i.e. [t,t + 3] in this example). It should be noted that this is based on the assumption that the respective relation also holds between those points in time where no information is available. A maintenance rule is used for merging such relations which belong together. It checks the repository for pairs of qualitative relations with the same name, value and identifications of the reference and primary Zone-of-Influence, whose intervals are not in an after or before relation to each other (cf. Figure 3.11), as just relations that either meet or overlap to a certain extent have to be merged. For each such pair, it replaces the interval of one relation with the union of the two intervals
5.2 Rule-Based Qualitative Spatial Reasoning
111
(i.e. its interval is extended) and removes the other relation from the repository. For the above example of a relation inserted at time t, this means that if there already exists an equal relation at the preceding point in time, its interval is extended to the end-point t and the other relation is removed. Another maintenance rule could be defined for limiting the length of the relations history for example, simply by removing relations whose end times range too far (i.e. greater than a defined number of discrete points in time) in the past. Listing 5.1: Maintenance rule for merging equal relations. rule " merge - equal - relations " when r1:Relation ( n:name , v:value , p Z:pZoI , rZ:rZoI , i1:interval ) r2:Relation ( name == n , value == v , pZoI == pZ , rZoI == rZ , i2:interval ) eval (! r1 . equals ( r2 )) // r1 and r2 are different relations eval (! i1 . before ( i2 ) && ! i1 . after ( i2 )) // i1 not after or before i2 then r1 . setInterval ( i1 . merge ( i2 )); up date ( r1 ); // extend interval of r1 retract ( r2 ); // remove relation r2 from the repository
Listing 5.1 shows the maintenance rule for merging equal relations, which is given in the syntax of the Drools rule engine [PNF+ 08]; however, it should be noted that other production rule systems could also be used. The rule’s condition (when clause) specifies a pattern on the stored relations; in this example, it is an implicit and combination of two qualitative relations as well as two eval statements containing Java code that returns a boolean value. Relation is a Java class, whose member variables (i.e. referred to as fields in [PNF+ 08]) – in this case the name and value of the relation, its primary and reference ZoI as well as the time interval in which the relation holds – can be used for constraints on field names of objects of that type. Two relation objects that match the given field constraints are bound to the variables r1 and r2, which are used in the first eval statement to ensure that different objects are compared. The fields of the first relation are bound to variables (e.g. v for the relation value) and used to constrain the value of the other relation (i.e. value==v in this example); thus, r1 and r2 must have the same name, value and ZoI identifications. The second eval statement limits a match to those relation pairs whose intervals are not in a temporal after or before relation. For each combination of Relation facts for which the condition – i.e. all four conditional elements – is fulfilled, the rule’s consequence (then clause) is executed. In the example of Listing 5.1, it extends the interval of relation r1 by setting it to the minimum start time and the maximum end time of the intervals i1 and i2 (method merge), and notifies the rule engine about this change with a subsequent update statement. Finally, r2 is retracted from the repository. Similar
112
5 Rule-Based Spatial Awareness
to the rule shown in Listing 5.1, other maintenance but also inference rules can be described. An important aspect is that the time required for pattern matching depends on how often facts change, as it is performed by the rule engine every time a new fact is inserted, modified or retracted. In the above example of a periodic recognition, all relations are recognized and inserted at each point in time. For the implementation of the Zones-of-Influence Framework discussed in Chapter 6 however, we used the more efficient event-based variant, where relationship recognition is performed just for changed and new Zones-of-Influence. Also the repository is updated with changed and new qualitative relations only, which together reduces the number of inserted and modified relation facts. The rule execution is performed immediately after the repository update, that is periodically at discrete points in time. For detecting those changes however, it is necessary to remember the Zonesof-Influence as well as relations previously recognized from them. With regard to time efficiency, it should be noted that the rule engine remembers past matches for facts that have not changed [PNF+ 08].
5.2.1 Inferring Relations and Application-Level Actions The rule-based approach described above is primarily used for inferring conclusions from spatial relations, which allows for defining the behavior of artifacts depending on their spatial environment. This comprises (i) the inference of relations based on known ones, which extends an artifact’s spatial awareness by making application-relevant knowledge available that is implicitly contained in the relations, as well as (ii) the inference of application-level actions, which are executed once the respective rule’s condition is fulfilled; a similar distinction can be found in [SGKK04]. We use so-called inference rules in both cases, which specify patterns on qualitative relations in their conditional parts. The basis for the first case – namely the inference of relations – is the representation of spatiotemporal relations presented in Section 3.2.5, where we discussed logical combinations of qualitative spatial and temporal relations as well as their interpretation as high-level spatiotemporal relations which exist in corresponding time intervals; in this regard, a distinct feature of our work is that such inferred qualitative relations can in turn be used like any other relation in the repository. We distinguish four general categories of inferred relations, depending on (i) if static or dynamic relations are combined and (ii) whether the represented situation relates to a single point in time or a time series of changing spatial relations. A classification of these four categories is given in Table 5.1 with two examples
5.2 Rule-Based Qualitative Spatial Reasoning
113
Table 5.1: Combinations of qualitative spatial and temporal relations. dynamic spatial relations
point in time
static spatial relations
oriented toward each other
approaching from the right
turning away to the right
moving around counterclockwise
passing by right
changing direction of movement
turning around for a while
time series
near and on the left hand side
at a time, showing situations of spatial relations between two Zones-of-Influence which are denoted as primary object p and reference object r in the following. First, if just spatial and no temporal relations are combined, the spatial situation at a certain point in time can be recognized. The combination in the respective inference rule’s condition could be e.g. front(p,r) ∧ opposite-dir(p,r) for the static spatial situation “oriented toward each other”, and leftward(p,r) ∧ toward(p,r) for the dynamic situation “approaching from the right”. As spatial relations are internally stored with time intervals, an additional combination of the relations’ intervals has to be considered, namely that they are not in a temporal after, before, meets or met-by relation to each other. Second, if a time series of changing relations is considered, both spatial as well as temporal relations between them have to be combined, where the respective combinations are for example back(p,r)[s1 ,e1 ] ∧ right(p,r)[s2 ,e2 ] ∧ front(p,r)[s3 ,e3 ] ∧ meets([s1 ,e1 ], [s2 ,e2 ]) ∧ meets([s2 ,e2 ], [s3 ,e3 ]) for the static situation “passing by right”, and away(p,r)[s1 ,e1 ] ∧ toward(p,r)[s2 ,e2 ] ∧ meets([s1 ,e1 ], [s2 ,e2 ]) for the dynamic situation “changing direction of movement”. In this regard, the subscripts of the primary object p in Table 5.1 indicate the discrete points in time with the respective spatial position or direction. Although these examples are fairly simple, more sophisticated combinations can of course be described. It would also be possible to define quantitative relations be-
114
5 Rule-Based Spatial Awareness
tween the start and end points s1 and e1 of intervals, for example to describe the situation “turning around for a while”, which however is not considered further as our focus is on qualitative temporal relations. In order to infer new from combinations of known relations as described above, the inference rule’s consequence part contains the insertion of the new relation in the repository, with a name and value corresponding to its semantics, an identification of the involved Zones-of-Influence as well as the time interval in which it holds. For the example in Table 5.1, this means that if ZoI p is both in the relations front and opposite-dir to r within a certain period of time, the resulting interval of the new relation oriented-toward-each-other(p,r) is the intersection of the intervals of the two relations due to the logical conjunction of the respective inference rule’s conditions (cf. Section 3.2.5); such new relations can eventually be used in the conditional parts of other rules. Listing 5.2 shows the rule for inferring the relation passing-by-right, which recognizes a sequence of static orientation relations between the same two Zonesof-Influence using temporal meets relations between the intervals in which the relations back, right and front hold. For reasons of simplicity, the relation names (i.e. their types) are disregarded in the following rule conditions, as we assume their values to be unique. A respective new relation is created in the consequence part of the rule and inserted in the repository. In the example of Listing 5.2, the name of the relation is InferredRelation, its value is passing-by-right and the resulting interval is a merge of the intervals i1, i2 and i3 as has been done in Listing 5.1 already. Listing 5.2: Inference rule for the relation passing-by-right. rule " passing - by - right " when Relation ( value == " back " , pZ:pZoI , rZ:rZoI , i1:interval ) Relation ( value == " right " , pZoI == pZ , rZoI == rZ , i2:interval ) Relation ( value == " front " , pZoI == pZ , rZoI == rZ , i3:interval ) eval ( i1 . meets ( i2 ) && i2 . meets ( i3 )) // temporal relations then insert ( new Q ua l it at i ve Re la t io n ( " InferredRelation " , " passing - by - right " , pZ , rZ , i1 . merge ( i2 . merge ( i3 ))));
An inferred can be used like any other relation, as for example in the condition of another rule. This is exemplarily shown with the rule in Listing 5.3, which combines a relation passing-by-right – i.e. generated by the previous rule – with a disjunction of the spatial distance relations near and medium-dist. A new relation passing-by-right-in-vicinity is inserted in the repository, whose interval equals the intersection of the two combined relations’ intervals.
5.3 Proof of Concept
115
In addition, such inference rules can also be used for triggering applicationlevel actions depending on observed patterns in the repository, which are encoded in the consequence part of the rule instead of or in addition to inserting a new relation. Examples for that have been published in [HHF08, HFH+ 08] and will be given in Chapter 7. Listing 5.3: Inference rule for the relation passing-by-right-in-vicinity. rule " passing - by - right - in - vicinity " when Relation ( value == " passing - by - right " , pZ:pZoI , rZ:rZoI , i1:interval ) Relation ( value == " near " || value == " medium - dist " , pZoI == pZ , rZoI == rZ , i2:interval ) eval (! i1 . before ( i2 ) && ! i1 . after ( i2 ) && ! i1 . meets ( i2 ) && ! i1 . metBy ( i2 )) then insert ( new Q u a l i t a t i v e R e l a t i o n ( " I n f e r r e d R e l a t i o n " , " passing - by - right - in - vicinity " , pZ , rZ , i1 . intersect ( i2 ))); end
5.3 Proof of Concept In order to evaluate the rule-based approach for spatial awareness presented in the previous sections in a real-world setting, we have implemented a simple prototype which uses physical sensors for tracking the position and direction of two objects. The aim of this evaluation was to show that (i) different types of static and dynamic qualitative spatial relations as discussed in Section 3.2.3 can reliably be recognized from sensor data, and (ii) that rules are a feasible technique for inferring new qualitative relations from them. The IS-900 motion tracking system from Intersense Inc. [Int08] mentioned in Section 3.1.2 has been used for it, which provides Cartesian coordinates (x,y,z) for position and Euler angles (yaw,pitch,roll) for the direction of tracked devices within a room, both given with respect to the local reference frame of the tracking system. Those trackers contain accelerometers and gyroscopes for calculating position and direction from linear and angular accelerations, and send them – together with signals received from ceiling-mounted ultrasonic strips – to an IS-900 base processor in the background via wireless communication, from which the position and direction of up to four trackers can eventually be accessed. We have chosen the IS-900 system for this evaluation because of the fact that it provides highly accurate spatial information in 6-DOF on the one hand (i.e. 3mm − 5mm for position and 0, 50°−1, 00° for direction), and that the tracked devices can be operated without cabling and thus allow for full freedom of movement on the other hand.
116
5 Rule-Based Spatial Awareness
Figure 5.2: Acquisition of position and direction from two IS-900 trackers.
For recognizing spatial relations over time, the position and direction of a primary object p and a reference object r are given by two such trackers, where each of them is connected to a wireless transmitter as shown in Figure 5.2. They are represented with point-shaped Zones-of-Influence, topological relations are not considered in this evaluation. While object r is placed stationary at the table, p is moved by hand as indicated with the dotted course of motion in the figure. The implemented prototype acquires position (given in meters) and direction (given in a range of ±180°) of the two trackers from the IS-900 processor periodically every 100ms, whereas the mean value of 10 such sensor readings per tracker is used to get smoother and less fluctuating data. For this reason, position and direction acquired from the two trackers are compared once every second, and quantitative static as well as dynamic positional and directional relations are processed and abstracted to qualitative relations. A 2-D projection of the tracked room is used, which means that the z-axis for position as well as pitch and roll for direction are disregarded in the following. It should be noted that the length of the interval between two relationship recognitions depends on the frequency of relation changes which is of relevance for the application. Figure 5.3 shows time series of the three types of quantitative static and dynamic spatial relations over 17 points in time, starting at t = 0 with a recognition interval of Δt = 1s. For this evaluation, dynamic relations are estimated by comparing the corresponding two successive static relation values, for which reason they are recognized first at t = 1 where the quantitative values of t = 0 and t = 1 are compared. First, it can be seen from the figure that the Euclidean static distance relations between p and r are decreasing and increasing again, and the dynamic distance relations – which represent distance changes at the respective points in
5.3 Proof of Concept
117
Figure 5.3: Trajectories of quantitative spatial relations.
time – are consequently negative in the former and positive in the latter case. Second, the static orientation of p with respect to the direction axis of r (i.e. intrinsic orientation) is first negative, which means that p starts its motion left behind r, moves to right behind r from t = 6 to t = 7 then (which results in a jump of the orientation angle) and finally passes r on its right hand side. A dynamic orientation relation represents the direction in which p is moving with respect to r, and it is calculated by determining the orientation of the position of p at the current relative to that at the preceding point in time, again with respect to the intrinsic direction of r. Third and last, static directional relations represent the differences between the direction axes of p and r, which results in positive and negative directional relations if p is directed away and toward r, respectively. The dynamic are simply the differences between two successive static directional relations, thus leading to a trajectory similar to that of the corresponding static relations.
118
5 Rule-Based Spatial Awareness
The recognition of quantitative relations described above is usually the first step followed by their qualitative abstraction, which is considered in the Zones-ofInfluence Framework by allowing qualitative relation recognizers to access results from quantitative ones (cf. Chapter 6). For this evaluation, the quantitative static and dynamic spatial relations are abstracted as shown in the Figures 3.6 and 3.8. The qualitative distances near, medium-dist and far are defined as distances less than 0, 30m, between 0, 30m and 0, 60m and greater than 0, 60m, respectively. In order to represent the absence of motion, the dynamic orientation, distance and directional relations stable-orient, stable-dist and stable-dir are used. The recognized qualitative spatial relations are finally – i.e. once per second in this experiment – inserted in the working memory of the Drools rule engine [PNF+ 08], where they can be queried and used for inferring high-level relations or triggering application-level actions; the time series of these qualitative relations can be seen in Figure 5.4. Due to the continuous movement with object p, no stable dynamic relations are recognized. An important aspect regarding the qualitative abstraction is the epsilon-neighborhood introduced in Section 3.2.3, which is necessary to avoid frequent relation changes at the transition from one to a neighboring qualitative static relation. They occur without actual movement but just because of the tracking system’s inaccuracies, and have to be taken into account depending on the semantics of the respective relation. In Figure 5.4 for example, there is no change from the qualitative relation medium-dist to near at t = 6 although the quantitative relation is below the threshold of 0, 30m, because the quantitative range of each distance relation is extended with epsilon = 0,05m. Similarly, qualitative dynamic relations do not change unless the quantitative difference between two successive points in time is sufficiently high. From the qualitative spatial relations in the working memory, new relations can be inferred using rule-based reasoning as described in Section 5.2. First, the relation passing-by-right is inferred with the rule of Listing 5.2. Every second, after recognizing the static and dynamic qualitative spatial relations, all rules are fired and the respective actions – in this case the insertion of the new passing-by-right relation – are executed. The relation is inferred first and inserted in the repository at t = 10, and at the points t = 11..16 its interval is extended by the maintenance rule of Listing 5.1. Due to using the minimum and maximum of the start- and endpoints of the three orientation relations back, front and right (cf. Section 3.2.5), the interval of the new relation is [0, 16] at the end (cf. Figure 5.4). Second, the inference rule passing-by-right-in-vicinity shown in Listing 5.3 uses the relation passing-by-right in its condition, and it results in a respective new relation with the interval [5, 13], in which the relations passing-by-right(p,r) and either near(p,r) or medium-dist(p,r) exist, in the working memory of the rule engine. Such inferred
5.4 Summary and Open Issues
119
Figure 5.4: Recognized and inferred qualitative relations.
relations constitute – together with those which are recognized by a comparison of Zones-of-Influence – the spatial relationship awareness of a digital artifact, and they can in turn be used in other rules.
5.4 Summary and Open Issues We have proposed a rule-based approach for reasoning about qualitative spatial and temporal relations, which constitutes a core component of the Zones-ofInfluence Framework presented in the next chapter. First, we argued for using rules to achieve spatial awareness, which is mainly due to their higher flexibility with regard to describing an artifact’s behavior, the more intuitive way of programming and the comparatively high speed and scalability by using a rule engine. In the field of pervasive computing, and especially with regard to context-aware applica-
120
5 Rule-Based Spatial Awareness
tions, the use of rules is widespread and can thus be considered as an established approach. We opted for production rules, and explained in detail the inference of conclusions from qualitative spatial relations therewith. We discussed examples of maintenance and in particular of inference rules, which are used for inferring both new relations and application-level actions, and presented a distinction of four categories of spatial situations that can be described with inference rules. These categories depend on whether static or dynamic relations are combined and if the represented situation relates to a single point in time or a time series of changing spatial relations. As a proof of concept, we have implemented a prototype which periodically recognizes and infers relations from position and direction information of a 6-DOF motion tracking system, and showed that our rule-based approach works reliably for recognizing and inferring different types of qualitative spatial and spatiotemporal relations. There are several open issues that have not been addressed yet. As will be seen in Chapter 7, the use of rules in connection with qualitatively abstracted spatial relations allows for developing spatially aware applications with less effort and in a more natural way. In this regard, it would be of interest to compare this declarative with an imperative approach for dealing with space, both with respect to the required storage and processing resources and concerning the application development time. A related open point is a user interface support for constructing rules, e.g. by using a graphical user interface which facilitates the process of selecting and combining relations to rules. The second issue is the use of rules with compound relations that comprise multiple alternatively possible base relations, like for example the combined orientation and distance relations we have used for compositional reasoning in Section 4.2.2. Although inference rules could be used to implement composition tables by inserting the respective relations in the repository for each combination of observed base relations, it is an open issue how to meaningfully use them for inferring high-level relations and application-level actions, as such compound relations are represented with multiple relations which may but need not hold. Another potentially problematic aspect in connection with inferred spatiotemporal relations is the fact that their intervals may range in the past and thus potentially cause actions to become invalid which have already been executed.
6 Zones-of-Influence Framework To me, the primary motivation behind the information appliance is clear: simplicity. Design the tool to fit the task so well that the tool becomes a part of the task, feeling like a natural extension of the person. This is the essence of the information appliance. Donald Norman, 1935-
In this chapter we present the Zones-of-Influence Framework, a software solution for enabling spatial awareness of autonomous embedded systems. It is based on the concepts presented in previous chapters, in particular the qualitative abstraction of spatial relations and the rule-based reasoning about them, and has been designed to facilitate the development of spatially aware applications for autonomous and spontaneously interacting systems. First, Section 6.1 presents design considerations for and an overview of the framework architecture, which has been developed with flexibility, modularity, efficiency and scalability in mind. It is split into components for the discovery of artifacts and the exchange of self-descriptions with them, the maintenance of a Zones-of-Influence model with the spatial properties of discovered artifacts, the application-dependent recognition and qualitative abstraction of spatial relations as well as the rule-based reasoning about qualitative relations over time in order to infer new relations or trigger according applicationlevel actions. An overview of these components – which provide spatial awareness to the application level – is given in Section 6.2, including a detailed description of their interfaces and relevant implementation-specific issues. The results of performance tests are presented in Section 6.3, which have been carried out in order to evaluate the runtime efficiency and scalability of the framework implementation; a qualitative evaluation by means of multiple application scenarios will be given afterwards in Chapter 7. Finally, Section 6.4 discusses features and properties of the framework and identifies open issues for future work. The development of the Zones-of-Influence Framework architecture and its implementation have been conducted in an industrial research project [FDE+ 08] with Siemens AG Germany.
122
6 Zones-of-Influence Framework
6.1 Architecture For the development of the framework architecture, several requirements had to be considered. They were derived from the general aim of this work, namely to make autonomous embedded systems – which are mobile, have to operate without a centralized instance, are potentially limited in their computational and communication capabilities and have to be able to incorporate physical sensors and actuators – aware about abstracted spatial relations over time, and will be discussed in the following subsection. Afterwards, the architecture of the Zones-of-Influence Framework is presented in Section 6.1.2, which has been developed according to these design considerations and builds on the general architecture for spatial awareness already presented in Section 2.5. Before going into implementation details in Section 6.2, the runtime platform – which has been used for it – is presented at the end of this section.
6.1.1 Design Considerations The Zones-of-Influence Framework should not only enable spatial awareness of spontaneously interacting autonomous embedded systems, which we have defined as a system’s awareness about its own spatial properties as well as about spatial relations to other systems, but it should also facilitate the development of spatially aware applications. We proposed a general architecture for achieving spatial awareness in Section 2.5, which consists of multiple layers through which lowlevel spatial context information is processed until it reaches the application. As we have mentioned in Section 2.1 already, layered architectures are used in many context-aware embedded systems – as for example in [GSB02, HBB02, HKG+ 05, SGKK04] – and can thus be considered as a common practice for building such systems. The layers of this previously discussed general architecture determine the functional requirements for the Zones-of-Influence Framework, which serves as a middleware between the spatial sensors and the communication module on the one hand and the application layer for running spatially aware applications on the other hand. These requirements are basically (i) the acquisition and representation of spatial contexts from one or multiple local sensors or by exchanging self-descriptions with other spatially aware artifacts, (ii) the application-dependent recognition and qualitative abstraction of spatial relations by comparing two Zones-of-Influence at a time, (iii) the maintenance of a spatial model of the environment which consists of recognized and inferred qualitative relations as well as (iv) the declarative rule-based approach for reasoning about them over time, as has been discussed in
6.1 Architecture
123
the Chapters 3 to 5. Although no general limitation, the functional scope within the present thesis is on processing the spatial properties position, direction and physical extension (cf. Section 3.1), and recognizing corresponding qualitative positional, directional and topological relations from them (cf. Section 3.2). The framework should enable the development of arbitrary spatially aware applications, and allow for dynamically adapting the incorporation of spatial sensors, the availability and semantics of spatial relations as well as the rules for reasoning about them according to the current application demands. Besides the functions the Zones-of-Influence Framework should provide, we identified several non-functional requirements for the design of its architecture, including but not limited to the following: • Flexibility with respect to the utilized spatial contexts, which means that it should be possible to exploit the spatial properties of arbitrary sensors with different reference systems, and to recognize, represent and reason about corresponding spatial relations • Modularity in order to allow for exchanging components by defined interfaces and facilitate the development of different spatially aware applications due to a clear separation of concerns • Efficiency of the implemented components concerning both processing and storage requirements, which contributes to their operation on resource-constrained embedded system platforms • Scalability with regard to the number of represented Zones-of-Influence, the maintained spatial or spatiotemporal relations and the rules for reasoning about them
6.1.2 Architecture Overview In this subsection, the architecture of the Zones-of-Influence Framework is presented, which is shown in Figure 6.1 and has been published in [HFH+ 08]. It consists of several components which cover different responsibilities, provide respective services to other components and use event-based communication for the exchange of data among them. In the following, their functional principles are explained in short, a detailed description of the components and their actual implementations will be given in Section 6.2. It should be noted that the current implementation is in a prototypical stage in that our focus was on the recognition and representation of qualitative relations from Zones-of-Influence contained in
124
6 Zones-of-Influence Framework
Figure 6.1: Overview of the Zones-of-Influence Framework architecture.
self-descriptions as well as means for reasoning about them, functionalities beyond that have been implemented only as far as required for evaluation purposes. The base component is the Digital Artifact Service, whose main functions are the discovery of artifacts and the exchange of self-descriptions with them, which contain – among others – information about their Zones-of-Influence and qualitative relations they are aware of (cf. Section 3.3); therefore, it also has to manage the local artifact’s self-description. The currently available implementation however just allows for a manual generation of self-descriptions, the discovery and communication functionalities have no been implemented yet. The Zones-of-Influence Service builds on the Digital Artifact Service and maintains an up-to-date model with the Zones-of-Influence of discovered artifacts. It registers as a listener with the Digital Artifact Service to get notified about new or changed self-descriptions, parses such self-descriptions for contained Zones-
6.1 Architecture
125
of-Influence upon notification and keeps them up-to-date (e.g. due to receiving a new self-description with changed spatial properties of an already known ZoI). For ZoIs which refer to dynamic sensor data, which means that the self-descriptions in which they are represented do not directly contain all of their spatial properties but sensor data references instead specify where they can be acquired from (cf. Section 3.3.1), the Zones-of-Influence Service continuously fetches the missing data with corresponding sensor data accessors and updates the ZoIs accordingly. An example of such a sensor is the IS-900 motion tracking system from Intersense Inc. [Int08], which has been used for evaluation purposes in this thesis. The Visualization Service visualizes currently maintained Zones-of-Influence, for which purpose it registers as a listener for new, changed or removed ZoIs with the Zonesof-Influence Service. An idea for future work that has not been addressed by now is to make the scenes – they determine which ZoIs are visualized and how – exchangeable, which is comparable to the runtime-adaptation of so-called themes in the peripheral display framework presented in [FEST06, Fer07a]. With regard to this thesis, the Relations Service is the heart of the Zones-ofInfluence Framework, as it is responsible for recognizing, representing and reasoning about qualitative relations. It allows for plugging in modules to recognize application-specific quantitative or qualitative spatial relations at runtime, which therefore register with the Zones-of-Influence Service as listeners for new, changed or removed ZoIs. Due to this dynamic adaptation, it is possible to recognize just those relations which are required by running applications and therewith save computing resources. In this regard, qualitative recognizers can register for changed relations with quantitative ones, which allows for separating qualitative abstractions from the quantitative relationship recognition and thus for using a single quantitative by multiple qualitative recognizers. Recognized qualitative relations are inserted in the relations repository, which is the working memory of the JBoss Drools 4.0 [PNF+ 08] rule engine in the current framework implementation (cf. Figure 5.1). For reasons of efficiency, relationship recognition is performed for changed or new Zones-of-Influence only, and qualitative relations are inserted or modified just in the case they are new or have changed (cf. Section 5.2). Applications can register with the Relations Service to get notified about relation changes or directly access the relations repository. The rule engine is used for inferring relations or application-level actions depending on the known relations stored in the repository. Additionally, relations can also be received with self-descriptions of artifacts in range, for which purpose the Relations Service registers as listener for new or changed self-descriptions with the Digital Artifact Service; however, this functionality has not been implemented yet.
126
6 Zones-of-Influence Framework
Two more components are shown in the architecture of Figure 6.1, but also not available at the time of this writing. The first one is the Rules Service, which encapsulates the production memory of the rule engine (cf. Figure 5.1) and provides means for managing rules – i.e. removing, modifying or adding them – at runtime. In the current implementation, rules are stored in a single file which is accessed from the Relations Service. For this reason, they cannot be modified at runtime, which however was not required for the evaluation. Secondly, the Query Service serves as an interface for applications to access spatial information; however, query techniques are beyond the scope of this thesis. In the case that the Drools rule engine is used, the Query Service could for example use both the Rules Service and the Relations Service in order to deploy so-called query rules and retrieve the respective results. Query rules are dedicated rules which have no consequence but just a conditional part, and their results – i.e. bound variables in the rule conditions – can be iterated by referring to the name of the query [PNF+ 08].
6.1.3 Runtime Platform For the implementation of the Zones-of-Influence Framework we decided to use the Java-based OSGi framework as runtime platform in order to meet the requirements discussed in Section 6.1.1. It implements a dynamic component model, in which the components (i.e. referred to as bundles) can be installed and managed at runtime and cooperate in a single Java Virtual Machine, and it has been used in a series of domains ranging from applications for mobile phones to the Eclipse IDE (cf. [All07]). The open-source Equinox [Ecl08] implementation of the OSGi R4 framework specification [All05] has been developed for the latter, and it also serves as a platform for our middleware framework. The OSGi framework is divided into several layers shown in Figure 6.2, which are in detail explained in [All05]. On top of the Java execution environment (e.g. J2SE or J2ME), the module layer defines a modularization model which provides means for sharing and hiding packages between bundles. The life-cycle layer provides a runtime model for managing the bundles in the module layer, and a corresponding API which allows for starting, stopping, installing, updating and uninstalling them at runtime. This is actually the most prominent feature, and it allows for changing the available services at runtime. The service layer provides a programming model for the development of bundles, which is based on a decoupling of their implementations and interfaces; bundles can dynamically register services with their interfaces, discover existing services or receive notifications when their registration state changes. The OSGi framework thus provides a service-oriented architecture that supports loosely-coupled application design, whereas an OSGi
6.2 Components
127
Figure 6.2: Layered architecture of the OSGi framework, after [All05, All07].
service is any Java object of a bundle which is made available to other bundles [All07]. The security layer extends the built-in security model of Java 2 with a strict separation of the bundles from each other. Using OSGi as runtime platform contributes to the ability of embedded systems to dynamically adapt their behavior, and it is thus valuable for their operation in changing environments. Moreover, such a service-oriented design is resource efficient as just services which are required by a running application have to be loaded and different applications can access the same service which therefore has to be loaded only once. From a software development viewpoint, using OSGi facilitates the integration of existing software and allows for developing small, loosely coupled components that can be composed to larger systems at runtime [All07]; however, it should be mentioned that OSGi is not mandatory for the Zones-ofInfluence Framework, but facilitates its implementation and deployment.
6.2 Components In this section, the interfaces and implementation details of the core framework components – i.e. the Digital Artifact Service, the Zones-of-Influence Service and the Relations Service – are presented, with the aim to clarify their internal structures and the interplay with other components.
6.2.1 Digital Artifact Service The Digital Artifact Service implements the communication component of the general architecture proposed in Section 2.5 and it is responsible for discovering artifacts in communication range and exchanging self-descriptions with them, which contain – among other things – information about their spatial contexts.
128
6 Zones-of-Influence Framework
It thus provides the basis for the recognition of spatial relations, which relies on the knowledge of the Zones-of-Influence of surrounding artifacts to which the relations should be calculated. The functionalities of the Digital Artifact Service are basically (i) the notification about discovered and no longer available artifacts as well as about changed self-descriptions of available artifacts, (ii) the selfdescription exchange with discovered artifacts as well as (iii) the management of the artifact’s local self-description at runtime (e.g. adding and removing Zones-ofInfluence). Figure 6.3 shows the main Java interfaces of the Digital Artifact Service component and their dependencies among each other; for the sake of a better understandability, not all methods contained in the respective interfaces are shown. The DigitalArtifactService interface provides methods for registering and unregistering DigitalArtifactAvailabilityListener objects, and notifies registered listeners about discovered or no longer available artifacts using the callback methods daAvailable or daUnavailable, respectively. Digital artifacts are represented with objects of type DigitalArtifact, whereas implementations of the DigitalArtifactService interface internally maintain a list of available artifacts that can be retrieved with the method getAvailableDigitalArtifacts. How artifacts are discovered depends on the actual DigitalArtifactService implementation, for example by broadcasting advertisements and taking advantage of timeouts as realized in [FHdSR+ 07] and [HKG+ 05]. The interface DigitalArtifact is used for retrieving an artifact’s self-description with the method getDADescription and for registering and unregistering as DADescriptionChangeListener in order to get notified whenever its self-description changes. The updateDADescription methods with and without parameter are called from the DigitalArtifactService to update the self-description of an artifact in the former or to notify about an already performed update in the latter case. The Zones-of-Influence Service as well as the Relations Service, which will be presented in the Sections 6.2.2 and 6.2.3, register both as DigitalArtifactAvailabilityListener with each Digital Artifact Service and as DADescriptionChangeListener with each available digital artifact to be notified about new and changed self-descriptions. As presented in Section 3.3, the XML-based self-description of an artifact may contain several elements of type DADescriptionElement, where each Zoneof-Influence or qualitative relation is represented by one such element. A selfdescription is encapsulated in a Java class DADescription which provides means for parsing its XML representation as well as for modifying and accessing the contents; every time the Digital Artifact Service discovers a new artifact, it adds a DigitalArtifact object with its self-description of type DADescription to the
6.2 Components
129
Figure 6.3: Interfaces of the Digital Artifact Service.
list of known artifacts. The self-description elements contained therein are represented with objects implementing the DADescriptionElement interface shown in Figure 6.3, which allows for retrieving an identification of the element type and the corresponding XML document. In order to be able to parse elements of different types, the DADescription class makes use of the factory design pattern. The DigitalArtifactService interface allows for registering and unregistering DADescriptionElementFactory objects, as for example by the Zonesof-Influence Service for parsing XML elements of type ZoI and SensorData (cf. Section 3.3.1) and creating DADescriptionElement objects which represent one Zone-of-Influence at a time; more details on that will be given in Section 6.2.2. By retrieving the corresponding factory with the DigitalArtifactService interface, the DADescription instances are able to parse all kinds of XML elements for which types respective factories have been registered, and maintain the generated DADescriptionElement objects in an internal data structure to be accessed by other services. As mentioned in the previous section already, just prototypical implementations for testing purposes which allow for a manual generation of self-descriptions are available at the time of this writing, but the discovery of artifacts and the communication between them have not been implemented yet. A potential system for that would be the Java-based Peer-it hardware/software platform proposed in [FHdSR+ 07], which could be integrated in our modular Zones-of-Influence Framework. Due to the loose coupling of components, multiple Digital Artifact Service instances at a time are possible, for example to support different communication technologies. Our first implementation of the DigitalArtifactService interface provides a GUI for loading self-descriptions from XML files, generates DigitalArtifact objects out of them and notifies registered listeners about the availability of new artifacts. The second implementation additionally serves as a Visualization Service, and thus both visualizes currently known Zones-of-
130
6 Zones-of-Influence Framework
Influence in 2-D and allows for interactively creating new and modifying existing ones; some screenshots will be seen later in Chapter 7. It therefore registers as a listener for new, changed and removed ZoIs with the Zones-of-Influence Service using the ZoIAvailabilityListener and ZoIChangeListener interface (cf. Section 6.2.2), generates DigitalArtifact instances corresponding to interactively created ZoIs and changes the self-descriptions of artifacts whose ZoIs have been modified. Registered DigitalArtifactAvailabilityListener and DADescriptionChangeListener objects are notified accordingly, in particular the Zones-of-Influence Service which thereupon updates its list of maintained Zones-of-Influence and in turn notifies registered listeners – including the interactive Visualization Service – about those that changed.
6.2.2 Zones-of-Influence Service The Zones-of-Influence Service builds on the Digital Artifact Service. It serves as an intermediary layer which collects the local (i.e. of the artifact which runs the framework) and remote (i.e. of artifacts in range) self-descriptions, extracts ZoI representations from them and provides an up-to-date model of all currently known Zones-of-Influence for the recognition of qualitative relations. In the case of dynamic Zones-of-Influence where one or more spatial properties are determined by sensor values, a retrieval of data from the respective sensors is required and will be explained later on. The Zones-of-Influence are internally maintained as a list of ZoI objects, which can be accessed via the ZoIService interface with several getter methods such as getZoIs for retrieving the whole list or getZoIsOfArtifact for retrieving all Zones-of-Influence belonging to a certain artifact; these two and related Java interfaces are shown in Figure 6.4. The ZoIService implementation also implements the interfaces DigitalArtifactAvailabilityListener and DADescriptionChangeListener, which contain callback methods for notifying about the availability of artifacts and changes of their self-descriptions (cf. Section 6.2.1), and is therewith able to constantly update the list of ZoIs accordingly. Whenever a digital artifact is notified to be available, the method daAvailable of the ZoIService implementation is called with a respective DigitalArtifact instance. It extracts all DADescriptionElement entries of type ZoI contained in the self-description, and adds the respective ZoI objects to its local list from which they are removed again in the method daUnavailable when the artifact becomes unavailable. As described in Section 6.2.1 already, the Zones-of-Influence Service registers a specialized factory of type DADescriptionElementFactory with the Digital Artifact Service for that, which is thus able to parse self-descriptions containing Zones-of-Influence, generate a ZoIDADescriptionElement object for
6.2 Components
131
Figure 6.4: Interfaces of the Zones-of-Influence Service.
each one and add them to the DADescription instance of the respective artifact. As can be seen in the ZoIDADescriptionElement interface of Figure 6.4, a respective ZoI object can be acquired with the method getZoI. In the case that a self-description has changed, the callback method daDescriptionChanged of the ZoIService implementation checks for changed Zones-of-Influence of the corresponding artifact and updates its list of ZoI objects accordingly. In order to notify other services like the Relations Service about available and no longer available Zones-of-Influence, the ZoIService interface provides methods for registering and unregistering ZoIAvailabilityListener objects which implement the callback methods zoiAvailable and zoiUnavailable. Each Zone-of-Influence is represented with an object that implements the ZoI interface shown in Figure 6.4. It contains methods for retrieving the name of the ZoI with getName, its full name (i.e. a colon-separated concatenation of the artifact’s name as well as the name of the ZoI which is unique per artifact) with getFullName as well as means for retrieving its position with getPosition, its direction with getDirection and a list containing one or more shapes with the method getShapes. In the current implementation of the framework, a Zone-ofInfluence comprises one position, one direction and one shape (cf. Section 3.3.1). For notifications 