Lecture Notes in Computer Science Commenced Publication in 1973 Founding and Former Series Editors: Gerhard Goos, Juris Hartmanis, and Jan van Leeuwen
Editorial Board David Hutchison Lancaster University, UK Takeo Kanade Carnegie Mellon University, Pittsburgh, PA, USA Josef Kittler University of Surrey, Guildford, UK Jon M. Kleinberg Cornell University, Ithaca, NY, USA Alfred Kobsa University of California, Irvine, CA, USA Friedemann Mattern ETH Zurich, Switzerland John C. Mitchell Stanford University, CA, USA Moni Naor Weizmann Institute of Science, Rehovot, Israel Oscar Nierstrasz University of Bern, Switzerland C. Pandu Rangan Indian Institute of Technology, Madras, India Bernhard Steffen University of Dortmund, Germany Madhu Sudan Microsoft Research, Cambridge, MA, USA Demetri Terzopoulos University of California, Los Angeles, CA, USA Doug Tygar University of California, Berkeley, CA, USA Gerhard Weikum Max-Planck Institute of Computer Science, Saarbruecken, Germany
5764
Sergey Balandin Dmitri Moltchanov Yevgeni Koucheryavy (Eds.)
Smart Spaces and Next Generation Wired/Wireless Networking 9th International Conference, NEW2AN 2009 and Second Conference on Smart Spaces, ruSMART 2009 St. Petersburg, Russia, September 15-18, 2009 Proceedings
13
Volume Editors Sergey Balandin Nokia Research Center Itamerenkatu 11-13, 00180 Helsinki, Finland E-mail:
[email protected] Dmitri Moltchanov Yevgeni Koucheryavy Tampere University of Technology Department of Communications Engineering Korkeakoulunkatu 10, 33720 Tampere, Finland E-mail: {moltchan, yk}@cs.tut.fi
Library of Congress Control Number: 2009933595 CR Subject Classification (1998): C.2, B.8, C.4, D.2, K.6, D.4.6, K.6.5 LNCS Sublibrary: SL 5 – Computer Communication Networks and Telecommunications ISSN ISBN-10 ISBN-13
0302-9743 3-642-04188-4 Springer Berlin Heidelberg New York 978-3-642-04188-4 Springer Berlin Heidelberg New York
This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, re-use of illustrations, recitation, broadcasting, reproduction on microfilms or in any other way, and storage in data banks. Duplication of this publication or parts thereof is permitted only under the provisions of the German Copyright Law of September 9, 1965, in its current version, and permission for use must always be obtained from Springer. Violations are liable to prosecution under the German Copyright Law. springer.com © Springer-Verlag Berlin Heidelberg 2009 Printed in Germany Typesetting: Camera-ready by author, data conversion by Scientific Publishing Services, Chennai, India Printed on acid-free paper SPIN: 12753468 06/3180 543210
Preface
We welcome you to the joint proceedings of the 9th NEW2AN (Next-Generation Teletraffic and Wired/Wireless Advanced Networking) and the Second ruSMART conferences held in St. Petersburg, Russia during September 15-17, 2009. This year NEW2AN featured significant contributions to various aspects of networking. Presented topics encompassed several layers of communication networks: from physical layers to transport protocols. In particular, issues of QoS in wireless and IP-based multi-service networks were dealt with. Cross-layer optimization, traffic characterization were also addressed within the program. It is also worth mentioning the emphasis placed on wireless networks, including, but not limited to, cellular networks, wireless local area networks, personal area networks, mobile ad hoc networks, and sensor networks. The Second Conference on Smart Spaces, ruSMART 2009, was targeted at attracting the attention of academic and industrial researchers to an emerging area of smart spaces that creates completely new opportunities for making fully customized applications and services for the users. The conference is a meeting place for leading experts from top affiliations around the world, with particularly active participation and strong interest from Russian attendees that have a good reputation for high-quality research and business in innovative service creation and applications development. The NEW2AN/ruSMART 2009 call for papers attracted 82 papers from 22 countries, resulting in an acceptance rate of 39%. With the help of the excellent Technical Program Committee and a number of associated reviewers, the best 32 high-quality papers were selected for publication. The conference was organized in seven single track sessions. We wish to thank the Technical Program Committee members of both conferences and the associated reviewers for their hard work and important contribution to the conference. The Technical Program of both conferences benefited from two keynote speakers: - Aaron J. Quigley, University College Dublin, Ireland - Manfred Schneps-Schneppe, Ventspils University College, Latvia This year the conferences were supported by the sponsor packages provided by NOKIA, Nokia Siemens Networks, Ubitel (Russia) and Baltic IT Ltd (Russia) and organized in cooperation with ITC (International Teletraffic Congress), IEEE, and Popov Society. The support of these organizations is gratefully acknowledged. Finally, we wish to thank many people who contributed to the organization. In particular, Jakub Jakubiak (TUT, Finland) carried a substantial load of the submissions and reviews, website maintaining, did an excellent job on the compilation of camera-ready papers and liaising with Springer. Many thanks go to
VI
Preface
Natalia Avdeenko and Ekaterina Antonyuk (Monomax Meetings & Incentives) for their excellent local organization efforts and the conference’s social program preparation. We believe that the 9th NEW2AN and the Second ruSMART conferences provided an interesting and up-to-date scientific program. We hope that the participants enjoyed the technical and social conference program, Russian hospitality and the beautiful city of St. Petersburg.
July 2009
Sergey Balandin Dmitri Moltchanov Yevgeni Koucheryavy
Organization
NEW2AN International Advisory Committee Ian F. Akyildiz Nina Bhatti Igor Faynberg Jarmo Harju Andrey Koucheryavy Villy B. Iversen Paul K¨ uhn Kyu Ouk Lee Mohammad S. Obaidat Michael Smirnov Manfred Sneps-Sneppe Ioannis Stavrakakis Sergey Stepanov Phuoc Tran-Gia Gennady Yanovsky
Georgia Institute of Technology, USA Hewlett Packard, USA Alcatel Lucent, USA Tampere University of Technology, Finland ZNIIS R&D, Russia Technical University of Denmark, Denmark University of Stuttgart, Germany ETRI, Korea Monmouth University, USA Fraunhofer FOKUS, Germany Ventspils University College, Latvia University of Athens, Greece Sistema Telecom, Russia University of W¨ urzburg, Germany State University of Telecommunications, Russia
NEW2AN Technical Program Committee TPC Chair Dmitri Moltchanov Mari Carmen Aguayo-Torres Ozgur B. Akan Khalid Al-Begain Sergey Andreev Tricha Anjali Konstantin Avrachenkov Francisco Barcelo Sergey Balandin Thomas M. Bohnert Torsten Braun Chrysostomos Chrysostomou Georg Carle Ibrahim Develi Roman Dunaytsev Eylem Ekici
Tampere University of Technology, Finland University of Malaga, Spain METU, Turkey University of Glamorgan, UK State University Aerospace Instrumentation, Russia Illinois Institute of Technology, USA INRIA, France UPC, Spain Nokia, Finland SAP Research, Switzerland University of Bern, Switzerland University of Cyprus, Cyprus University of T¨ ubingen, Germany Erciyes University, Turkey Tampere University of Technology, Finland Ohio State University, USA
VIII
Organization
Sergey Gorinsky Markus Fidler Giovanni Giambene Stefano Giordano Ivan Ganchev Vitaly Gutin Martin Karsten Andreas Kassler Maria Kihl Tatiana Kozlova Madsen Yevgeni Koucheryavy Jong-Hyouk Lee Vitaly Li Lemin Li Leszek T. Lilien Saverio Mascolo Maja Matijaˇsevic Paulo Mendes Ilka Miloucheva Edmundo Monteiro Se´an Murphy Marc Necker Mairtin O’Droma Jaudelice Cavalcante de Oliveira Evgeni Osipov George Pavlou Simon Pietro Romano Alexander Sayenko Dirk Staehle Sergei Semenov Burkhard Stiller Weilian Su Veselin Rakocevic Dmitry Tkachenko Vassilis Tsaoussidis
Washington University in St. Louis, USA NTNU Trondheim, Norway University of Siena, Italy University of Pisa, Italy University of Limerick, Ireland Popov Society, Russia University of Waterloo, Canada Karlstad University, Sweden Lund University, Sweden Aalborg University, Denmark Tampere University of Technology, Finland (Chair) Sungkyunkwan University, R. Korea Kangwon National University, R. Korea University of Electronic Science and Techn. of China, China Western Michigan University, USA Politecnico di Bari, Italy University of Zagreb, FER, Croatia INESC Porto, Portugal Salzburg Research, Austria University of Coimbra, Portugal University College Dublin, Ireland University of Stuttgart, Germany University of Limerick, Ireland Drexel University, USA Lulea University of Technology, Sweden University of Surrey, UK Universit` a degli Studi di Napoli “Federico II”, Italy Nokia Siemens Networks, Finland University of W¨ urzburg, Germany Nokia, Finland University of Zurich and ETH Zurich, Switzerland Naval Postgraduate School, USA City University London, UK IEEE St.Petersburg BT/CE/COM Chapter, Russia Demokritos University of Thrace, Greece
Organization
Christian Tschudin Andrey Turlikov Kurt Tutschku Alexey Vinel Lars Wolf
University of Basel, Switzerland State University of Aerospace Instrumentation, Russia University of Vienna, Austria SPIIRAN, Russia Technische Universit¨ at Braunschweig, Germany
NEW2AN Additional Reviewers B. Alhaija A. Baryun C. Callegari S. Diamantopoulos R. Garroppo P. Godlewski J. Granjal C. Hoene
D. Iacono J. Jakubiak I. Komnios J. H. Lee E. Leitgeb D. Milic P. Mitoraj M. Okada
C. Pelizzoni J. Peltotalo V. Pereira M.-D. Perez Guirao S. Poryazov T. Staub M. Waelchli
ruSMART Executive Technical Program Committee Sergey Boldyrev Nikolai Nefedov Ian Oliver Alexander Smirnov Vladimir Gorodetsky Michael Lawo Michael Smirnov Dieter Uckelmann Cornel Klein Maxim Osipov
Nokia Research Center, Helsinki, Finland Nokia Research Center, Zurich, Switzerland Nokia Research Center, Helsinki, Finland SPIIRAS, St.-Petersburg, Russia SPIIRAS, St.-Petersburg, Russia Center for Computing Technologies (TZI), University of Bremen, Germany Fraunhofer FOKUS, Germany LogDynamics Lab, University of Bremen, Germany Siemens Corporate Technology, Germany Siemens CT, Embedded Linux, Russia
ruSMART Technical Program Committee Juha Laurila Sergey Balandin Alexey Dudkov Didem Gozupek Kim Geunhyung Reto Krummenacher Prem Jayaraman Michel Banˆ atre Sergei Bogomolov
Nokia Research Center, Switzerland Nokia Research Center, Finland University of Turku, Finland Bogazici University, Turkey Dong Eui University, Korea STI Innsbruck, Austria Monash University, Australia IRISA, France LGERP R&D Lab, Russia
IX
X
Organization
Gianpaolo Cugola Dimitri Konstantas Markus Taumberger Bilhanan Silverajan Aaron J. Quigley
Politecnico di Milano, Italy University of Geneva, Switzerland VTT, Finland Tampere University of Technology, Finland University College Dublin, Ireland
ruSMART Additional Reviewers Y. Koucheryavy
Organization
XI
Table of Contents
I
ruSMART
Invited Talk ITU G.hn Concept and Home Automation . . . . . . . . . . . . . . . . . . . . . . . . . . Manfred Schneps-Schneppe
1
Session I Extending Context Spaces Theory by Predicting Run-Time Context . . . . Andrey Boytsov, Arkady Zaslavsky, and K˚ are Synnes
8
Cross-Domain Interoperability: A Case Study . . . . . . . . . . . . . . . . . . . . . . . . Jukka Honkola, Hannu Laine, Ronald Brown, and Ian Oliver
22
A Topology Based Approach for Context-Awareness in Smart Environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Antonio Coronato and Giuseppe De Pietro Anonymous Agent Coordination in Smart Spaces: State-of-the-Art . . . . . Alexander Smirnov, Alexey Kashevnik, Nikolay Shilov, Ian Oliver, Sergey Balandin, and Sergey Boldyrev
32 42
Session II On-the-Fly Situation Composition within Smart Spaces . . . . . . . . . . . . . . . Prem Prakash Jayaraman, Arkady Zaslavsky, and Jerker Delsing Empower Mobile Workspaces by Wireless Networks and Wearable Computing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Michael Lawo, Otthein Herzog, Michael Boronowski, and Peter Knackfuß Multimodal Interaction with Intelligent Meeting Room Facilities from Inside and Outside . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A.L. Ronzhin and V.Yu. Budkov
52
66
77
Session III Ubi-Check: A Pervasive Integrity Checking System . . . . . . . . . . . . . . . . . . . Michel Banˆ atre, Fabien Allard, and Paul Couderc
89
XIV
Table of Contents
Towards a Lightweight Security Solution for User-Friendly Management of Distributed Sensor Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Pentti Tarvainen, Mikko Ala-Louko, Marko Jaakola, Ilkka Uusitalo, Spyros Lalis, Tomasz Paczesny, Markus Taumberger, and Pekka Savolainen Cross-Site Management of User Online Attributes . . . . . . . . . . . . . . . . . . . . Jin Liu
II
97
110
NEW2AN
Teletraffic Issues Analysis and Optimization of Aggregation in a Reconfigurable Optical ADD/DROP Multiplexer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . J.M. Fourneau, N. Izri, and D. Verch`ere Teletraffic Capacity Performance of WDM/DS-OCDMA Passive Optical Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Mohammad Gharaei, Catherine Lepers, Olfa Affes, and Philippe Gallion
120
132
Estimation of GoS Parameters in Intelligent Network . . . . . . . . . . . . . . . . . Irina Buzyukova, Yulia Gaidamaka, and Gennady Yanovsky
143
Multi-Skill Call Center as a Grading from “Old” Telephony . . . . . . . . . . . Manfred Schneps-Schneppe and Janis Sedols
154
Traffic Measurements, Modeling, and Control A Real-Time Algorithm for Skype Traffic Detection and Classification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . D. Adami, C. Callegari, S. Giordano, M. Pagano, and T. Pepe
168
HTTP Traffic Measurements on Access Networks, Analysis of Results and Simulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Vladimir Deart, Vladimir Mankov, and Alexander Pilugin
180
The Video Streaming Monitoring in the Next Generation Networks . . . . . A. Paramonov, D. Tarasov, and A. Koucheryavy The Poisson Cluster Process Runs as a Model for the Internet Traffic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Jerzy Martyna
191
206
Table of Contents
XV
Peer-to-Peer Systems Proactive Peer-to-Peer Traffic Control When Delivering Large Amounts of Content within a Large-Scale Organization . . . . . . . . . . . . . . . . . . . . . . . . Chih-Chin Liang, Chia-Hung Wang, Hsing Luh, Ping-Yu Hsu, and Wuyi Yue
217
ISP-Driven Managed P2P Framework for Effective Real-Time IPTV Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Seil Jeon, Younghan Kim, Jonghwa Yi, and Shingak Kang
229
Fault-Tolerant Architecture for Peer to Peer Network Management Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Maryam Barshan, Mahmood Fathy, and Saleh Yousefi
241
Security Issues Public Key Signatures and Lightweight Security Solutions in a Wireless Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Dmitrij Lagutin and Sasu Tarkoma
253
On the Operational Security Assurance Evaluation of Networked IT Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Artur Hecker and Michel Riguidel
266
Trust Management Using Networks of Volunteers in Ubiquitous Computing Environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Mijeom Kim, Mohan Kumar, and Sukju Jung
279
A Fast and Efficient Handover Authentication Achieving Conditional Privacy in V2I Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Jaeduck Choi, Souhwan Jung, Younghan Kim, and Myungsik Yoo
291
Wireless Networks: Ad Hoc and Mesh Robust and Efficient Routing in Wireless Mesh Networks with Unreliable Nodes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Apostolia Papapostolou, Vasilis Friderikos, Tara A. Yahiya, and Hakima Chaouchi Video Compression for Wireless Transmission: Reducing the Power Consumption of the WPAN Hi-Speed Systems . . . . . . . . . . . . . . . . . . . . . . . Andrey Belogolovy, Eugine Belyaev, Anton Sergeev, and Andrey Turlikov Link Stability-Aware Ad Hoc Routing Protocol with Multiple Sampling Rates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Myungsik Yoo, Junwon Lee, Younghan Kim, and Souhwan Jung
301
313
323
XVI
Table of Contents
Wireless Networks: Capacity and Mobility A Quality of Service Based Mobility Management in Heterogenous Wireless Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Tara A. Yahiya and Hakima Chaouchi
334
A Novel Cooperative Relaying Scheme for Next Generation Wireless Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Hong Zhou
346
Frequency Allocation Scheme to Maximize Cell Capacity in a Cellular System with Cooperative Relay . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Wooju Lee, Kyongkuk Cho, Dongweon Yoon, Sang Kyu Park, and Yeonsoo Jang
358
Achieving Secondary Capacity under Interference from a Primary Base Station . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Shadi Ebrahimi Asl and Bahman Abolhassani
365
Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
377
ITU G.hn Concept and Home Automation Manfred Schneps-Schneppe Ventspils University College Inzenieru st 101,Ventspils, Latvia
[email protected]
For long time, many in the buildings industry have been looking for a day when home automation systems would become fully integrated with communication and humaninterface practices, with standards widely employed for information technology. Now the time is coming in the form of the ITU Recommendations G.hn.
1 SmartHouse Concept The most important approach in the field in home automation is the European “SmartHouse Code of Practice” [1]. Figure 1 shows the key role of the residential gateway (home gateway) in a SmartHouse framework.
Fig. 1. Role of the RG in a SmartHouse framework
The SmartHouse consists of a large and wide ranging set of services, applications, equipment, networks and systems that act together in delivering the “intelligent” home in order to address security and control, communications, leisure and comfort, environment integration and accessibility. Table 1 below illustrates the relationship between the technical requirements on bandwidth and SmartHouse services offered. The G.hn approach should enable the ability to manage home electronic systems from one main control point and make our household run smoother, feel better and save energy [2], taking into account many home automation requirements, particularly: S. Balandin et al. (Eds.): NEW2AN/ruSMART 2009, LNCS 5764, pp. 1–7, 2009. © Springer-Verlag Berlin Heidelberg 2009
2
M. Schneps-Schneppe Table 1. Examples of SmartHouse services and associated bandwidth
Interoperability. A good example of interoperability is having the lights turn off, the thermostats set back when you press a “goodbye” button on a keypad or when a motion sensor notices that you have exited a room. Remote Access. Remote access capabilities allow us to monitor our home’s environment and alter the settings of the lights, thermostats and other gear if necessary all from your laptop or cellphone. Expandability. It is important that a home automation system can be easily expanded both vertically to incorporate additional products and horizontally to support additional rooms. Manufacturers can support vertical and horizontal expandability by designing their systems to speak a common network language, like IP (Internet Protocol), and by offering wireless products that can communicate with a home’s existing network of wired products. Upgradeability. Software is the driving force of an automation system. The more sophisticated that software is, the more the system can do (see HGI picture below).
ITU G.hn Concept and Home Automation
3
Variety of Interfaces. In the age of universal graphical user interfaces (GUI), is there any reason to continue using a different home automation manufacturer's graphics instead of Web-browser-type operator interfaces? Energy-Savings. One of the hottest topics in the consumer media is energy conservation. Automation systems can help save energy by turning off electronics devices automatically.
2 ITU Recommendations G.hn The ITU concept G.hn is a worldwide standard, addresses technology for wired home-networks. G.hn is the common name for the "next generation" home network technology standard. The G.hn technology goal is to unify connectivity of digital content and media devices by providing a network over three popular types of wiring found in homes today: coax cable, phone lines, and AC power wiring and to supply data rates up to 1 Gbit/s. The ITU G.hn standard’s goal is to specify the physical (layer 1) and link (layer 2) layers for home wired networks (Fig. 2). This work culminated in Recommendation ITU G.9960 specifying G.hn’s Physical Layer. The G.hn Data Link Layer is divided into three sub-layers: 1) Application Protocol Convergence (APC) Layer, which receives Ethernet frames from the application layer and encapsulates them into G.hn MAC Service Data Units, 2) Logical Link Control (LLC), which is responsible for encryption, aggregation, segmentation and automatic repeat-request, and 3) Medium Access Control (MAC), which schedules channel access in according to TDMA method to avoid collisions. G.hn specifies a Physical Layer based on fast Fourier transform (FFT) Orthogonal frequency-division multiplexing (OFDM) modulation and Low-Density Parity-Check
Fig. 2. G.hn node protocol stack
4
M. Schneps-Schneppe
Fig. 3. Home phoneline networking data frame is based on Ethernet standards
FEC code. OFDM systems split the transmitted signal into multiple orthogonal carriers. These carriers are modulated using Quadrature amplitude modulation (QAM). The G.hn Physical Layer is divided into three sub-layers: 1) Physical Coding Sublayer (PCS), for generating PHY headers (Fig. 3), 2) Physical Medium Attachment (PMA), for scrambling and FEC coding/decoding, 3) Physical Medium Dependent (PMD), for bit-loading/stuffing and OFDM modulation. The PMD sub-layer is the only sub-layer in the G.hn stack that is "medium dependent" (ie, some parameters may have different values for each media - power lines, phone lines and coaxial cable). The rest of sub-layers (APC, LLC, MAC, PCS and PMA) are "medium independent". G.hn proponents are working now to make G.hn the future universal wired home networking standard worldwide. G.hn profiles for home automation are on further study yet (Fig. 4).
Fig. 4. Example of a G.hn network
ITU G.hn Concept and Home Automation
5
3 Home Gateway Initiative: RG Software Development Approach 1 There are currently several competing service delivery architectures now. OSGi Alliance is one of erlier, and Home Gateway Initiative is the newiest. The Home Gateway Initiative (HGI) is an open forum launched by a number of telephone companies (Belgacom, BT, DT, FT, KPN, Teliasonera, NTT, Telefonica, Telecom Italia) in December 2004 with the aim to release specifications of the home gateway. Many manufacturers (including Microsoft) have joined the alliance. The initiative will drive the development of residential gateways supporting the delivery of services. The initiative will take as a basis the work undertaken within existing bodies (such as ITU-T, Broadband forum, DLNA, OSGi Alliance, etc). The goals of the initiative are to produce and downstream requirements for a residential gateway enabling end to end delivery of services (Fig. 5), to work with manufacturers in order to leverage volumes, to validate with manufacturer against uses cases and requirements, to ensure interoperability.
Fig. 5. Home gateway environment according to the HGI concept
4 OSGi Alliance: RG Software Development Approach 2 The OSGi Alliance(formerlyas the Open Services Gateway initiative) was founded by Ericsson, IBM, Motorola, Sun Microsystems and others join in 1999. Among its members are more than 35 companies from quite different business areas. The Alliance and its members have specified a Java-based service platform that can be remotely managed (Fig. 6). (We are with Java-based service platform for long timr, see http://abava.blogspot.com/.)
6
M. Schneps-Schneppe
Fig. 6. OSGi Service Gateway Architecture
5 Conclusion The right time is now to develop the G.hn profiles for home automation. According to Wikipedia [4], three home networking organizations that promoted previously incompatible technologies (CEPCA, HomePNA and the Universal Powerline Association), announced that they had agreed to promote G.hn as the single nextgeneration standard for wired home networking, and to work to ensure coexistence with existing products in the market. The Continental Automated Buildings Association (CABA) and HomeGrid Forum signed a liaison agreement to support HomeGrid Forum’s efforts in conjunction with ITU-T G.hn to make it easy for consumers worldwide to connect devices and enjoy innovative applications using existing home wiring. Case studies. The paper is illustrated by case studies (from Latvia): Case 1. The KNX (Konnex) standard based demonstration room is arranged by Urban Art company (Riga) in cooperation with University of Latvia. Time switch programs enable users to heat or ventilate each room individually, while a remote control is also available for operating light switches, monitoring windows and doors for home security. The web server enables users to operate intelligent room from any PC or smart phone, requiring only a common operating system and a browser. Another new feature is the alarm function that sends e-mail or text message warnings to four predefined recipients.
ITU G.hn Concept and Home Automation
7
Case 2. Abavanet company (Ventspils) has developed a series of m-bus controlled devices for water supply automatic measurement (wireline and wireless). Case 3. MikroDators company (Ogre) has produced wireless measuring devices for hot water supplier Rigas Siltums company.
References 1. 2. 3. 4.
SmartHouse Code of Practice, CENELEC, CWA 50487 (November 2005) http://www.electronichouse.com/article/ 10_features_to_look_for_in_a_home_automation_system/ Home Gateway Technical Requirements (2008), http://www.homegateway.org/ http://en.wikipedia.org/wiki/G.hn
Extending Context Spaces Theory by Predicting Run-Time Context Andrey Boytsov, Arkady Zaslavsky, and Kåre Synnes Department of Computer Science and Electrical Engineering Luleå University of Technology, SE-971 87 Luleå {Andrey.Boytsov,Arkady.Zaslavsky,Kare.Synnes}@ltu.se
Abstract. Context awareness and prediction are important for pervasive computing systems. The recently developed theory of context spaces addresses problems related to sensor data uncertainty and high-level situation reasoning. This paper proposes and discusses componentized context prediction algorithms and thus extends the context spaces theory. This paper focuses on two questions: how to plug-in appropriate context prediction techniques, including Markov chains, Bayesian reasoning and sequence predictors, to the context spaces theory and how to estimate the efficiency of those techniques. The paper also proposes and presents a testbed for testing a variety of context prediction methods. The results and ongoing implementation are also discussed. Keywords: context awareness, context prediction, context spaces theory, pervasive computing, Markov model, Bayesian network, branch prediction, neural network.
1 Introduction Pervasive computing is a paradigm where computing systems are integrated into the everyday life and environment in a non-intrusive, graceful and transparent manner. For example, it can be a smart home or office, where doors are automatically opened and light is automatically turned on right before a person enters the room [18]. Or it can be a smart car, which suggests the fastest way to the destination and which assesses its own condition and proposes maintenance plan. Many pervasive computing systems are now being introduced into our life. Context awareness and context prediction are relatively new research areas, but they are becoming an important part of pervasive computing systems. This paper analyzes various context prediction techniques and proposes a plug-in approach to context prediction for pervasive computing systems at run-time. This approach is proposed as an extension of context spaces theory [17, 19]. This paper focuses on how to apply various available context prediction techniques and how to estimate the efficiency of those techniques. This paper also proposes validation of the pluggable context prediction techniques using the “Moonprobe” model and application scenario that imitates movement of vehicle over a sophisticated landscape. The article is structured as follows. Necessary definitions are included in section 2. Section 3 provides a brief overview of the context spaces theory – its essence, S. Balandin et al. (Eds.): NEW2AN/ruSMART 2009, LNCS 5764, pp. 8–21, 2009. © Springer-Verlag Berlin Heidelberg 2009
Extending Context Spaces Theory by Predicting Run-Time Context
9
addressed problems, proposed solutions and current research challenges. In section 4 we propose context prediction methods and develop the algorithms for their adaptation to the theory of context spaces. In section 5 we introduce “Moonprobe” model and application scenario – a testing framework that we developed to estimate context prediction efficiency. Section 6 summarizes the paper and proposes future work.
2 Definitions This paper addresses many important issues of pervasive computing systems. Generally, pervasive systems comprise many small and inexpensive specialized devices. If a device is used to specifically obtain information from the environment that device is usually referred to as a sensor. Each device performs its own functions, but to be able to perform them effectively the device usually needs to communicate with other devices and process the information that it obtains from them. One of the most important characteristics of a pervasive computing system is its context. In earlier works on context awareness different definitions of context were proposed. Comprehensive overview of those efforts was presented by Dey and Abowd [6]. They define context as “any information that can be used to characterize the situation of an entity.” In fact, every piece of information that a system has is a part of that system’s context. The system is context aware if it can use context information to its benefit. Reasoning about context is the process of obtaining new knowledge from current and possibly predicted context. Context model is a way of context representation that is used for further reasoning. We assume that context aware pervasive computing systems have the following features: y Sensors supply data that will be used by pervasive system for further reasoning. Examples of sensors may include light sensors, altitude sensors, accelerometers, etc. y Sensors transfer all the data to reasoning engine via a sensor network. y The reasoning engine, which can possibly be a distributed system, performs reasoning, makes context predictions and produces some output that can be used for modifying system’s behavior. Many important issues of pervasive computing systems are left out of scope of this paper, for example, security, privacy, need for distributed computations and other related problems.
3 Context Spaces Theory The theory of context spaces [17, 19] is a recently developed approach for context awareness and reasoning which addresses the problems of sensors uncertainty and unreliability. It also deals with situation reasoning and the problems of context representation in a structured and meaningful manner. Context spaces theory is designed to enable context awareness in clear and insightful way. This theory uses spatial metaphors for representing context as a multidimensional space. To understand context spaces theory we need to introduce several new
10
A. Boytsov, A. Zaslavsky, and K. Synnes
terms. Any kind of data that is used to reason about context is called context attribute. Context attribute usually corresponds to a domain of values of interest, which are either measured by sensors directly or calculated from other context attributes. It can be either numerical value or a value from pre-defined set of non-numerical options. Context state represents the set of all relevant context attributes at a certain time. A set of all possible context states constitutes application space. Therefore, application space can be viewed as a multi-dimensional space where the number of dimensions is equal to the number of context attributes in the context state. The state of the system is represented by a point in the application space and the behavior of the system is represented by a trajectory moving through the application space over time. Situation space is meant to represent real life situation. It can be defined as subspace of the application space. So if context state is in the subspace representing situation S, it means that situation S is occurring. Situation S has the level of occurrence certainty. It depends on the probability of the context state to be within S and it also depends on the subspace within S. See figure 1 for simple illustration. We’ll keep referring to that example for context prediction approaches illustration throughout the article.
Fig. 1. Context spaces theory
In context spaces theory several methods were developed for reasoning about the context. Bayesian reasoning [23] or Dempster-Schafer algorithm [24] are used to get overall confidence in the fact that a situation is occurring. Algebraic operations on situations and some logic-based methods were developed for reasoning in terms of situations. Context spaces theory was implemented in ECORA [20] – Extensible Context Oriented Reasoning Architecture. ECORA is a framework for development of contextaware applications. That framework provides its functionality as a set of Java classes to be integrated into the prototypes of context-aware systems. The research presented in this paper develops context prediction methods that can be used to extend the context spaces theory and benefit ECORA-based applications.
Extending Context Spaces Theory by Predicting Run-Time Context
11
4 Context Prediction for Context Spaces Theory The ability to predict future context will benefit runtime performance of context aware applications. For example, use cases for context prediction can be found in [16]. Predicted context can be used for early warnings, for power management, for system reconfiguration etc. Context prediction is one of research challenges for context spaces theory. In practice, most of current context prediction approaches involve inferring various context features during run-time. The results of inference are then used for context prediction. That inference is actually a machine learning task. For this purpose neural networks [12], Markov chains [5] and Bayesian networks [23] might be sufficiently effective tools. Here we analyze possible context prediction techniques and develop the appropriate algorithms for two-way mapping between the context spaces theory and the analyzed context prediction technique. Sequence predictors for context prediction. Many context prediction approaches were inspired by research in UNIX next command prediction [8, 9]. There authors suggested to represent each UNIX command as a symbol and the whole command flow – as symbol sequence. Also they proposed a set of algorithms for sequence prediction. Sequence prediction techniques were later used in a variety of systems from “Smart Home” architectures to prediction of movement in cellular networks. Various modifications of LZ78 algorithm [26] were used in various papers [1, 4, 7, 10, 15, 22]. It is worth noting the paper [15] which presents general application-independent architecture for context prediction. [15] also suggests that Active LeZi algorithm can be used to predict future situations (note that the concept of a situation in [15] differs from the concept of a situation in context spaces theory). The Active LeZi algorithm itself is described in [10]. According to [15], Active LeZi provides up to 99% of accuracy on 1-step prediction. Sequence prediction techniques are feasible in context spaces theory as well. For prediction purpose context can be represented as a sequence of situations. When situation occurs, a special symbol is added to the sequence. When situation is over, another special symbol can be added to the sequence as well. Sequence prediction techniques can be applied to predict the next symbol in that sequence and, therefore, to predict the occurrence of a situation. For example, context state flow in Fig. 1 can be described by following sequence: (SS3-in)(SS4-in)(SS3-out)(SS4-out)(SS1-in) Sequence prediction algorithms should be adjusted to the fact that situations have certain levels of confidence. It can be done by introducing the threshold for the level of certainty. We consider that the situation is occurring if its level of confidence is above the threshold and we consider that the situation is over if its level of confidence drops below the threshold. In [15] active LeZi showed very good performance in a variety of cases. Incorporation of active LeZi algorithm into context spaces theory is in progress. Its efficiency is currently being investigated. Neural networks for context prediction. Some techniques of context prediction use neural networks to learn from user behavior. For example, [14] describes “Neural Network House” project where special equipment predicts expected occupancy patterns of rooms in the house, estimates hot water usage, estimates the likelihood for a room to be entered soon, etc. In [14] authors used feedforward neural networks trained
12
A. Boytsov, A. Zaslavsky, and K. Synnes
with back propagation. Projects related to indoor movement prediction considered neural network approach as well. In [25] authors addressed the problem of predicting next room the person is going to visit. A neural network took current room number as an input and produced the most probable next room number as an output. Logs of user movement were used to train the network. For prediction purposes authors chose multi-layer perceptron with one hidden layer and back propagation learning algorithm. Neural networks adaptation for context spaces theory can use several approaches. For example, the neural network can accept current situation (with its level of confidence) and predict next situation to occur. Alternatively, the neural network can accept the whole set of current situations and return a whole set of situations expected over a period of time. Additionally, neural network can accept context coordinates and infer the expected future coordinates or expected future situations. Markov models for context prediction. Markov models proved to be a feasible way of context prediction. In [2, 3] authors used Markov chains to predict what internet site user is likely to visit next. Context prediction using Markov chains based techniques and Bayesian networks based techniques were considered in the [13]. In there authors addressed the active device resolution problem, when the user has a remote control to interact with a set of devices, but there are always several devices in user proximity. Some papers use hidden Markov model to learn from user behavior. For example, in [11] hidden Markov models are used to obtain prediction of next room that the person is likely to visit. The layout of rooms in the house, represented as a graph, is handled as a hidden Markov model, and the history is used to obtain the probabilities of moving from one room to another. Context in our approach can be viewed as a Markov model as well. For that we need to represent the context as a graph where every node represents a situation. However, Markov model states have to be mutually exclusive. It is not generally so in context spaces theory. To make situations mutually exclusive, we need to find all kinds of situations intersection and extract them as new situations. To do this we developed the following algorithm. Algorithm MM4CS. Step 1. Create three entities: Entity 1: set of situations. Initially it contains all defined situations. From now on initial situations will be referred to as S1, S2, S3, …, SN. Also they will be referred to as old situation spaces. Newly acquired situations may appear as a result of situation decomposition. They will be referred to according to the situation algebra expressions they were obtained from, for example, S1∩S2∩!S3 or S10∩S8, etc. From now on, for simplicity, when talking about situations and it does not matter whether those situations are initial or newly acquired we’ll refer to them as A,B,C, etc. Subspace of application space that includes whole application space except the situation A will be referred to as !A. Entity 2: set of situation pairs. Elements of the set will be referred like (A, B), where A and B are some situations. Expression (A,*) is a joint name of all elements containing the situation A. Pair of situation with itself like (A,A) are not supported. Situation pairs are unordered, i.e. (A,B) and (B,A) both identify the same set element. Initially the set contains all available pairs of starting set of situations, i.e. (Si, Sj) for every i,j ϵ [1.. N] and i≠j.
Extending Context Spaces Theory by Predicting Run-Time Context
13
Entity 3: dependency graph. Initially it has 2 sets of vertices. Each vertex of each set represents a situation. Vertex will be referred as (A)j where the A represents situation and the superscript represents the set number (1 or 2). Dependency graph will be used for prediction in terms of old situation spaces. Initially it has only vertices (Si)1 and (Si)2 and edges between (Si)1 and (Si)2 (i ϵ [1.. N]). Edges are undirected. Also in the graph special temporary vertices can appear during the runtime. They will be referred as, for example, (A)j:B. Step 2. Take any element from a situation pairs set. Let’s say, it is element (A, B). See, if situation spaces A and B have any intersections in application space. It can be done using situation algebra provided in [17]. If there are no intersections between A and B – just remove the element (A, B) from the situation pair set. If there are any intersections, a set of substeps has to be performed: Substep 2.1. Remove element (A, B) from the set. Substep 2.2. Iterate through the situation pairs list and perform following changes: Substep 2.2.1. Every (A,*) element should be replaced with two new elements. Element (A∩B, *) is created in all cases and element (A∩!B, *) is created if the subspace A∩!B is not empty. Substep 2.2.2. Every (B,*) element should be replaced with two new elements as well. Element (A∩B, *) is created if it was not created on substep 2.2.1 and element (!A∩B, *) is created if the subspace !A∩B is not empty. Substep 2.3. Update dependency graph: Substep 2.3.1. Vertex (A)2 should be removed and several new vertices should be created instead. Vertex (A∩B)2:A should be created in all cases and vertex (A∩!B)2 should be created if subspace A∩!B is not empty. Newly created vertices should have the edges to every element that (A)2 had the edge to. Substep 2.3.2. Vertex (B)2 should be removed and several new vertices should be created instead. Vertex (A∩B)2:B should be created in all cases and vertex (!A∩B)2 should be created if subspace !A∩B is not empty. Newly created vertices should have the edges to every element that (B)2 had the edge to. Substep 2.3.3. Vertices (A∩B)2:A and (A∩B)2:B should be merged to new vertex (A∩B)2. All the edges leading to either (A∩B)2:A or (A∩B)2:B should be redirected to (A∩B)2 instead. Substep 2.4. In situation set remove situations A and B and introduce all nonempty situations of these: A∩B, !A∩B, A∩!B. Step 3. If situation pair list is not empty – go to step 2. If situation pair list is empty – add new situation to the situations list: !S1∩!S2∩!S3∩!...∩!SN. Also add corresponding vertex to the dependency graph: (!S1∩! S2∩!S3∩!...∩!SN)2. It has no edges. Another option is not to mention !S1∩! S2∩!S3∩!...∩!SN situation and consider the system being in process of transition when no situation is triggered. ■ Resulting list of situations will now be referred to as new situation spaces. Resulting dependency graph will have old situation spaces in one set of vertices and new situation spaces in another. Edges between the vertices mean that old situation space has corresponding new situation space as a part of its decomposition. Algorithm has to be applied to the system only once – before the runtime. When this algorithm is complete, mutually exclusive set of situations is obtained. Now this
14
A. Boytsov, A. Zaslavsky, and K. Synnes
new set of situation spaces can be represented as states of the Markov model. All Markov model based context predictors are applicable for it. For illustration see Fig. 2. There are two options of Markov models for the system described in Fig. 1 and it depends on what option do we take on step 3 of an algorithm.
Fig. 2. Markov model for Fig.1
Reasoning in terms of old situation spaces can be easily done using new situation spaces. Let’s say, we need prediction for the situation Si from old situation spaces. It can be done in the following manner. Step 1. Find (Si)1 vertex. It is connected to some other vertices. Let them be (D1)2, (D2)2, (D3)2, … (Dk)2,etc. Step 2. Take predicted levels of confidence for those vertices. It is done using Markov predictor directly. Step 3. Infer the level of confidence of situation: P(Si) = P(D1U D2U D3U…U Dk) Where P(Si) – predicted level of confidence in situation Si, and sign U is a operator for uniting situations. Uniting situations can be done by the means of situation algebra provided by [17]. P(Si) is the demanded result. However, these 3 steps should be done on every case of prediction and therefore they will introduce some run-time overhead. So, Markov models seem to be feasible way of context prediction for context spaces theory. However, to make it applicable for context spaces some pre-processing is needed. Some computation overhead in the run-time will be introduced as well to transition from new situation spaces to old ones. Bayesian networks for context prediction. Bayesian network approach for context prediction was considered in the paper [21]. That research addressed the problem of predicting next room person is likely to visit. Context was represented as dynamic Bayesian network, where current room depended on several rooms visited previously and duration of staying in current room depended on current room itself, time of the day and week day. In [21] dynamic Bayesian network learned from the history of indoor movements. One more case of context prediction using Bayesian networks was described in [13]. There authors addressed active device resolution problem. [13] paper was already discussed in previous section.
Extending Context Spaces Theory by Predicting Run-Time Context
15
Context within context spaces theory can actually be represented by dynamic Bayesian network. In a simplest form it can allow to establish correlations between situations in time (and situations do not have to be mutually exclusive in this case). Dynamic Bayesian networks can also help if there are any additional suggestions about factors that influence the context but are not included directly in the application space. Branch prediction for context prediction. Paper [18] considered applying branch prediction algorithms for context prediction in pervasive computing systems. Branch prediction algorithms are used in microprocessors to predict the future flow of the program after branch instruction. In [18] authors consider another example: branch prediction is used to predict moving of the person around the house or office. Branch prediction algorithms can be applied to context spaces theory as well. Once again, context should be represented in terms of situations, and the situations should be represented as a graph. One option is to use mutually exclusive situation decomposition that was presented for Markov model predictors. Then we can predict moving through the graph over time using branch prediction techniques. Another option is to use the same approach like we took to apply sequence predictors: when situation happens or wears off, special symbol is added to the sequence. Branch predictors can be used to predict next symbol there. Summary. The features of different approaches to context spaces theory can be summarized in following way (see table 1). Table 1. Context prediction approaches summary
Approach Sequence predictors
Benefits and shortcomings Good results are proven in practice. Low adaptation efforts are needed.
Neural networks
Large variety of options to choose and low adaptation efforts needed. However, additional thorough testing is needed by every application to determine what exact neural network type is the most feasible. They can be applied. However, this approach requires splitting the application space in a set of non-overlapping situations, which is not natural for context spaces theory and requires significant pre-processing Approach is able to estimate the influence of different factors, not depending on whether they are in the context or not yet. Low adaptation efforts are needed. The algorithms are simple. However, prediction results are influenced only by small part of the history and the algorithms are capable of working with simple patterns only.
Markov chains
Bayesian networks Branch predictors
Summary This approach is the most perspective . The approach is quite perspective. The efficiency of this approach needs to be evaluated. The approach is very perspective. In general case that approach is least applicable.
16
A. Boytsov, A. Zaslavsky, and K. Synnes
Common challenges and future work directions. As it was shown all context prediction methods have to overcome some specific challenges to be applied to context spaces theory. But there is one common challenge, which was not yet addressed: all the defined context prediction methods deal with a discrete set of possible events or situations. So to apply the prediction methods context should be represented as a set of situations. But the flow of any prediction algorithm depends on how we define situation spaces within application space. Different situation spaces will result in different algorithm flows, even if application space is generally the same. This effect might be vital. So the question arises: what is the best way to define situations inside context space for prediction purposes? Results can depend on exact prediction algorithm or exact application. Situations should be both meaningful for the user and good for prediction. This question needs further research.
5 Testbed for Context Prediction Methods For testing context prediction methods identified above, a testbed is needed. Testbed application should include following features: 1. Widest range of context reasoning and context prediction methods should be applicable. 2. Testbed should be able to estimate the effectiveness of those techniques. 3. Testbed should be configurable and flexible. 4. Architecture of the testbed system must provide the way to replace context reasoning or context prediction method easily. 5. Conforming to the rules above, testbed model still has to be as simple as possible. For validating context prediction methods we introduce “Moonprobe” model and application scenario. “Moonprobe” is a model that simulates vehicle going over some sophisticated landscape in 2D environment. The aim of the vehicle is to reach some destination and not to crash or run out of fuel on its way. This model was developed using XJ Technologies AnyLogic modeling tool ([27]). “Moonprobe” testbed model represents a vehicle. That vehicle is affected by a group of forces (see formula (1)): Fres = Fgr + N + Fen .
(1)
Where: Fres – resulting force. It is the vector sum of all the forces affecting the moonprobe. Fgr – gravity force. It is constant and always affects the probe. Fen – engine force. Moonprobe has an engine, which can be used to dampen the fall or just keep going forward. Engine force is set by probe controlling device. Engine has limited fuel capacity. If the probe ran out of fuel engine force is constantly equal to zero. N – support reaction force. It affects the probe only if it is on the ground. Acceleration of the probe can be found with formula (2). A = (Fgr + N + Fen)/M Where M is the mass of the vehicle.
(2)
Extending Context Spaces Theory by Predicting Run-Time Context
17
In a very simplified manner the model can be described by the following set of equations (see formula (3)).
. |
(3)
|
Where t is time, X is probe coordinate vector, V is probe velocity vector, M is mass of the probe (known system parameter), N is support reaction vector (directed perpendicular to the ground), Fl – fuel level remained, g – gravity vector, FCR – fuel consumption rate (known system parameter), Fen – engine force vector (value and direction set by probe control system). In some cases probe can leave the ground (see Fig. 3). When probe lands, it will experience ground collision.
Fig. 3. Probe leaves the ground
Ground collisions are considered to be dangerous for the probe. Collision effect is estimated by the projection of probe speed perpendicular to the ground level. If it exceeds the threshold, the probe crashes. Engines can be used to slow down the landing and therefore avoid the crash. The “Moonprobe” model deals with the problems that every pervasive system deals with: sensor uncertainty, sensor outages, determining environment characteristics, fusing heterogeneous sensor data etc. By now the moonprobe uses around 20 context parameters to reason about the situations. Moonprobe has a set of situations to be aware of. They can be classified in several major groups. Group 1. Sensor outages. Any sensor can break down. Scenarios of sensor outages are configurable. The behavior of failed sensor is tunable as well, but the most convenient ways seem to be following: (1) Sensors start to show random values (2) Sensors keep returning the same value all the time not regarding to characteristic under measurement (3) Sensors increase their errors
18
A. Boytsov, A. Zaslavsky, and K. Synnes
Fig. 4. Moonprobe system architecture
Fig. 5. "Moonprobe" system working
Time between sensor outage and sensor outage detection can be used as a metric to estimate the quality of reasoning about that situation. Group 2. Current progress of going to the destination point. The probe needs to estimate whether it is going the right way (considering the fact not all the sensors might be working). Group 3. Safety conditions. The probe needs to estimate the level of crash risk. The architecture of the model is following (see Fig. 4). System consists of several components. • •
“Environment and probe” component. It implements the physical model of the system. Also it monitors what happens to the probe (crash, sensor outage etc.). “Sensors” component. Sensors measure the data from the probe in certain moments of time and introduce some errors.
Extending Context Spaces Theory by Predicting Run-Time Context
19
Fig. 6. Situation estimations in “Moonprobe”
• •
“Controlling engine” component. It takes sensed data and provides control information (e.g. desired engine power). “Main” component provides overall assessment of experimental results.
All the context prediction methods described in section 4 can be applied to moonprobe scenario. Context prediction techniques can be incorporated in “Controlling engine” component. There context prediction can be used to find optimal values of engine power to minimize crash risk and reduce fuel consumption. The development of the “Moonprobe” testbed is now complete. Screen dumps of running “Moonprobe” is depicted on Fig. 5(complete) and Fig.6 (partial). Model evaluation has shown that the “Moonprobe” model is really capable of estimating prediction results and that context prediction and context awareness algorithms are really pluggable into it. Exact measurements of the result of different context prediction methods are in progress.
6 Conclusion and Future Work Context spaces theory is a promising approach, which addresses many problems of context awareness in pervasive computing. Adding accurate and reliable context prediction methods is one of the research challenges of that theory. In this paper a set of available context prediction techniques was proposed. Twoway mapping algorithms were developed to apply these context prediction methods to context spaces theory. Efficiency of context prediction methods was addressed. A
20
A. Boytsov, A. Zaslavsky, and K. Synnes
special testbed was developed to evaluate the effectiveness of context prediction methods. Feasibility of the proposed testbed has been also proven. As a result, context spaces theory was enhanced with context prediction algorithms and their applicability and feasibility were demonstrated. By extending context spaces theory with context prediction techniques it can address not only problems of uncertain sensors or situation reasoning, but context prediction problem as well. It will benefit ECORA-based applications and widen the applicability of ECORA to new class of tasks – the tasks which require prediction capability. The proposed approach incorporates pluggable algorithms of context prediction, so for every application it can be defined what exact context prediction approach is the best. Future work will concentrate on accuracy metrics and analytical comparison of the efficiency of context prediction methods. Another interesting challenge for future work is how to define the situation spaces in a manner that both grants usability and enhances context prediction.
References 1. Ashbrook, D.: Learning significant locations and predicting user movement with gps. In: ISWC 2002: Proceedings of the 6th IEEE International Symposium on Wearable Computers, Washington, DC, USA. IEEE Computer Society, Los Alamitos (2002) 2. Albrecht, D.W., Zukerman, I., Nicholson, A.E.: Pre-sending documents on the WWW: A comparative study. In: IJCAI 1999 – Proceedings of the Sixteenth International Joint Conference on Artificial Intelligence (1999) 3. Bestavros, A.: Speculative data dissemination and service to reduce server load, network traffic and service time in distributed information systems. In: Proceedings of the 1996 International Conference on Data Engineering (1996) 4. Bhattacharya, A., Das, S.K.: Lezi update: An information-theoretic approach to track mobile users in PCS networks. In: Mobile Computing and Networking, pp. 1–12 (1999) 5. Cappe, O., Moulines, E., Ryden, T.: Inference in Hidden Markov Models. Springer, Heidelberg (2005) 6. Dey, A.K., Abowd, G.D.: Towards a better understanding of context and contextawareness. Technical report (1999) 7. Das, S.K., Cook, D.J., Bhattacharya, A., Heierman III, E.O., Lin, T.-Y.: The role of prediction algorithms in the MavHome smart home architecture. IEEE Wireless Communications 9(6), 77–84 (2003) 8. Davison, B.D., Hirsh, H.: Toward an adaptive command line interface. In: Advances in Human Factors/Ergonomics: Design of Computing Systems: Social and Ergonomic Considerations, pp. 505–508. Elsevier Science Publishers, San Francisco (1997); Proceedings of the Seventh International Conference on Human-Computer Interaction 9. Davison, B.D., Hisrch, H.: Probabilistic online action prediction. In: Proceedings of the AAAI Spring Symposium on Intelligent Environments (1998) 10. Gopalratnam, K., Cook, D.J.: Active Lezi: An incremental parsing algorithm for sequential prediction. In: Proceedings of the Florida Artificial Intelligence Research Symposium (2003) 11. Gellert, A., Vintan, L.: Person Movement Prediction Using Hidden Markov Models. Studies in Informatics and Control 15(1) (March 2006) 12. Haykin, S.: Neural Networks: A Comprehensive Foundation, 2nd edn. Prentice Hall, Englewood Cliffs (1999)
Extending Context Spaces Theory by Predicting Run-Time Context
21
13. Kaowthumrong, K., Lebsack, J., Han, R.: Automated selection of the active device in interactive multi-device smart spaces (2002) 14. Mozer, M.C.: The neural network house: An environment that adapts to its inhabitants. In: Proceedings of the AAAI 1998 Spring Symposium on Intelligent Environments, pp. 110–114. AAAI Press, Menlo Park (1998) 15. Mayrhofer, R., Radi, H., Ferscha, A.: Recognizing and predicting context by learning from user behavior. Radiomatics: Journal of Communication Engineering, special issue on Advances in Mobile Multimedia 1(1) (May 2004) 16. Nurmi, P., Martin, M., Flanagan, J.A.: Enabling proactiviness through Context Prediction. In: Proc. Workshop on Context Awareness for Proactive Systems, pp. 159–168. Helsinki University Press, Helsinki (2005) 17. Padovitz’s, A.: Context Management and Reasoning about Situations in Pervasive Computing, PhD Theses, Caulfield School of Information Technology, Monash University, Australia (2006) 18. Petzold, J., Bagci, F., Trumler, W., Ungerer, T.: Context Prediction Based on Branch Prediction Methods (July 2003) 19. Padovitz, A., Loke, S.W., Zaslavsky, A.: Towards a theory of context spaces. In: Proceedings of the Second IEEE Annual Conference on Pervasive Computing and Communications Workshops, 2004, pp. 38–42 (2004) 20. Padovitz, A., Loke, S.W., Zaslavsky, A.: The ECORA framework: A hybrid architecture for context-oriented pervasive computing. Pervasive and Mobile Computing 4(2), 182–215 (2008) 21. Petzold, J., Pietzowski, A., Bagci, F., Trumler, W., Ungerer, T.: Prediction of Indoor Movements Using Bayesian Networks. In: Strang, T., Linnhoff-Popien, C. (eds.) LoCA 2005. LNCS, vol. 3479, pp. 211–222. Springer, Heidelberg (2005) 22. Roy, A., Das Bhaumik, S.K., Bhattacharya, A., Basu, K., Cook, D.J., Das, S.K.: Location aware resource management in smart homes, pp. 481–488 (2003) 23. Russell, S.J., Norvig, P.: Artificial Intelligence: A Modern Approach, 2nd edn. Prentice Hall, Englewood Cliffs (2003) 24. Shafer, G.: A Mathematical Theory of Evidence. Princeton University Press, Princeton (1976) 25. Vintan, L., Gellert, A., Petzold, J., Ungerer, T.: Person Movement Prediction Using Neural Networks. In: First Workshop on Modeling and Retrieval of Context, Ulm, Germany (September 2004) 26. Ziv, J., Lempel, A.: Compression of individual sequences via variable-rate coding. IEEE Transactions on Information Theory 24(5), 530–536 (1978) 27. How to Build a Combined Agent Based / System Dynamics Model Logic in AnyLogic (2008), http://www.xjtek.com/file/161
Cross-Domain Interoperability: A Case Study Jukka Honkola, Hannu Laine, Ronald Brown, and Ian Oliver Nokia Research Center P.O. Box 407 FI-00045 NOKIA GROUP
[email protected],
[email protected],
[email protected],
[email protected]
Abstract. We describe a case study of the behaviour of four agents using a space based communication architecture. We demonstrate that interoperability may be achieved by the agents merely describing information about themselves using an agreed upon common ontology. The case study consists of an exercise logger, a game, a mood rendered attached to audio player and phone status observing agents. The desired scenario emerges from the independent actions of the agents.
1 Introduction Interoperability between devices and the software and applications that run on those devices is probably the major goal for many developers of such systems. This can be achieved in a number of ways: particularly through open, agreed standards or often through monopoly. Whichever particular ‘interoperability’ route is taken there will always be a plurality of standards relating to how such systems communicate [6][1]. For example, the Universal Plug and Play interoperability standard is plagued with manufacturer and device specific variations which complicate and nullify to some extent the long and complex standardization process employed to avoid this. Technologies such as the Semantic Web [2] provide enablers for solutions to these problems. Within the Semantic Web exist information representation formats such as the Resource Description Framework (RDF) [10] and the web ontology language OWL [9] which themselves build upon enablers such as XML. Using these technologies we can address and provide solutions1 to the interoperability problem. At one level there are existing solutions such as Web Services (SOAP, WSDL, UDDI etc) and various stacks of middleware for processing representation formats and preservation and reasoning about the semantics of messages. At the other, there are solutions based around more declarative mechanisms such as TripCom [11] and the space-based solutions, for example Java Spaces [4] for the transport and processing of messages and information to assist in interoperability. We do not believe that interoperability will be achieved through standardization committees, nor through standard, globally accepted semantics and ontologies but rather by the unification of semantics in a localized manner, as described in previous work [8]. 1
Plural, never singular!
S. Balandin et al. (Eds.): NEW2AN/ruSMART 2009, LNCS 5764, pp. 22–31, 2009. c Springer-Verlag Berlin Heidelberg 2009
Cross-Domain Interoperability: A Case Study
23
In this paper we describe an approach to solving the interoperability problem through a combination of context gathering, reasoning and agents within a space-based infrastructure taking advantage of technologies such as RDF. We demonstrate this through the ad hoc integration or mash-up of a number of distinct applications to demonstrate the principles of this approach.
2 The M3 Concept The M3 system consists of a space based communication mechanism for independent agents. The agents communicate implicitly by inserting information to the space and querying the information in the space. The space is represented by one or more semantic information brokers (SIBs), which store the information as an RDF graph. The agents can access the space by connecting to any of the SIBs making up the space by whatever connectivity mechanims the SIBs offer. Usually, the connection will be over some network, and the agents will be running on various devices. The information in the space is the union of the information contained in the participating SIBs. Thus, the agent sees the same information content regardless of the SIB to which it is connected. The high-level system architecture is shown in Figure 1.
Agent Agent
Agent
Agent
M3 Space SIB SIB SIB
Fig. 1. A diagram of the general system architecture: Agents, M3 spaces and SIBs
The agents may use five different operations to access the information stored in the space: Insert : Insert information in the space Remove : Remove information from the space Update : Atomically update the information, i.e. a combination of insert and remove executed atomically Query : Query for information in the space Subscribe : Set up a persistent query in the space; changes to the query results are reported to the subscriber
24
J. Honkola et al.
In addition to the four access operations there are Join and Leave operations. An agent must have joined the space in order to access the information in the space. The join and leave operations can thus be used to provide access control and encrypted sessions, though the exact mechanisms for these are still undefined. In its basic form the M3 space does not restrict the structure or semantics of the information in any way. Thus, we do not enforce nor guarantee adherence to any specific ontologies, neither do we provide any complex reasoning2. Furthermore, information consistency is not guaranteed. The agents accessing the space are free to interpret the information in whatever way they want. We are planning to provide, though, a mechanism to attach agents directly to the SIBs. These agents have a more powerful interface to access the information and can be e.g. guaranteed exclusive access to the information for series of operations. Such agents may perform more complex reasoning, for example ontology repair or translation between different ontologies. However, they may not join any other spaces but are fixed to a single SIB and thus a single space. The M3 spaces are of local and dynamic nature, in contrast to semantic web which embodies Tim Berners-Lee’s idea of semantic web [2] as a “giant global graph”. The locality and dynamicity—we envision that the spaces will store very dynamic context information, for example—poses different challenges than the internet-wide semantic web. For example, in order to provide a true interoperability for local ubiquitous agents, the space (i.e. SIBs) will have to provide a multitude of connectivity options in addition to http: plain TCP/IP, NoTA [7], Bluetooth,. . . Furthermore, the space should be fairly responsive. While we do not aim for real-time or near real-time system, even half minute long response times for operations are unacceptable. The responsiveness is one of the factors behind the fundamental decision to not enforce any specific ontologies and allowing the agents to interpret the information freely, as it lessens the computational burden of the infrastructure. Another, and more important reason is that we explicitly want to allow mashing up information from different domains in whatever way the agents see best. Strict ontology enforcement would make this kind of activity extremely difficult as all new ways of mashing up the information would require approval from some ontology governance committee. However, as mentioned above, we still plan to provide means for ontology enforcement for cases where the space provider explicitly wishes to restrict the ways the information is used as there are bound to be also situations where this is the best approach. The information content in a M3 space may be distributed over several SIBs. The distribution mechanism assumes that the set of SIBs forming a M3 space are totally routable but not necessarily totally connected. The information content that the agents see is the same regardless the SIB where they are connected. 2.1 Applications in M3 Spaces The notion of application in M3 space is differs radically from the traditional notion of a monolithic application. Rather, as a long term vision, we see the applications as possible scenarios which are enabled by certain sets of agents. Thus, we do not see an 2
The current implementation of the concept understands the owl: sameAs concept.
Cross-Domain Interoperability: A Case Study
25
email application running in M3 space, but we could have a collection of agents present which allow for sending, receiving, composing and reading email. For this kind of scenario based notion of application, we also would like to know whether the available agents can succesfully execute the scenario. The envisioned model of using this system is that the user has a set of agents which are capable of executing certain scenarios. If a user needs to perform a new scenario that the current set of agents are not capable of executing, she could go and find a suitable agent from some directory by describing the desired scenario and the agents she already has. Thus, we need some formal or semi-formal way of describing agent behavior both with respect to the M3 space and to the environment. While there exists research addressing behavior in multi-agent systems, for example by Herlea, Jonker, Treur and Wijngaards [5], this kind of ad-hoc assembly of agents in order to execute a certain scenario seems to be quite unaddressed in current research. However, slightly similar problems have been addressed in e.g. web service orchestration research [3], but these still seem to concentrate on design-time analysis rather than run-time analysis. As for shorter term, our vision is that sets of existing applications would be enhanced by being able to interoperate and thus allow execution of (automatic) scenarios that would have been impossible or required extensive work to implement without the M3 approach.
3 The Case Study The case study consists of four agents, Exercise logger, SuperTux game, Phone line observer, and Mood renderer, running on several devices. The exercise logger and phone line observer run on Nokia S60 platform phones, and the SuperTux and mood renderer run on Nokia N800 internet tablets. The M3 space infrastructure runs on a Linux laptop. 3.1 Application Development Vision The case study setup illustrates a “legacy enhancement” approach to application development in M3 spaces. Conceptually, we added M3 space capability to existing applications (SuperTux, Mood renderer, SportsTracker, Telephony) even if the SportsTracker and Telephony were implemented as standalone simple applications for practical reasons. The starting point for the case study was a single person (as defined in Person ontology, section 3.2) who would be exercising and tracking the workouts with an exercise logger, playing computer games and listening to music. The goal was to demonstrate the benefits of interoperability between existing application, that is, to be able to execute a scenario where the SuperTux game would award extra lives for exercising, a mood renderer embedded in a media player would play suitable music depending on a game state, and the game and media player would react accordingly if the person receives a call. 3.2 Ontologies The ontologies used in this case study have been defined informally, and are not enforced in any way. The different components of the case study assume that they are used correctly. A pseudo-UML diagram of the complete ontology is shown in Figure 2,
26
J. Honkola et al. SuperTux, Mood renderer, Telephony status observer
SuperTux, Mood renderer
OperatorAccount
Game m3:sessionstart m3:sessionend m3:lives
m3:phonelinestatus
m3:player
m3:mobileaccount
All
SuperTux, Workou monitor
m3:user
Workout
Person
m3:totaldistance m3:totaltime m3:stoptime m3:username m3:starttime
m3:username
Fig. 2. An informal description of used ontologies. The using agents are named on top of the “class” box. m3:person m3:workout
m3:supertux
rdf:type rdf:type ?1
m3:user
m3:player
rdf:type
m3:username
m3:starttime ?3
?2 m3:sessionstart
"SportsTrackerUser" m3:lives
m3:totaltime m3:mobileaccount
"4" "2008−10−30T11:30:02"
?4 m3:phonelinestatus "idle"
"2008−10−29T07:02:30"
m3:stoptime "2008−10−29T08:05:12" m3:totaldistance "2053132"
"3738786"
rdf:type m3:operatoraccount
Fig. 3. An example RDF graph during the execution of scenario
and an example instantiation of the ontology when a game is being played and the phoneline is idle is shown in Figure 3. As a general note, the needs of the scenario studied drove the modeling of the ontologies. Most of the choices of what to include and what to exclude were based on the need of the information in executing the chosen scenario. A broader study might need to revisit the ontologies and generalize end expand them. The agents use different parts of the combined3 case study ontology. The Workout Monitor agent understands Workout and Person ontologies, the SuperTux agent understands all ontologies, the Telephony Observer agent understands OperatorAccount and Person ontologies, and the Mood Renderer understands Game and OperatorAccount ontologies. 3
A combination of Person, OperatorAccount, Game, and Workout ontologies.
Cross-Domain Interoperability: A Case Study
27
Person. The person ontology describes the information about a person necessary for this case study. A person can have a name (m3:username) and an operator account (m3:mobileaccount). Operator Account. The operator account ontology contains information about the phoneline status of the account. In a more realistic case, there could also be information about the phone number, used talking time, amount of data transferred etc. Workout. The workout ontology describes a generic distance-oriented workout. Thus, we include information about the length of the workout, distance traveled, person who has done the workout etc. but no information related to, for example, weightlifting where we probably would like to know the number of repeats and weights instead of time used and distance. The choice of what to include in the ontology was influenced by the information available in the SportsTracker application. Game. The Supertux game ontology includes information about the gaming session start time, stop time, lives left and player. The session start is defined to be the time when the game executable was started, showing a welcome screen. The lives value is inserted when a player starts a game, new or saved, from the welcome screen. When the player quits the game completely, that is, exits the welcome screen, the session stop time is inserted. The ontology could include also information about current score, amount of coins, field being played etc. which the mood renderer could use. 3.3 Agents Next we describe the operation of agents making up the case study. It should be noted that the agents are only communicating implicitly through the M3 space by reading the information inserted to it by other agents. Workout Monitor. The workout monitor reads SportsTracker workout files and writes a summary of the workouts to M3 space. We currently read the information related to general description of the workout and insert that in the M3 space. Any path information is ignored as we have no use for it in the current scenarios. The workout information is related to the person doing the workouts. If a person with same name as the SportsTracker username is found, we attach the workout in question to that person, otherwise we create a new person. We use an SportsTracker internal workout identifier to identify workouts so that we do not insert duplicate workouts if the monitor is run again with same workouts as before. Supertux. Supertux is a classic jump’n’run sidescroller game which has taken strong inspiration from the original SuperMario games for Nintendo. Supertux is open source and available under GPL. The ontologies related to Supertux game are the Person ontology, the Workout ontology and the Supertux ontology. Each Supertux game is assigned with player information. If the person class instance of the player can not be found from the M3 space the
28
J. Honkola et al.
game creates one and inserts player information according to the Person ontology. The person class instance also binds the player to the workout information inserted to the M3 space. Prior starting the game supertux queries workout information from the M3 space. The first query searches for all instances of the Workout class. For each found workout class instance a new query is performed in order to find out the details of the workout. If there are long and recent enough workouts performed by the player the game awards two extra lives to a game level. The game creates also an instance of itself and inserts information such as reference to the person class of the player, session start and stop time to the M3 space. In addition, the game updates how many lives the player has left during playing a game level. Every time the player looses or gains one life in the game, the game removes old livesinformation and inserts the new information to the M3 space. Telephony status observer. The telephony voice line status is part of a OperatorAccount class. Operator accounts are bound to a person i.e. it is assumed that each account belongs to a person4 . The voice line status only has two states, namely active and idle. The phone line status monitor subscribes to voice line status changes provided by Symbian’s CTelephony API. When the user dials a phone number to establish voice connection or when there is incoming call to the phone, CTelephony API provides the current voice line status to the observer application. The observer application transforms the CTelephony states (e.g. EStatusRinging, EStatusDialling, EStatusIdle, . . . ) to corresponding values defined by the OperatorAccount ontology and updates the information to the M3 space. Mood renderer. A mood renderer conceptually would have interfaces to sensory, i.e. mood rendering devices of a smart space invironment in order to cooridinate their controls according to a mood determined by its knowlege processing. The questions would be many about such knowlege for determing a mood and how to render it for a given environment. The mood renderer of the case study is a very simple start on the problem, which side-steps many issues for practical cases, while showing the essence of knowlege processing for the given concept. The mood renderer in this case study is based on music as the only rendering device, and thus, the simple assumption that music, more precisely some audio track, maps to a determinable mood, and still further, that the mapping stems solely from the state of a partical game being played and applies invariently with regard to time. This dodges questions about personal tastes and more than one person contributing to the smart space mood. In choosing a track to play, volume is not controlled, but play, pause and stop are used where appropriate. Mood rendering is based solely on two sources of information in the smart space environment. The first is an operating supertux game and the second is phone status. Note then, that this mood renderer is solely reactive to information about a dynamic smart space environment to which it belongs—for simplicity, it does not contribute information even though it clearly impacts it. Given the simplifications stated, and that the mood renderer understands the ontologies of the supertux game and telephony status observer. It follows their two independent 4
Or a group of persons—we do not prevent that.
Cross-Domain Interoperability: A Case Study
29
contributions to the smart space environment and combines them to render a mood appropriate for the state of game play, while maintaining the courtesy of pausing an audio track, if being played while any phone status of the smart space is active. The mood renderer is implemented as two persistent queries for smart space information. The first is looking for an active game in order to render its mood, and the second is looking for any phone with an active status. When an active game is found, a third persistent query, particular to this game, looks for its level of m3:lives and renders a mood as CALM, for no lives before the real play starts, and then when m3:lives exist, either “UP-BEAT” or “DOWN-BEAT” according to a threshold configuration on the number of lives. When the game ends, audio rendering is stopped. The mood renderer’s behavior can be described as follows: – Start with renderer.setTrack (NONE), renderer.setMode (STOP), and gameSelected=FALSE. – Subscribe to the call-status attribute for all phone-status-observers. Monitor subscription results, for any attibute value of “active”, if currentMode is PLAY, then do renderer.setMode (PAUSE); otherwise, if currentMode is PAUSE, do renderer.setMode (PLAY) i.e. any playing pauses during any phone call, or resumes when no phone call if paused during a game. – Subscribe to a supertux game, which has a session-start and no session-end, or session-start after session-end. Monitor subscription results, and, whenever fulfilled, first ensure that rendering is stopped and end subscriptions for any earlier game. Then render music/mood mapping for the selected game using the next subscription. – Subscribe to session-end and lives attributes of the selected game. When fulfilled, set renderer.setMood (mood), for mood according to lives as follows: • no lives: CALM i.e. game started, but not yet played. • lives above threshold: UP-BEAT • below threshold: DOWN-BEAT When session-end of game is after session-start, stop rendering and end subscriptions about this game.
4 Conclusions The idea of agents interoperating by communicating implicitly through M3 spaces worked well for this case study. We were able to execute the scenario in mind and add the phone agent to it without modifying the existing ontology. However, this work was performed inside a single team in a single company. This kind of close cooperation between people implementing the agents obviously makes things easier. Most of the work required in integrating the different agents into a coherent scenario was related to the mood renderer observing the state of the SuperTux game. The ontology did not directly provide information about all the state changes in the game. For example, when starting to play a level in the game, the mood renderer deduce this by observing the appearance of the lives attribute in the M3 space. This is however something that will probably be a very common case as the ontologies can not and
30
J. Honkola et al.
should not try to describe all possible information that some agent may need. It should suffice that the information is deducible from the explicitly stated information. Another aspect of local pervasive computing environments that was not really addressed in this case study was the handling of resources that can only be used by one agent at a time. In this case the mood renderer represents such resource (audio player), and implicitly it is usually the game that uses it as the mood renderer deduces the mood based on the game state alone. When the phone rings, the mood renderer will stop the music and continue after the call is finished. However, in a more complex case there needs to be (preferably) clear policy for each resource on who can use it. We currently think that this can be handled at the ontology level, with suitable coordination structures and an agent acting as a gatekeeper. In general, this is however a subject for further study. The basic principles of agent-dependent interpretation of information and implicit communication proved useful when the phoneline status information was added to the case study. When we only describe the information and not the actions to be taken by the agents, the agent can select suitable course of action based on the detailed knowledge of the agent in question. In this particular case, the correct course of action for the SuperTux game is to pause when the phoneline is active but not automatically resume the game when it goes idle, as the player is not necessarly ready to continue immediately after ending the call. On the other hand, the mood renderer should continue playing the music immediately after the distraction (in this case call) has ended. If we expanded the case study with more persons with each having mobile phones, the difference with mood renderer and SuperTux regarding the behavior when a call arrives would be even more. Sensible behavior for mood renderer would be to stop the music whenever anyone in the same (acoustic) location receives a call, while the game should only react to calls to the player of the game. Of course, in some cases it might be desirable to have coordinated response from a group of agents to certain events. We believe that this is best handled outside the system, by e.g. requirements in the scenario descriptions or ontology documentation. Anyhow, such coordinated responses would not apply to all agents but rather to only certain groups of agents involved in executing a given scenario. An obvious next step related to the interoperability hypothesis is to have external research partners extend the demo with agents implemented outside Nokia. This would clarify the required level of ontology and agent behavior description as the communication would then approximate more the real world case where the agents would be implemented by multiple vendors.
Acknowledgements The authors would like to thank Vesa Luukkala for helping with ontology modeling, and Antti Lappetel¨ainen for helpful comments.
References 1. Bergamaschi, S., Castano, S., Vincini, M.: Semantic Integration of Semi-structured and Structured Data Sources. SIGMOD Record 28(1), 54–59 (1999) 2. Berners-Lee, T., Hendler, J., Lassila, O.: The semantic web. Scientific American (May 2001)
Cross-Domain Interoperability: A Case Study
31
3. Foster, H., Uchitel, S., Magee, J., Kramer, J.: Compatibility verification for web service choreography. In: Proceedings of IEEE International Conference on Web Services, July 6-9, pp. 738–741. IEEE, Los Alamitos (2004) 4. Freeman, E., Hupfer, S., Arnold, K.: JavaSpaces Principles, Patterns and Practice. AddisonWesley, Reading (1999) 5. Herlea, D.E., Jonker, C.M., Treur, J., Wijngaards, N.J.E.: Specification of Bahavioural Requirements within Compositional Multi-agent System Design. In: Garijo, F.J., Boman, M. (eds.) MAAMAW 1999. LNCS, vol. 1647, pp. 8–27. Springer, Heidelberg (1999) 6. Lewis, G.A., Morris, E., Simanta, S., Wrage, L.: Why Standards Are Not Enough To Guarantee End-to-End Interoperability. In: Proceedings of 7th IEEE International Conference on Composition-Based Software Systems (ICCBSS), Madrid, Spain, February 25–29. IEEE, Los Alamitos (2008) 7. Network on terminal architecture (November 2008), http://www.notaworld.org 8. Oliver, I., Honkola, J.: Personal Semantic Web Through A Space Based Computing Environment. In: Second IEEE Interntional Conference on Semantic Computing, Santa Clara, CA, USA, August 4-7. IEEE, Los Alamitos (2008) 9. Web ontology language, http://www.w3.org/2004/OWL/ 10. Resource description framework, http://www.w3.org/RDF/ 11. Teymourian, K., Nixon, L., Wutke, D., Krummenacher, R., Moritsch, H., K¨uhn, E., Schreiber, C.: Implementation of a novel semantic web middleware approach based on triplespaces. In: Second IEEE International Conference on Semantc Computing, Santa Clara, CA, USA, August 4-7, pp. 518–523. IEEE, Los Alamitos (2008)
A Topology Based Approach for Context-Awareness in Smart Environments Antonio Coronato and Giuseppe De Pietro Institute of High Performance Computing and Networking ICAR-CNR, Via P. Castellino 111, 80131, Naples - Italy
[email protected],
[email protected]
Abstract. This paper presents logic models and mechanisms to achieve context awareness and future context awareness. The proposed approach relies on handling the topology of the physical environment. We describe methods and tools to specify and implement topology services that can provide mechanisms for both rapid prototyping application services and handling context awareness information. Finally, we report some case studies and applications of the proposed approach1 .
1
Introduction
Ambient intelligence and Smart Environments are emerging discipline and technologies that brings intelligence to our common physical environments. Such intelligent environments relies on the use of a large variety of sensors and sensor networks, smart objects and devices, artificial intelligence technologies, etc. The resulting environments are systems capable of knowing and understanding the physical and virtual context of users and objects and responding intelligently to such knowledge [3]. This paper proposes logic models ed implementation mechanisms to achieve context-awareness in smart environments. All solutions are based on the physical topology of the environment. The logic model relies on the classification of locations in two classes: semantic (Rooms, Corridors, Laboratories, Floors, etc.) and physics (the area sensed by a short range RFID, the one sensed by a temperature sensor, the one covered by a WiFi access point, etc.). Since the physical locations are the ones sensed by sensors and positioning systems, whereas context-aware services and applications need semantic locations (e.g. in which room is the user? Where is he going to? Which is the temperature of the laboratory? etc.) we have defined a logic model that joins semantic locations and physical locations by means of a virtual concept that we named Atomic Location. In addition to this, we have defined logic models to represents properties like the contiguity and distance of locations. This makes the environment capable of inferring several kind of situations. 1
This work has been partly supported by the project OLIMPO within the POR Misura 3.17 funded by the Regione Campania.
S. Balandin et al. (Eds.): NEW2AN/ruSMART 2009, LNCS 5764, pp. 32–41, 2009. c Springer-Verlag Berlin Heidelberg 2009
A Topology Based Approach for Context-Awareness in Smart Environments
33
The logic models are only part of the contribution of this paper. In addition, we have also provided a set of formal tools that enable the designer to specify the topology and some dynamic aspects in a rigorous and verifiable way and a set of ontologies and rules that enable the developer to implement intelligent mechanisms for context-awareness. The choice of formal tools is due to the fact that smart environments and ambient intelligence are more often applied to critical scenarios (e.g., healthcare, emergency situations, disaster recover, etc.). As well known from literature, they represent an effective tool to prevent faults [3]. The rest of this paper is organized as follows. Section 2 presents the logic models. Section 3 describes the formal tools that we apply to specify the topology. Section 4 presents the ontology on which relies our context-service and the rules that make it able to infer context information. Finally, Section 5 concludes the paper.
2
Logic View
The approach presented in this paper relies on a topology model that we have defined to provide a unique and uniform representation for location information, independently of the particular positioning system. The model is based on the concepts of physical and semantic locations. These notions are not new in literature [2], but we have partially re-elaborated them [5]. A physical location specifies the position of a mobile entity and is characterized by different granularities and scopes, depending on the particular positioning system adopted. Instead, a semantic location specifies the meaning of a location and usually groups more physical locations. As an example, GPS coordinates represent physical locations, whereas a semantic location can be a building, an office inside a building, a meeting room, an area in front of a wall-monitor, etc. The proposed model describes physical locations as the proximity to wellknown points. The technique of proximity is applicable when the environment is equipped with sensors to reveal the presence of mobile users, or with positioning systems to detect the position of mobile devices. In such cases, a sensor/positioning device, which covers a specific area, defines a physical location corresponding with the covered area. In the model, the covered area is called SensedArea. Let us show an example with two specific positioning systems. The former relies on the Wi-Fi technology to locate Wi-Fi enabled mobile devices. The latter uses RFID identification systems to locate RFID tagged mobile users. Corresponding sensed areas are: – Wi-FiSensedArea, which is identified by the physical area covered by a specific Wi-Fi AP, i.e. the region in which the AP is able to grant access to a Wi-Fi mobile device; – RFIDSensedArea, which is identified by the physical area covered by a specific RFID reader; i.e., the region in which the RFID reader is able to detect an RFID tagged user.
34
A. Coronato and G. De Pietro
In addition to these, a set of semantic locations, like Building, Floor, Room and Corridor, have been defined. Finally, the concept of AtomicLocation is used to represent the linking element between physical and semantic locations. Its physical dimension corresponds with the one of the SensedArea of the finest positioning system in the environment. In the proposed example, an AtomicLocation corresponds with an RFIDSensedArea because an RFID reader covers a smaller area than the one covered by a Wi-Fi AP. As shown in figure 1, the concept of AtomicLocation is used to build a grid of locations in the physical environment (AtomicLocation Perspective). Therefore, some RFID readers are positioned in order to match with AtomicLocations. Not all the AtomicLocations are covered by an RFIDReader (RFIDSensed Perspective). Differently, each Wi-Fi AP spreads a signal over many AtomicLocations (Wi-FiSensedArea Perspective). Finally, semantic locations in turn group many atomic locations and correspond with well defined physical spaces (SemanticLocation Perspective). As a result, relationships between sensed areas and semantic locations are built by means of atomic locations. This makes the model able to adapt easily
Fig. 1. Logic Models
A Topology Based Approach for Context-Awareness in Smart Environments
35
Fig. 2. Contiguity relationships between locations
to changes like the introduction of new positioning systems or the definition of new semantic locations. The introduction of a new positioning system obviously means that the physical dimension of an atomic location has to be changed, but the location model still remains applicable. This model provides some mechanisms that enable the conferring of semantic to some information. As an example, in addition to the localization problem that has already been discussed, let us focus on the setup of sensors, for example a temperature sensor. Let us also suppose that there’s the environment is equipped with a topology service that handles the topology model. When the sensor is installed, the topology service is updated with information on i) the new resource; ii) its location; and, iii) its covered area. From then on, all information coming from that sensor are associated to the covered area; that is, the sensed temperature is the temperature of the covered area. Another aspect, which is relevant to our aims, is the contiguity of locations. We say that two locations (L1 and L2 ) are contiguous if it is possible to move from L1 to L2 directly and viceversa. The relation of contiguity among locations is logically represented by bi-directional graphs as the one shown in figure 2. In the graph, the weight of each arch, d(x,y), represents the distance among the locations Lx and Ly or the minimum time a mobile entity needs to move from the starting location to the destination. This model is useful for tow main reasons. First, the system can predict the future position of a moving user. For an example, in a museum the visitor who is in a room will next be in one of the contiguous rooms. As a result, the system can prepare its resources depending on the future position. Second, this can help in detecting and avoiding inconsistencies. As a matter of fact, a tracking system can easily detect a large set of false positive using the concepts of distance and time.
3
Specification View
In this section we briefly describe some formal tools adopted to specify the topology of a smart environment and several dynamic aspects.
36
3.1
A. Coronato and G. De Pietro
Ambient Calculus
Ambient Calculus was proposed by Luca Cardelli and Andrew Gordon to describe the movement of processes and devices. It is a framework that derives from π-calculus [9]. In Ambient Calculus, all entities are described as processes. An important kind of process is an ambient, which is a bounded place where computation happens. An ambient delimits what is and what is not contained and executed within it. Each ambient has a name and can be moved as a whole, into and out of other ambients, by performing in and out operations. Ambient Calculus allows a description of complex phenomena in terms of creation and destruction of nested ambients, and movement of processes into and out of these ambients. In Ambient Calculus, n[P] represents an ambient, n is the name of the ambient, and P is the process running inside it. The process “0” is the one that does nothing. Parallel execution is denoted by a binary operator “|” that is commutative and associative. “!P” denotes the unbounded replication of the process P within its ambient. The following expression n[P1 | . . . | Pj | m1 [...] | . . . | mk [...]]
describes an ambient with the name, n, that contains j processes with the names P1 , ..., Pj and k ambients with the names m1 , ...mk . Figure 3 reports the equivalent graphic representation.
Fig. 3. Ambient Calculus: Graphical Representation for Ambients
Some processes can execute an action that changes the state of the world around them. This behavior of processes is specified using capabilities. The process M.P executes an action described by the capability M, and then continues as the process P. There are three kinds of capabilities, for entering (in), for exiting (out ) and for opening up (open) an ambient, respectively. Figure 4 shows these three capabilities. The process “in m.P” instructs the ambient surrounding “in m.P” to enter a sibling ambient named m. The reduction rule, which specifies the change in state of the world is: n[in m.P] | m[Q] → m[n[P] | Q] (a)
The action “out m.P” instructs the ambient surrounding “out m.P” to exit its sibling ambient named m. The reduction rule is: m[n[out m.P] | Q] → n[P] | m[Q] (b)
A Topology Based Approach for Context-Awareness in Smart Environments
37
Fig. 4. Ambient Calculus: Capabilities
The action “open m.P” provides a way of dissolving the boundary of an ambient named m located at the same level as open. The rule is: open m.P | m[Q] → P | Q (c)
Output and input actions are respectively denoted by “M.P” and “(x).P”, to release and capture capabilities. Finally, the restriction operator (Vn)P creates a new (unique) name n within a scope P. The new name can be used to name ambients and to operate on ambients by name. 3.2
Ambient Logic
Ambient Logic is a temporal (modal) logic that allows the expression of properties with respect to a specific location (ambient). Logical formulas in ambient logic include i) those defined for propositional logic, like negation (¬), disjunction (∧), and conjunction (∨); ii) universal (∀) and existential (∃) quantifications, as in first-order-logics; iii) eventually (3) and always () temporal operators, as in temporal logics; and, in additionto classic logics, iv) spatial operators such as somewhere ( ) and everywhere ( ). The satisfaction relation P |= A means that the process P satisfies the closed formula A (i.e. A contains no free variables). More specifically, the formula P |= A@L means that the process P satisfies the closed formula A at the location L. In the temporal modality, the relation P → P0 indicates that the ambient P can be reduced to P0 using exactly one reduction rule. Similarly, in the spatial modality, the relation P ↓ P0 indicates that the ambient P contains P0 within exactly one level of nesting.
38
3.3
A. Coronato and G. De Pietro
Real-Time Temporal Logic
Despite their name, temporal logics (e.g. Ambient Logic) do not provide mechanisms for reasoning about “time” in a quantitative sense; i.e. they do not enable you to express real-time constraints. For this reason, several extensions to temporal logics have been proposed [1], which fall into two categories: explicit time logics or implicit time logics. For the rest of the paper, we will refer to the Real-Time Temporal Logic (RTTL), which is an explicit time formalism that relies on the use of a special variable T to indicate the value of time at any state and use first-order quantification over that variable to make real-time assertions. With this logic, the formula: ∀ t ≥ T0 : (α ∧ t=T )
means that for any instant after T0 , α is always true. 3.4
Application of Such Formal Tools
This sub-section shows how to apply the formal tools presented so far. In particular, as first it should be provided a structural view of the topology. To achieve this, we use Ambient Calculus, each location being an ambient. Indeed, we can model a room as an ambient of ambient calculus. The room contains resources that are steel described as ambients and is located at a floor, which is also an ambient. Next, we can describe the movements that mobile entities (users and resources) can perform within the environment by means of the in and out capabilities (users are modeled as ambients as well). Finally, constraints regarding the movements within the environment and the temporal duration of movements can be specified using Ambient Logic and the Real Time Temporal Logic respectively.
4
Implementation View
Our implementation relies on the use of Formal Ontology and reasoners. In literature one can find many definitions of ontology. For the purposes of this work, we are interested in ontology as a formal tool for modeling domains [6]. Main elements to take into account in an ontology are classes, sometimes called concepts, which represent entities of the domain being modeled, and relations among classes. A class can have subclasses that represent concepts, which are more specific than the superclass, and instances, which are elements of the real world classified as members of a class. In practical terms, developing an ontology includes: 1. 2. 3. 4.
defining the classes in the ontology; arranging the classes in a taxonomic (subclass–superclass) hierarchy; defining the relations; figuring out relations among classes and entities.
A Topology Based Approach for Context-Awareness in Smart Environments
39
In the proposed methodology, ontology is used to define, in a formal and unambiguous way, all the entities (resources, services, users, etc.) involved in the application domain being designed, and the relations among them. Relations are formally defined as primitives [6] by specifying their meaning and what mathematical properties (reflexive, transitive, commutative, etc.) they satisfy. Figure 5 presents an example of ontology.
Fig. 5. Ontology
It is quite obvious that a Building contains one or more Floors, which, in turn contain Rooms, Corridors, Laboratories, etc. These are semantic locations that are related to physical locations through Atomic Locations. Thus, physical locations like RFID Sensed Area, WiFi Sensed Area, Temperature Sensed Area, etc, as well as semantic locations, maps some atomic locations. It is important to note that other concepts, not reported in the figure, are related to atomic locations. Indeed, each sensor (e.g. RFID Reader, WiFi Accecc Point, Temperature Sensor, etc.) is physically installed in a specific atomic location. At the same way, a User moves from atomic locations to atomic locations. The difference between the relations contains and maps is that the latter one doesn’t imply a perfect correspondence between the locations. In other words, we can say, for example, that a WiFi sensed area contains (maps) a specific atomic location with a certain degree of approximation due to the variability of the electromagnetic fields. On the contrary, a floor contains completely a room without uncertainty. The relation isContiguous has already been defined in section 2. On top of the ontology it is possible to specify several rules that enable the system to infer context information. For example, rules can represent the logic and reasoning mechanisms the environment utilizes to choose the location information with the most proper level of granularity when a mobile object is located
40
A. Coronato and G. De Pietro
Fig. 6. Rules
by more than one positioning system. To achieve this aim, we report in figure 6 two SWRL rules we have realized, built on the previous ontology. In detail, the first rule makes the environment able to conclude that, if a mobile entity is sensed by an RFID reader, then it is located in the atomic location associated with the sensed area covered by that RFID reader. The second rule enables the environment to conclude that, if a mobile entity is sensed by a Wi-Fi access point and is not sensed by an RFID reader, then it is located in the atomic location associated with the sensed area covered by that Wi-Fi access point. When a mobile entity is sensed by both an RFID reader and a Wi-Fi AP, both the rules shown above enable the environment to choose the atomic location with the finest granularity. As a matter of fact, since the mobile entity is sensed by an RFID reader, the condition of the second rule is not verified. Otherwise, the condition of the first rule is verified and, thus, the environment concludes that the mobile entity is located in the atomic location associated with the sensed area covered by the RFID reader. In addition to this, it is possible to conceive other rules (not reported for the sake of brevity) that equip the environment with mechanisms to: 1. assign semantic; as an example, the temperature sensed by a temperature sensor, which is located in an atomic location, is automatically considered as the temperature of the Temperature Sensed Area associated to that atomic location; 2. predict future position of a user; from the current position the user can move only in semantic location that is contiguous to the current one; 3. avoid inconsistent location detections; if a user is located in a location “far” (by considering the physical distance and the minimum time a user needs to move) from the last one in which he has been correctly located, the location system is faulty.
A Topology Based Approach for Context-Awareness in Smart Environments
5
41
Conclusions
In this paper we have presented logic models and implementation mechanisms to achieve context-awareness in smart environments. All solutions are based on the physical topology of the environment. In addition to this, we have presented a framework of formal tools that enable designers to specify in rigorous and verifiable way several aspects of a topology, both static and dynamic. Such a choice has been motivated considering that more often smart environments and smart technologies are applied to critical scenarios, for which a good practice is to specify requirements and functionalities in an unambiguous way. Such solutions appear to be effective in providing smart environments with intelligent facilities.
References 1. Heitmeyer, C., Mandrioli, D.: Formal Methods for Real-Time Computing. John Wiley & Sons, Inc., Chichester (2004) 2. Pradhan, S.: Semantic locations. Personal Technologies 4(4), 213–216 (2000) 3. Cook, D.J., Augusto, J.C., Jakkula, V.R.: Ambient intelligence: Technologies, applications, and opportunities. Pervasive and Mobile Computing (April 15, 2008) (in press) (accepted manuscript), ISSN 1574-1192, doi:10.1016/j.pmcj.2009.04.001, http://www.sciencedirect.com/science/article/B7MF1-4W2W5PH-1/2/ 7ac602bb16ca409e2f70c13aa24ae0d5 4. Saha, G.K.: Software fault avoidance issues. ACM Ubiquity 7(46), 1–15 (2006) 5. Coronato, A., De Pietro, G., Esposito, M.: A Multimodal Semantic Location Service for Intelligent Environments: An Application for Smart Hospitals. Journal of Personal Communications (to appear) 6. Guarino, N.: Formal Ontology, Conceptual Analysis and Knowledge Representation. International Journal of Human-Computer Studies. Special Issue on Formal Ontology, Conceptual Analysis and Knowledge Representation (1995) 7. Cardelli, L., Gordon, A.D.: Ambient logic. Theoretical Computer Science. Elsevier (ed.) (June 2000) 8. Cardelli, L., Gordon, A.: Anytime, anywhere modal logics for mobile ambients. In: 27th ACM Symposium on Principles of Programming Languages (2000) 9. Milner, R.: The Pi Calculus and Its Applications (Keynote Address). In: Int. The Proc. of the 1998 Joint International Conference and Symposium on Logic Programming. MIT Press, Cambridge (1998)
Anonymous Agent Coordination in Smart Spaces: State-of-the-Art Alexander Smirnov1, Alexey Kashevnik1, Nikolay Shilov1, Ian Oliver2, Sergey Balandin2, and Sergey Boldyrev2 1 SPIIRAS, 39, 14 line, 199178, St.Petersburg, Russia {smir,alexey,nick}@iias.spb.su 2 Nokia Research, Itämerenkatu 11-13, 00180, Helsinki, Finland {ian.oliver,sergey.balandin,sergey.boldyrev}@nokia.com
Abstract. Development of new technologies brings people new possibilities such as smart spaces. Smart spaces can provide better user experience by allowing a user to connect new devices flexibly and to access all the information in the multi device system from any of the devices. The paper describes work-in-progress performed under a joint project of SPIIRAS and Nokia. The project is aimed at the analysis of existing anonymous agent coordination languages and adaptation of them to coordination of smart space devices. Keywords: smart space, coordination, anonymous agents.
1 Introduction Modern device usage is moving towards so called “smart spaces” where a number of devices can use a shared view of resources and services [1, 2]. Smart spaces can provide better user experience by allowing a user to bring in new devices flexibly and to access all the information in the multi device system from any of the devices. Examples of smart spaces can be found in [3, 4, 5]. As is assumed by such environment, one of the essential features is the information sub-system that should provide a permanent robust infrastructure to store and retrieve the information of different types from the multitude of different environment participants. Further development of Smart Spaces methodologies and techniques is a key for creating attractive use case studies and building the developer eco-system in the future. In the previous project [6] a good grounding to start implementing some of the key ideas of Smart Spaces has been prepared. In the earlier definitions of Smart Space no coordination between agents has been considered yet. However, in reality there is a need for some coordination between the agents, e.g. for resolving conflicts of simultaneous access to the shared data resource. The main focus of this paper is analyzing the state-of-the-art in the area of agent coordination languages and mechanisms of conflict resolution in at the time of agent coordination. S. Balandin et al. (Eds.): NEW2AN/ruSMART 2009, LNCS 5764, pp. 42 – 51, 2009. © Springer-Verlag Berlin Heidelberg 2009
Anonymous Agent Coordination in Smart Spaces: State-of-the-Art
43
The analysis of the state-of-the-art is performed taking into account earlier developed reference model of the smart space [6] and additional requirements such as agents anonymity and compatibility with the developed approach.
2 Reference Model of Smart Space The general reference model of the profile-based context-aware smart space is illustrated by Fig. 1. Each node (a logical unit capable to perform some actions depending on information units stored in accessible information storages) can be located at several physical devices or several nodes can be located at one physical device. A node can access a predefined limited set of information storages to acquire information units from them. The nodes exchange information units (IU) / fragments, which are stored in the distributed storage accessible by the nodes of the smart space. Domain Ontology
Configurator Node User User Service Service
User Node 1
…
User Node k
Distributed Information Unit/Fragment Storage Node 1 … Node n
User Profile
User Profile
Node Profiles Context Context History
Fig. 1. Reference model of the information recycling-enabled smart space
IU represents a logical expression: “subject”-“predicate”-“object” = [true | false]. Subject is usually an actor (human or node that performs or is supposed to perform a certain action, e.g. a multimedia centre). Predicate is an action that has to be performed (e.g., “playing music”). Object points to the target with which the action has to be performed (e.g., a song being played). Information fragment – is an IU extended by the “precondition” statement, which defines when the appropriate information unit is valid, and also might include the “post-condition” statement. The post-condition is aimed to describe what is supposed to be done when the precondition is met the information unit is hold. Each information fragment may consist of several preconditions, information units and post-conditions. The ontology defines possible capabilities of the nodes in the smart space and sets some constraints on these capabilities in the form of IUs and information fragments, which are stored in the information storages. The information storages can be physically located at one or several physical devices or several information storages can be located at one physical device. IUs can
44
A. Smirnov et al.
be transferred between information storages in order to become accessible to appropriate nodes. To estimate the efficiency of the information allocation the cost function has been introduced. The “cost” of IU transfer can be calculated as a sum of IU transmission and IU receiving costs. These costs are different for different information storages. However, cost of IU transmission from the given information storage is constant and does not depend on the destination information storage. Analogously, the cost of IU receiving for a given information storage is constant and does not depend on the transmitting information storage. To increase the efficiency of the smart space operation the personification is required. For this purpose the users and devices have to be described by a set of characteristics required for interaction with other nodes. User nodes communicate with user profiles. The user profiles are stored in information storages accessible by user nodes and contain the following information: - User identification and auxiliary information (user ID, name, personal data, contact information such as e-mail, mobile phone, etc.) - User privacy policies (e.g. calendar can be shared with certain configurator nodes, phonebook cannot be shared) - User preferences (detectable rules of the user preferred behaviour, e.g., microphone loudness, light brightness, etc.). Models and algorithms for detection of the user preferences is out of the scope of the project. Node profiles help to communicate with various nodes. The node profile is stored in an information storage accessible by the configurator node and contains the following information: - Node identification (its ID and name) - Possible statuses of the node (e.g., light is on, 30% brightness) - Node capabilities (what the node can do) - described via a domain ontology The context describes the current status of the smart space. It is represented by the set of statuses of nodes constituting the smart space. The context forms a part of the information environment where the nodes interact [6]. In the current project the information environment is a smart space. The urgency of the information contained in the context can be defined in relation to all nodes taking part in the information exchange. In course of time the context changes, thus a new context appears. The context history stores these contexts. The history of contexts enables: (i) information recycling (in what conditions an information unit/fragment was actual), (ii) finding behaviour patterns for users (user preferences), and (iii) finding behaviour patterns for nodes (anticipation of node statuses). User nodes can use additional modules (e.g., user nodes can apply to the calendar in order to get information about current user’s appointments). This interaction can be done via exchange of same information units. E.g., having access to the current smart space context the calendar can generate information unit that the user is going to have a presentation in the room, where he/she is now. One of the major principles of the smart space organisation is that the participating agents do not know about the presence of each other. There are no any direct links between them (Fig. 2). Each agent can put facts (information units) into the smart
Anonymous Agent Coordination in Smart Spaces: State-of-the-Art
45
space (shared information storage), which are accessible by other participating agents, read them and modify. But it is left unknown which agent put each particular information unit (an agent only knows which information units were put by it). In this study the agents are considered to be non-antagonistic, i.e. they do not put information with an intention to disturb/destroy the smart space. The issues of security are out of the scope of this study. The problem of simultaneous access to the shared information is similar to that in the database management systems. It is necessary to make sure that the information units stored cannot be simultaneously modified and read or modified and deleted, etc. This mechanism used in the database management systems is referred to as transactions. In smart spaces such a mechanism has to be developed and described using ontology.
Agent 1
X
Agent 2
I/F
Shared Information Storage Fig. 2. Communication of anonymous agents
As a result the requirements and associated problems have been formulated as criteria for the state of the art (Table 1). Table 1. Requirements and associated problems Problems to be solved Agent anonymity Conflict resolution Coordination Control flow Interoperability Information transfer Information consistency
Requirements Anonymity in terms of “agents’ knowing each other” Avoiding problems arising from simultaneous access (read/delete/update) to shared information Compatibility with the approach Information driven control has to be provided Interoperability has to be supported by ontologies Compatibility with information transfer via information units (RDF triples) Checking of relationships between information units
3 State-of-the-Art: Results Studied research efforts consider the agent coordination problem from different aspects. These include: agent coordination languages [7-22], coordination of entities
46
A. Smirnov et al.
[23-27], and conflict resolution at the time of agent coordination [28-30]. Such review made it possible to cover all problems set (see Introduction). Below the summary of the technologies and techniques used for problem resolution is presented. 3.1 Agent Anonymity All considered works can be split into two major groups: (i) agents communicate directly and (ii) agents communicate through a certain information space (tuple space, dataspace, shared space of information, etc.). In the latter case the direct communication between agents assumes certain knowledge of each other. The former case is very similar to the smart space philosophy where agents share information via putting it into the shared storage. The Encapsulation Coordination Model (ECM) [18] also assumes anonymous communication through specific communication means, but this approach is not compatible with the smart space approach. 3.2 Conflict Resolution In this section the conflict is considered as a problem of simultaneous access to the shared information is similar to that in the database management systems. Three major techniques for solving this problem have been considered. The database transactions guarantee that database operations are processed reliably, without data corruption. They also provide for information consistency by checking records in the related tables. The semaphores are technique preventing simultaneous access to the same piece of information by two or more entities. They are implemented via variables and have a number of disadvantages (e.g., a semaphore can be locked and not released, deadlocks may occur). The elaboration of semaphores is the technique of monitors. The monitors are usually integrated in compilers or interpreters to enable transparent locking and unlocking of jointly used resources. In the current project the technique of semaphores is planned to be used with elements of the transaction and monitor mechanisms. 3.3 Control Flow Two types of control flows are usually identified: event-driven control and information-driven control. The event-driven control assumes the following behavior “do something if a certain event occurs”. This is the most common way of creating information systems, which assumes production, detection, consumption of, and reaction to events [31]. For example, when a as soon as the temperature in the room decreases below given level the heating is switched on. Information-driven control assumes presence of intelligence. E.g., an agent may decide to “do something if certain knowledge becomes available”. To make this work, the idea of information must be treated in such a way as to give semantic properties (meaning, content) a role in the explanation of system behaviour. On this interpretation, some of the behaviour of information-driven control systems is causally explained by the statistical correlations that exist between internal states and the external conditions about which they carry information [32, 33]. Simple examples include computing systems that support instructions like: if
then
Anonymous Agent Coordination in Smart Spaces: State-of-the-Art
47
else . However, the way a condition works may change conditionally under other controls. For example, the way A influences B can depend on C [34]. The major difference is that in the information driven control a performed action is not necessarily related to the new knowledge. This new knowledge can be used by an agent to clarify some existing knowledge that, in turn, becomes the reason for the action [35]. In the smart space approach corresponds to the information-driven control due to the fact that actions are performed by the agents depending on the information available in the shared storage. 3.4 Information Transfer Information transfer formalisms in the reviewed approaches can be divided into two major groups: (i) formalisms specifically developed for particular tasks/projects, and (ii) commonly used formalisms. The first group includes: - tuple based formalism {“a string”, 15.01, 17, “another string”} [9]; - XML dataspaces [8]; 3 foo blahblah 17 - rule based predicates [21] adjacent(X,Y), location(robot, X), location(car, X) The second group includes RDF formalism, i.e. the whole document can be seen as a set of subject-predicate-object triples [27]: {:John :Loves :Mary}. Since in the smart space approach the formalisms are planned to be used by various devices the usage of specifically developed formalisms is not reasonable. Among the commonly used formalisms the rule-based predicates and tuples can be mentioned. The latter are well compatible with RDF used in smart spaces, which is basically an RDF triple. As a result the techniques used in approaches assuming tuple-based information exchange can be considered as compatible with the smart space approach. 3.5 Coordination 3 techniques of coordination are used: - Negotiation - Organizational structuring - Multiagent planning Generally speaking, all three techniques can be called negotiation. However, there is some difference. While pure negotiation assumes information exchange and following certain behavior strategies by agents, the organizational structuring techniques try to
48
A. Smirnov et al.
provide this by supplying an a priori organization by long-term relationships between agents [17], (e.g., master/slave). Multiagent planning assumes existence of a certain planning engine that produces action plans for all agents to follow taking into account their capabilities and preferences. Since the smart space approach assumes the information driven control and consists of anonymous agents the negotiation mechanisms could not be directly implemented. The reason for that is that negotiation assumes direct communication between agents, which is not possible in case of agents’ anonymity and interaction through an information space. However, certain scenarios might involve elements of other coordination techniques such as multiagent planning (e.g., in a conference room the planning node may create presentation schedules for participants). 3.6 Interoperability In most systems the communication between agents is provided by the common understanding of messages, which in turn is supported by ontologies. In other systems the interoperability is achieved via using the same limited specified language but this is not applicable to the smart space approach. In technical level interoperability can be achieved using techniques like web-services, which has a unified description language WSDL and protocol for structuring messages exchange SOAP (e.g., [36]). There are following approaches for determine the degree of interoperability between two systems. The paper [37] introduces the five Levels of Conceptual Interoperability Model: - “System Specific Data”, no interoperability between two systems. Data is used within each system in a proprietary way with no sharing. - “Documented Data”, data is documented using a common protocol, such as the Object Model Template and is accessible via interfaces. - “Aligned Static Data”, data is documented using a common reference model based on a common ontology. - “Aligned Dynamic Data”, the use of the data within the federate/ component is well defined using standard software engineering methods such as UML. - “Harmonized Data”, semantic connections between data that are not related concerning the execution code is made obvious by documenting the conceptual model underlying the component. Usage of the ontologies is perfectly compatible with the smart space approach (Aligned Static Data interoperability level). Though, in different projects different knowledge representation formalisms are used (RDF, plain XML, other formalisms) this is not a critical difference and the technologies/techniques used in such projects can be easily adapted. 3.7 Information Consistency To provide for information consistency two techniques have been found. The first one is used in database transactions. They preserve the consistency of the information in the information space (tuple space, XML dataspace, etc.), e.g., if a records is deleted, the other records depending on it will be deleted automatically. The other technique
Anonymous Agent Coordination in Smart Spaces: State-of-the-Art
49
assumes usage of reasoners compatible with the information representation formalism (RDF). This will make it possible to avoid information inconsistencies in the shared storage. Since in the presented approach the consistency of the information inside the smart space is not important to a certain degree, these problems will not be considered.
4 Conclusion The preliminary list of techniques can be formulated as follows. For resolving the problem of simultaneous access to information the techniques of semaphores and monitors will be used. The particular implementation issues are a subject of the future research. For resolution of conflicts of interests the techniques of organizational structuring and multiagent planning are to be applied. And for interoperability support the technology of ontology management together with the RDF are to be applied. The presented results are work-in-progress. The future work will include development of anonymous agent coordination/behavior strategy. For this purpose it is planned to choose the “driver” techniques and technologies defined as the basis with their further adaptation to the smart space approach and testing through a case study. Acknowledgments. The paper is due to the joint project between SPIIRAS and Nokia Research.
References 1. Oliver, I., Honkola, J.: Personal Semantic Web Through A Space Based Computing Environment. In: Middleware for Semantic Web 2008 at ICSC 2008, Santa Clara, CA, USA (2008) 2. Oliver, I., Honkola, J., Ziegler, J.: Dynamic, Localised Space Based Semantic Webs. In: WWW/Internet Conference, Freiburg, Germany (2008) 3. Oliver, I.: Design and Validation of a Distributed Computation Environment for Mobile Devices. In: European Simulation Multiconference: Modelling and Simulation 2007, Westin Dragonara Hotel, St.Julian’s, Malta, October 22-24 (2007) 4. Oliver, I., Nuutila, E., Seppo, T.: Context gathering in meetings: Business processes meet the Agents and the Semantic Web. In: The 4th International Workshop on Technologies for Context-Aware Business Process Management (2009) 5. Jantunen, J., Oliver, I., Boldyrev, S., Honkola, J.: Agent/Space-Based Computing and RF memory Tag Interaction. In: The 3rd International Workshop on RFID Technology Concepts, Applications, Challenges (2009) 6. Smirnov, A., Krizhanovsky, A., Shilov, N., Lappetelainen, A., Oliver, I., Boldyrev, S.: Efficient Distributed Information Management in Smart Spaces. In: Proceedings of the Third International Conference on Digital Information Management (ICDIM 2008), UK, London, November 13-16, pp. 483–488 (2008); [Electronic Resource], IEEE Catalog Number: CFP08DIM-CDR, ISBN: 978-1-4244-2917-2 7. 3APL An Abstract Agent Programming Language, http://www.cs.uu.nl/3apl/
50
A. Smirnov et al.
8. Cabri, G., Leonardi, L., Zambonelli, F.: XML Dataspaces for Mobile Agent Coordination. In: Proceedings of the 2000 ACM Symposium on Applied Computing, pp. 181–188 (2000) 9. Carriero, N., Gelernter, D.: Linda in context. Artificial Intelligence and Language Processing, Cohen, J. (ed.) 32(4), 444–458 (1989) 10. Castellani, S., Ciancarini, P., Rossi, D.: The ShaPE of ShaDE: a coordination system. Technical Report UBLCS, Dipartimento di Scienze dell’Informazione, Università di Bologna, Italy (1995) 11. Ciancarini, P., Knoche, A., Tolksdorf, R., Vitaly, F.: PageSpace, An Architecture to Coordinate Distributed Applications on the Web. In: Fifth International World Wide Web Conference, Paris (1996) 12. Barbuceanu, M., Fox, M.: COOL: A Language for Describing Coordination in MultiAgent Systems. In: Proceedings of the First International Conference oil Multi-Agent Systems (ICMAS 1995), pp. 17–24 (1995) 13. Dastani, M., Dignum, F., Meyer, J.: Autonomy and Agent Deliberation. In: Proceedings of The First International Workshop on Computatinal Autonomy - Potential, Risks, Solutions, Melbourne (2003) 14. Agent Communication Language, Foundation for Intelligent Physical Agents, FIPA, Ver. 1.0, Part 2 (1997) 15. Hindriks, K., de Boer, F., van der Hoek, W., Meyer, J.-J.C.: Agent programming in 3APL. Autonomous Agents and Multi-Agent Systems 2(4), 357–401 (1999) 16. Katasonov, A., Terziyan, V.: Semantic Agent Programming Language (S-APL): A Middleware Platform for the Semantic Web. In: 2nd IEEE International Conference on Semantic Computing, Santa Clara, USA, pp. 504–511 (2008) 17. Schumacher, M., Chantemargue, F., Hirsbrunner, B.: The STL++ Coordination Language: a Base for Implementing Distributed Multi-Agent Applications. In: Third International Conference, Coordination Languages and Models, Amsterdam, pp. 399–414 (1999) 18. Schumacher, M.: Objective Coordination in Multi-Agent System Engineering. LNCS (LNAI), vol. 2039. Springer, Heidelberg (2001) 19. Tolksdorf, R.: Coordination in Open Distributed Systems. PhD thesis, Technische Universität Berlin (1994) 20. Tolksdorf, R.: Laura - a service-based coordination language. Science of Computer Programming 31(2-3), 359–381 (1998) 21. Rao, A.: AgentSpeak(L): BDI agents speak out in a logical computable language. In: Perram, J., Van de Velde, W. (eds.) MAAMAW 1996. LNCS, vol. 1038, pp. 42–55. Springer, Heidelberg (1996) 22. Wells, G.: Coordination Languages: Back to the Future with Linda. In: Proceedings of the Second International Workshop on Coordination and Adaptation Techniques for Software Entities (WCAT 2005), Glasgow, Scotland, pp. 87–98 (2005) 23. Sixth Framework Programme: Context-aware business application Service Co-ordination in Mobile Computing Environments (2004-2007), http://www.ist-cascom.org 24. Herrero-Perez, D., Martinez-Barbera, H.: Decentralized Coordination of Automated Guided Vehicles. In: Padgham, Parkes, Müller, Parsons (eds.) Proc. of 7th Int. Conf. on Autonomous Agents and Multiagent Systems, Estoril, Portugal, pp. 1195–1198 (2008) 25. Simperl, E., Krummenacher, R., Nixon, L.: A Coordination Model for Triplespace Computing. In: Murphy, A.L., Vitek, J. (eds.) COORDINATION 2007. LNCS, vol. 4467, pp. 1–18. Springer, Heidelberg (2007) 26. Sixth Framework Programme: Triple Space Communication (2006-2009), http://www.tripcom.org
Anonymous Agent Coordination in Smart Spaces: State-of-the-Art
51
27. UBIWARE Platform (2007-2010), http://www.cs.jyu.fi/ai/OntoGroup/UBIWARE_details.htm 28. Wikipedia dictionary (2009), http://en.wikipedia.org/wiki/ACID 29. Dijkstra, E.: Cooperating Sequential Processes, Technical Report EWD-123, Technological University, Eindhoven, Netherlands (1965) 30. Hoare, C.: Monitors: an operating system structuring concept. Communications of the ACM 17(10), 549–557 (1974) 31. Chandy, K.: Event-Driven Applications: Costs, Benefits and Design Approaches, California Institute of Technology (2006) 32. Dretske, F.: The Explanatory Role of Information. Philosophical Transactions: Physical Sciences and Engineering 349(1689), 59–69 (1994) 33. Sloman, A.: What Sort of Control System Is Able To Have A Personality? In: Creating Personalities for Synthetic Actors: Towards Autonomous Personality Agents, pp. 166–208. Springer, Heidelberg (1995) 34. Sloman, A.: Semantics in an intelligent control system. Philosophical Transactions of the Royal Society 349(1689), 43–58 (1994) 35. Sloman, A., Prescott, A., Shadbolt, N., Steedman, M.: Semantics In An Intelligent Control System. Discussion. Philosophical Transactions-Royal Society of London. Physical Sciences and Engineering 349(1689), 43–58 (1994) 36. Watkins, E., Ardle, M., Leonard, T., Surridge, M.: Cross-middleware Interoperability in Distributed Concurrent Engineering 37. Tolk, A.: The Levels of Conceptual Interoperability Model. In: Fall Simulation Interoperability Workshop, Orlanda, Florida (2003)
On-the-Fly Situation Composition within Smart Spaces Prem Prakash Jayaraman1, Arkady Zaslavsky2, and Jerker Delsing2 1
Caulfield School of Information Technology, Monash University, 900 Dandenong Road, Melbourne, Victoria 3145 [email protected] 2 Department of Computer Science and Electrical Engineering, Lulea University of Technology, Lulea, Sweden {arkady.zaslavsky,jerker.delsing}@ltu.se
Abstract. Advances in pervasive computing systems have made smart computing spaces a reality. These smart spaces are sources of large amount of data required for context aware pervasive applications to function autonomously. In this paper we present a situation aware reasoning system that composes situations at runtime based on available information from the smart spaces. Our proposed system R-CS uses situation composition on-the-fly to compute temporal situations that best represent the real world situation (contextual information). Our proposed situation composition algorithm is dependent on underlying sensor data (hardware and software). These sensory data are prone to errors like inaccuracy, old data, data ambiguity etc. R-CS proposes algorithms that incorporate sensor data errors estimation techniques into our proposed dynamic situation composition based reasoning system. R-CS is built as an extension to Context Spaces, a fixed situation set based reasoning system. We implement R-CS dynamic situation composition algorithms over context spaces and validate our proposed R-CS model against context spaces’ fixed situation reasoning model. Keywords: Context Modelling, Smart Spaces, Reasoning Under Uncertainty.
1 Introduction Pervasive systems depend on contextual information to perform tasks in smart environments. The advances in pervasive computing research have made smart spaces a possibility. A smart space is a smart environment that has commuting embedded into it and can provide information that can be used to model the real world into the virtual computing world. We use the term smart spaces to represent such environments in the context of this paper. This contextual information originates from individual or multiple sensory sources. The sensor data by itself has no meaning but when fused with appropriate context provides a virtual view of the physical smart space. We define “Context” as “that which surrounds, and gives meaning to, something else” [1]. In smart spaces (pervasive environments), our definition of Context is realized by fusing data obtained from multiple sensors within the environment. S. Balandin et al. (Eds.): NEW2AN/ruSMART 2009, LNCS 5764, pp. 52–65, 2009. © Springer-Verlag Berlin Heidelberg 2009
On-the-Fly Situation Composition within Smart Spaces
53
In this paper we use a situation based reasoning approach. A situation as defined by [16] is “the combination of circumstances at a given moment”. We use contextual elements (attributes) to define these circumstances. For e.g. in a smart room, based on sensor inputs (contextual attributes) for light, noise and location, we can infer the occurrence of situations Meeting, Presentation. Hence a situation is a virtual world representation of a real world smart space. The situation based reasoning model Ranked-Context Spaces (R-CS) proposed in this paper evolves from Context Spaces (CS) [2, 14]. CS is a theoretical approach towards modelling context using situations. Each situation is a composition of context attributes with well defined regions. The CS model uses a fixed set definition of situations. This is acceptable in most cases but in smart spaces where new sensory inputs appear and disappear, the present CS system does not scale. We extend CS fixed situation space representation to a dynamic situation spaces composition where the situation space computed on the run is a collection of context attributes with it corresponding region that best defines the current situation within the smart space. To achieve this we use simple multi-attribute rating technique (SMART) [4] instead of the multi attribute utility theory (MAUT) [5] used by CS. The paper also proposes and implements algorithms into R-CS to incorporate sensor inaccuracies that improves the reasoning accuracy. By incorporating dynamic situation composition into CS model, our context model which is a multi dimensional virtual world representation of the physical smart space grows and shirks in with changing contextual attributes. We use the term multi-dimensional to represent multiple context attribute used to compose a situation. The key contribution of the paper includes proposing and implementing R-CS which incorporates dynamic situation composition and a novel sensor inaccuracy estimation algorithm into CS. We argue that current error estimation and fixed situation set representation of smart spaces used by CS is insufficient for changing pervasive environments. R-CS aims to develop and propose solutions to these challenges identified in CS. This papers contribution is a further work to our previous paper [15]. The rest of the paper is organized as follows. Section 2 presents related works in the area of context modelling. Section 3 provides a background into context spaces theory. Section 4 presents our proposed dynamic situation composition system R-CS, and our proposed sensor inaccuracy estimation algorithm which has been implemented in R-CS. Section 5 validates our proposed R-CS approaches by comparing the outcome of situation reasoning to CS approach. Section 6 concludes the paper.
2 Context Reasoning: An Overview Reasoning technologies generally used in context-aware pervasive system include rule based logic, Bayesian network and neural network. Pervasive environments need models that can handle reasoning under varying degrees of uncertainty. Context modelling and reasoning has been explored and presented by number of researchers. Context ToolKit [11] is a framework that presents a framework for interfacing between devices and software entities that provide contextual information. Five context abstractions namely Widgets, Interpreters, Aggregators, Services and Discoverers can be used by context-aware application developers to prototype applications. The
54
P.P. Jayaraman, A. Zaslavsky, and J. Delsing
widgets acquire context information from sensors, interpreter interprets that into high level information and aggregator collects related context information for a specific entity. Services execute behaviours and discovers maintain a list of services available to the application. Applications use discoverers to find specific components or a set of component that match certain criteria. Context models based on Logic and ontology have been presented in [12, 13]. In this paper we look at situation based reasoning approach. A situation is a higher level abstraction of the physical world which is inferred from low level context obtained from group of sensors (context attributes). A situation comprises number of context attributes, each having a well defined region within the specific situation space. The major challenge in situation reasoning is reasoning under uncertainty. One of the methods used for reasoning under uncertainty is Bayesian reasoning. In [6], a Bayesian technique for location estimation based on inaccurate sensor values is presented and [7] uses this approach for computing location of devices indoor. Applying Bayesian approach has the limitation of prior knowledge of probabilities which may not be always available. Fuzzy based systems [3] have been used in control systems that use Fuzzy Rule set to achieve specific results. Though Fuzzy systems are useful in controlling system process, they are less effective in reasoning situations [8] especially when we have situations with overlapping regions which makes reasoning under uncertainty harder. Neural network based approaches for classifying context has several shortfalls when used with quick deployable context aware applications. Neural network require an extensive training phase over sufficient number of patterns. Training can result in local minima, i.e., achieving less than optimal solutions [9]. Neural network are considered “black boxes” as it’s difficult to back track problem to the cause. Review of context modelling approaches shows that most have shortfalls which makes it difficult to provide a generic approach for context modelling under changing situations and applications domains. To this extent most of the work in the literature has not addressed the problem of dynamic smart spaces where new input data may influence the reasoning process. CS is also one such context model which provides a generic framework for context aware application to reason based on situation modelling. As mentioned in previous section, CS still does not address changing smart spaces. In the next section we present an overview of Context Spaces (CS) theory and introduce the heuristics incorporated in CS. We then identify the shortfalls of CS and propose additional heuristics that are incorporated into R-CS to aid reasoning under uncertainty in smart spaces.
3 Context Spaces The term context has a number of definitions. We provided one such definition in section 1. We elaborate the definition further re-defining context as that which refers to a particular centre of interest, providing added information about whom, where and when. We use context as a representation scheme for data that can be used to compose situations which can be reasoned upon. We use figure 1 o depict the relation between context and situations. The contextual information is dependent on
On-the-Fly Situation Composition within Smart Spaces
55
underlying sensor data (software and hardware) whose relations are computed by adding context to them at the context layer. The Situations layer is a composition of related context attributes each having a well defined region that best describes a situation. (E.g. Meeting, Presentation). CS uses the following definitions to represent the context-situation pyramid in the CS model [2, 14].
Situations
Context
Sensory Originated Data
Fig. 1. Context – Situation Pyramid
A “Context Attribute” defines data obtained from individual or group of sensor that is used in situation reasoning [2, 14]. By itself a “Context Attribute” is a single piece of data that has no meaning until we relate context to it (E.g.: Temperature vs. Temperature of Human Body). A “Situation Space” is a collection of context attributes that define a situation. Each context attribute within the situation has a well defined region. A “region” of a context attribute is a set of acceptable values that falls within the specific situation definition. CS uses a crisp representation of “region”. E.g. For a situation “Moderate Operating Environment of Machine A” context attribute temperature would be within the region “25 – 28 oC”. “Context State” represents the current value of a context attribute. CS uses multi attribute utility theorem to compute confidence in situation occurrence [2, 14]. The confidence is the measurement parameter used to determine the occurrence of a situation. To compute the confidence, CS uses two primary heuristics 1. Inaccuracies of Sensors using error probability 2. Contributing and Relevance factor of each context attribute ai in the situation space S. The heuristics identified provides sensor error probability, relevance and contribution value of the context attribute. The relevance and contribution are values given to the context attributes based on it context state (current value) in the situation space S. A contribution function computes the contribution (between 0 and 1) of context attribute ai over the region Ai defined in the situation space S. The relevance factor gives the attributes importance in the situation S. The situation space definition of CS has fixed set context attribute regions (A1, A2, An) whose importance (relevance given by a weight w) is given by
∑
n i =0
wi = 1
(1)
56
P.P. Jayaraman, A. Zaslavsky, and J. Delsing
We argue that in real world smart environments, this may not always be the case. Hence we propose the use of an extended situation space set and compute a temporal situation space on the run at every instance t, from available set of context attribute regions. The probabilistic error computation used in CS needs to have pre-determined probability but does not learn from other information like sensor data freshness, noise etc. Our proposal incorporates data freshness to estimate the sensor in-accuracy in addition to the probabilistic approach employed by CS aiming to decrease sensor inaccuracy caused due to aging data.
4 Situation Composition On-the-Fly The dynamic situation composition arises from our argument that fixed situation based reasoning does not cater to changing smart spaces where new context information is appears and disappears. Hence we need a model that can adapt to changing situations and compute situation definitions dynamically using available context attributes. Our proposal is to compute situation dynamically based on changing real world situations. We extend the situation space presented in CS from a finite set of highly relevant attribute to a universal situation space of all possible (not infinity) attributes. At any given instance t, R-CS builds a situation space on the run based on sensory input available within the smart space that has a equivalent context attribute definition in the universal situation space. Since our situation space definition is an extended set of all possible attributes, we use simple multi-attribute rating technique (SMART) [4] instead of Multi Attribute Utility Theory (MAUT) used in CS [5]. MAUT is an approach that uses a single utility value to fuse multiple attribute values. MAUT lacks ranking and attaching importance to these attributes. Hence we use SMART that used ranking approach to rank the attributes regions based on relevance and contributing factor. We also employ data quality parameters like data freshness into R-CS which is important while reasoning applications based on sensor originated data. Since our temporal situation space is a subset of the universal situation space, our context model that best represents the smart spaces grows and shrinks in size (attributes). 4.1 Situation Composition Based on Smart Space Information In [2] CS Situation space is defined as, a representation of a real world situation. Our proposal is to define a generic universal situation space as SU, a representation of situation with all possible combinations of context attributes and corresponding weights. As against the context spaces definition of weights in equation (1) we define this weight wi denoting the importance of the attribute region to the situation to be a value between a lower and a higher scale (application specific). CS [2, 14] uses MAUT approach to compute utility. We propose the use of (SMART) [4] that employs MAUT but adds ranking to MAUT. The ranking of a context region is dynamic and is dependent on both the relevance and it contribution to the situation. Based on this ranking, a temporal situation space ST is compute at runtime.
On-the-Fly Situation Composition within Smart Spaces
57
Definition1: Situation Space (SU) – A universal situation space which has all possible sets of context attributes and corresponding regions with their importance
SU = {w1 A1, w2 A2, ...wn An } where min ≤ w ≤ max
(2)
Definition2: Situation Space (ST) – A temporal situation space at time T which represents the real world situation with delta changes that has a list of context attributes and regions that best defines the situation at time T.
S T = {w1 A1 , w2 A2 ,...wm Am } where m ≤ n ,
∑
m i =0
wi = 1 and S T ⊆ SU
(3)
The temporal situation space ST is a dynamic representation of the situation at time t and changes as situation evolves. The temporal situation space is composed of the highly ranked attributes which is determined by the context attributes contribution and relevance values. By the use of a dynamic temporal situation composition, R-CS enforces dynamic ranking of attributes at runtime hence moving away from fixed situation space definitions to situation space definitions dynamically. The universal situation space at any point in time has n number of context attributes. Hence we propose a partition algorithm that partitions the situation space based on the relevance of the context attribute. This partition is application specific. The reason we chose a partitioned situation space is to reduce the complexity of iterating through all the attributes when confidence can be reached by considering the first few partitions. 4.2 Partitioning Parameter
δ
Relevance, the importance of the attribute region in the situation space SU is one of our key parameter to compute the partitioned situation space. We define a partition parameter δ called partitioner. Definition 3: Partition δ is defined as set of conditions C that satisfy a predicate P i.e. {C| P(C)}. The partition parameter partitions the universal situation space based on the predicate P. We use this partitioned space to compute the temporal situation space which takes into consideration the attributes contribution. E.g. the contribution for context attribute temperature for the situation “Machine Operating GOOD” will be 1 if temperature has value between 17 and 19 and will be 0 if it is outside the region. The temporal situation space is computed as a function of the relevance (weight) and the contribution. The algorithm iterates through every partition of the universal situation space until a confidence threshold is reached. The confidence threshold is defined as the difference in the confidence of the situations being reasoned. The higher the threshold, higher is the certainty of occurrence of one situation over the other. To reduce weight distribution while situations overlap, we propose a weight redistribution algorithm. The situation overlap occurs when non-coexisting situations attribute regions overlap with each. The
58
P.P. Jayaraman, A. Zaslavsky, and J. Delsing
weight redistribution is performed over the temporal space hence preserving the weights of context regions in the universal situation space. While reasoning two situations which are non-coexistent, the weights (importance) associated with that attribute regions is recomputed using the equation (5)
Max( wi ) − Min( wi ) Max( wi ) + Min( wi )
(4)
where wi is the set of weights assigned to attribute regions in the temporal situation space ST. Once the weight of overlapping attribute regions is computed, the entire situation space weights are normalized to compute the new weights. Figure 2 and 3 illustrates the pseudo-code to re-compute weights for overlapping regions and code to partition the Universal situation space.
Re-Initialize-Weights(TemporalSituation S, Temporal Situations S’) for each attribute-regions in S if (attribute-regions(S) = attribute-region(S’) attribute-regions(S).overlap = true end if end for for each attribute-region in S compute reduce-weight = (max(S-region-weight) min(S-region-weight)) / (max(S-region-weight) + min(S-region-weight)) if (attribute-region.overlap = true)) newweight = oldweight + reduce attribute = region.setweight (newweight) add attribute-region to returnSituationSpace end if return returnSituationSpace End
Fig. 2. Pseudo Code to recomputed overlapping attribute regions weights
Function Partition(UniversalSittuationsspace, δ) Declare PartitionedSituation for each attribute region in situationsspace for each predicate in δ if importance satisfies predicate add attribute region to partitionedSituation add partition number to attribute region end if end for return PartitionedSituation end procedure
Fig. 3. Pseudo Code to partition universal situation space SU
On-the-Fly Situation Composition within Smart Spaces
59
4.3 Incorporating Sensor Inaccuracy into Dynamic Situations The sensor inaccuracy used by CS is both primitive and has crisp boundary. The contribution using the error function is computed using the formula
ci = Perror ∗ RgionContribution
(5)
The RgionContribution is computed using a crisp boundary and is determined by checking if the sensor value falls within the attribute region. If it does not a value 0is returned and if it is a value 1 is returned. The Perror provides the probability by which the sensor reading is accurate. The crisp region does not suit all real world cases. Also CS does not change the sensor value based on the error probability. E.g. if there is 5% error, then the value can have a ± 0.05 variation from the outer boundaries. This is not considered in CS while computing the contribution. Hence we propose the use of non-crisp boundaries as illustrated in figure 4.
Fig. 4. Comparison between CS Crisp boundary and R-CS membership
To convert the crisp region into a value based on bordering regions, we employ the principle used by Fuzzy membership functions [3]. E.g. if we define a situation HOT whose temperature can be ranging from 34 to 40, a real value (context state) of 40.05 can still fall under the condition HOT if we have some tolerance in the outer regions of the context attribute. Our technique focuses on values that are inside and at the border of the defined region. Hence we re-define the context attribute’s region with two values namely outer range and inner range. The inner range represents values which returns contribution factor of 1 while the outrange is a ± value to the inner range values that returns a contribution based on where the sensor value falls in the outer range. Our approach tests the sensor value against the inner and outer region after applying the error probability over the sensed value. To compute sensor inaccuracy, we also introduce data freshness. By data freshness, we mean how recent the data is? We do not get into quality of context [10] as that is a topic of research by itself. Incorporating probabilistic sensed value and sensor data freshness, our new contribution function is given in (6). We implement these inaccuracy estimation algorithms into our proposed system. The freshness parameter is application specific.
ci = RgionContributioni ∗ σ i whereσ i = Perror ∗ Freshness
(6)
60
P.P. Jayaraman, A. Zaslavsky, and J. Delsing
is the composite error computed from probability of error and freshness. To improve sensor accuracy, more data quality related parameters can be added to σ.
5 Simulation and Evaluations R-CS has been implemented over CS model for sensor fusion and reasoning by incorporating the proposed algorithms for situation partitioning, re-computation of weights and improved error estimation. We present our evaluation by validating the proposed approaches against CS approach. 5.1 Sensor Inaccuracy We compare CS based confidence generation and R-CS based confidence generation using our proposed probabilistic incorporated, freshness based error computation algorithm. To test the effect of the improved error estimation algorithm, we ran simulations to reason a Presentation situation. The situation definition is shown in Table 1. Table 2, provides the output of using probabilistic error correction and data freshness technique. As it shows, the R-CS system computes a confidence of 1 for attribute noise taking into consideration its error rate while returns 0 to people attribute taking into consideration the freshness threshold is greater than 3. The result of our simulation run is shown in figure 5. The X-axis is the number of simulation run and the Y-axis is the confidence computed using CS and R-CS. The Table 1. Presentation Situation Definition Situation:
Presentation
Attributes
Regions
LIGHT NOISE PROJECTOR PEOPLE
“>-=10" " > 20 " "= ON" "> 5 "
"<=20" " < 30" "=OFF" "< 10 "
Table 2. Comparing CS and R-CS error estimation approach
Regions Threshold >2
Actual Sensor Values
Error
CS Freshness
R-CS
Confidence
Light
10 to 20
12
90%
1
1
1
Noise
20 to 30
19.95
90%
2
0
1
People
5 to 10
10
100%
3
1
0
On-the-Fly Situation Composition within Smart Spaces
61
R-CS approach takes into consideration both sensor freshness and sensor error probability. The Y value gives an indication on the how accurate the value read from the sensor is. The higher the accuracy the higher the confidence is and vice-versa. From the graph, we can see that variation in freshness and error probability reflects in the reasoning process shown by the change in confidence computed using CS and R-CS approach. The use of error estimation using data freshness and error rate incorporated in R-CS produces better results while reasoning compared to CS approach. For the simulation, we used a freshness threshold of 2. At simulation run 3, the freshness of all attributes was kept within 2 and the actual sensor values were within the attribute ranges and hence the confidence computed using CS and R-CS coincides. The results show how the effect of data quality of sensors influences the error rate hence the reasoning process. During the simulation, we changed the sensor values from inner to outer regions. We compute the error rate accuracy by changing sensor freshness and error probability values and comparing actual situation occurrence to the one suggested by R-CS and CS. The results show that incorporating data freshness heuristics greats improves the reasoning accuracy.
Confidence Simulation Run
Fig. 5. Varying Freshness Threshold to Compute Error Rate
5.2 Situation Partitioning and Re-computing Weights We computed the confidence using R-CS algorithms presented in figure 2 and 3 and compare it with CS approach in this section. We simulated the system based on different sets of attributes values and partitioners. Table 3 gives the universal situation definitions with the partitioning parameter. The temporal situation space definition computed during simulation considers the first partition before expanding it search into other partitions.
62
P.P. Jayaraman, A. Zaslavsky, and J. Delsing Table 3. Universal Situation Space Definition
Partitioner
0 < δ <= 2
Situation: Attributes A1 A2 A3
Presentation Regions "= DIM" "! ON" ">2" " <= 4" "= ON" "! OFF"
A4 A5
Situation:
">= 2 " "= Presentation Mode" "= Inside" Meeting
Attributes A1
Regions "! DIM"
A2 A3 A4 A5
"> 4 " "!ON" ">1" "! Presentation Mode" "= Inside"
A6
A6
2 < δ <= 5
"< 8 " "= Standby "
"! OFF"
"! OFF"
Relevance (0 to 5) 3 3 2 2 0.8
0.9 Relevance (0 to 5)
"< 10" "= OFF" " <= 3" "= Standby"
P1 P1 P2 P2 P2
"! Outside"
"= ON"
Partition
"= OFF"
"= OFF"
P2 Partition
3
P1
3 2 2 0.8
P1 P2 P2 P2
"! Outside"
0.9
P2
We ran simulations to compute the confidence using R-CS and CS approach using the above situation definitions by using a stream of simulated sensor data. Figure 6a and 6b are screen shots of the R-CS reasoning engine interface implemented in Java and figure 7 presents the results of the reasoning simulations. The simulation screen shot has definition of the situation with range of values and the actual sensor value (which is a simulated sensor stream). Based on the avaialble attributes, the situations Presentation and Meeting are reasoned. The reasoning interface is used to generate a stream of values that correspond to a specific situation which is used to validate the accuracy of our reasoning algorithms. We can see from the screen dumps that some of the attribute regions used to define Presentation and Meeting situations have overlapping real time values (sensor values). Such overlapping situations are not handled by context spaces which are overcome by the proposed R-CS approach that incorporates the weight re-distribution approach to reduce uncertainty due to overlapping attribute region values.
On-the-Fly Situation Composition within Smart Spaces
63
Fig. 6a. Simulation of Situation Presentation in R-CS
Fig. 6b. Simulation of Situation Meeting in R-CS
The result presented in the graph compares the confidence computed using CS and R-CS approach. The X-axis is the number of simulations and the Y-axis is the confidence obtained by the reasoning process. As illustrated in the graph, R-CS approach results in higher confidence using the re-weighted algorithm compared to CS.
64
P.P. Jayaraman, A. Zaslavsky, and J. Delsing
Another key point to note is the difference in the confidence values between Meeting and Presentation situation in R-CS and CS. The difference in confidence computed for situations is greater that CS using the proposed R-CS approach which aids in increasing the confidence of the reasoning process.
Fig. 7. Confidence computed using R-CS and CS approaches
6 Conclusion The paper has proposed, implemented and validated R-CS, ranked dynamic situation composition based context reasoning system. We have presented the heuristics used in context spaces identifying the shortfall in them. We have proposed R-CS as a solution to bridge CS shortfalls by proposing addition heuristics including partitioned universal and temporal situation spaces, sensor data quality computation and improved error estimation algorithm. The error estimation algorithm incorporated into R-CS takes error probability and data freshness to re-compute actual sensor (attribute) value and hence the attribute’s accurate contribution in the situation. This error estimation approach has proved to be more accurate than CS approach. The heuristics proposed aids the reasoning ability of the system. R-CS has been evaluated by implementing the proposed algorithms and comparing them with the CS approach. The results greats improve reasoning with the help of the proposed error estimation and partitioned situation spaces approaches. The results produce better reasoning outcomes when compared to static situation based CS approach. The results have proved our hypothesis and reinforce our proposal of using situation composition on the run and incorporation of heuristics like data freshness into the error estimation algorithm to improve the reasoning ability of the context aware system. We now look at further investigating approaches to identify relationships of newly discovered context attributes to existing situation spaces.
On-the-Fly Situation Composition within Smart Spaces
65
References [1] Context. (n.d.). The Free On-line Dictionary of Computing, from Dictionary.com, http://dictionary.reference.com/browse/context (retrieved June 4, 2008) [2] Padovitz, A., Loke, S.W., Zaslavsky, A.: Towards a Theory of Context Spaces. In: Das, C., Kumar, M. (eds.) Proceedings of the Second IEEE Annual Conference on Pervasive Computing and Communications Workshop, Orlando, Florida, March 14-17, pp. 38–42. IEEE Computer Society, USA (2004) [3] Mendel, J.M.: Fuzzy Logic Systems for Engineering: A Tutorial. Proceedings of the IEEE 83(3), 345–377 (1995) [4] Barron, F.H., Barrett, B.E.: The efficacy of SMARTER–simple multi-attribute rating technique extended to ranking. Acta Psychologica 93, 23–36 (1996) [5] Butler, J., Morrice, D.J., Mullarkey, P.W.: A Multiple Attribute Utility Theory Approach to Ranking and Selection. Manage. Sci. 47(6), 800–816 (2001) [6] Fox, D., Hightower, J., Liao, L., Schulz, D., Borriello, G.: Bayesian filtering for location estimation. IEEE Pervasive Computing (2003) [7] Castro, P., Munz, R.: Managing context data for smart spaces. IEEE Personal Communications 7(5), 4–46 (2000) [8] Zadeh, L.: Outline of a New Approach to the Analysis of Complex Systems. IEEE Transactions on System, Man and Cybernetics 3 (1973) [9] Shulin, Y., Chang, K.C.: Modular Neural Net Architecture for Automatic Target Recognition. In: Proc. of Signal Processing, Sensor Fusion, and Target Recognition, Orlando, Florida, April 1996, vol. 8-10, pp. 166–177 (1996) [10] Buchholz, T., Kpper, A., Schiffers, M.: Quality of context: What it is and why we need it. In: Proceedings of the Workshop of the HP OpenView University Association, Geneva, Switzerland (2003) [11] Salber, D., Dey, A.K., Abowd, G.D.: A conceptual framework and a tool-kit for supporting the rapid prototyping of context-aware applications. Anchor article of a special issue on context-aware computing in the Human-Computer Interaction (HCI) Journal 16(2-4) (2001) [12] Ranganathan, A., Campbell, R.H.: A Middleware for Context-Aware Agents in Ubiquitous Computing Environments. In: Endler, M., Schmidt, D.C. (eds.) Middleware 2003. LNCS, vol. 2672, pp. 16–20. Springer, Heidelberg (2003) [13] Chen, H., Finin, T., Anupam, J.: An Intelligent Broker for Context-Aware Systems. In: Proc. of Fifth International Conference on Ubiquitous Computing (Ubicomp 2003) (October 2003) [14] Padovitz, A., Loke, S.W., Zaslavsky, A., Burg, B., Bartolini, C.: An Approach to Data fusion for Context-Awareness. In: Dey, A.K., Kokinov, B., Leake, D.B., Turner, R. (eds.) CONTEXT 2005. LNCS (LNAI), vol. 3554, pp. 353–367. Springer, Heidelberg (2005) [15] Jayaraman, P.P., Zaslavsky, A., Delsing, J.: Smart Sensing and Sensor Data Collection on the move for Modelling Intelligent Environments. In: 1st Conference on Smart Spaces (ruSMART 2008), St.Petersburg, Russia (2008) [16] Situation, Wikitionary, http://en.wiktionary.org/wiki/situation (retrieved June 4, 2009)
Empower Mobile Workspaces by Wireless Networks and Wearable Computing Michael Lawo1, Otthein Herzog1, Michael Boronowski1, and Peter Knackfuß2 2
1 TZI - Universität Bremen, Am Fallturm 1, D28359, Bremen, Germany InfoConsult Gesellschaft für Informationstechnik mbH, Anne- Conway- Straße 4; D-28359 Bremen, Germany {mlawo,herzog,mb}@tzi.de, peter.knackfuß@infoconsult.net
Abstract. The concept of Wearable Computing can be used to support work processes of mobile workers e.g. in a factory environment or outdoors. The basis is a wireless network for an ubiquituous remote access to information. In the paper we explain the potential of Wearable Computing and focus on the technological challenges. During the last five years (2004 till 2009) the EC funded the worldwide largest project on Wearable Computing: wearIT@work (www.wearitatwork.com)1. The project did not focus on new technologies. However, with existing technologies new applications were developed. For new user groups of mobile workers a new quality of support had to be achieved. The paper reports at the end of the project on the main findings and lessons learned: An open hardware platform, an open software framework and for seven different application domains solutions were developed. Keywords: Wearable Computing; Mobile Computing; Context Detection; User Interface Design.
1 Introduction Wearable Computing is designed to abolish the distinction between information acquisition to support a task (e.g. repair an engine) and the work itself. This requires new technologies. The information presentation has to be redesigned e.g. using headmounted or arm-mounted displays. Furthermore the system input has to be redesigned e.g. by using speech, a gesture wristband or glove as well as sensors detecting the user’s context automatically (as using GPS signals in a route guidance system in a car). 1
wearIT@work is an Integrated Project to investigate “Wearable Computing” as a technology dealing with computer systems worn as unobtrusively as clothing. From 16 countries the project has 42 partners, among them EADS, HP, Microsoft, SAP, Siemens, Škoda, Thales and Zeiss. The consortium consists of end-user organizations with strong impact on the respective market like automotive and aeronautics. Furthermore strong partners are in the consortium to ensure that solutions found will benefit as far as standardization is addressed. SMEs are in the consortium as consultants and application developers and system integrators as it is expected that based on solutions found new business for this kind of companies is created. The project volume isf 23.7 million €€ and the EC funds it by 14.6 million €€under contract no. 004216.
S. Balandin et al. (Eds.): NEW2AN/ruSMART 2009, LNCS 5764, pp. 66–76, 2009. © Springer-Verlag Berlin Heidelberg 2009
Empower Mobile Workspaces by Wireless Networks and Wearable Computing
67
Industrial enterprises can improve their competitive position by Wearable Computing solutions that support or even change existing value-added processes. A radical change of value-added processes offers potentially a higher benefit but also higher risks under technological as organisational perspectives. The following four scenarios in general provide an idea about the different challenges and opportunities: •
•
•
•
We can use Wearable Computing solutions to simplify and accelerate existing processes without changing them. - An example is to take notes on performed tasks directly in a mobile system instead of taking notes first and later input them into a back-office system. – High user acceptance can be expected due to saved time and increased comfort. However only few such scenarios exist. We can change existing processes by eliminating steps or changing the involved actors. The expected gains could be e.g. acceleration, improved quality, cost reduction and/or increased turn-over. - An example is to monitor performed tasks directly in a Wearable Computing system without any preand post- process back-office action. – Here technological and organisational challenges and risks are higher then in the previous scenario. However many such scenarios exist.. We can modify the architecture of the value creation chain notably by the introduction of Wearable Computing systems. – An example is to let mobility become the central element of the accomplishment of an enterprise as UPS did for the delivery of parcels. - It is obvious that the user acceptance is a key and high effort is required to achieve this. We can radically replace the whole value creation system based on the idea of mobility. Then a new business with necessary new core competencies evolves. - An example of another domain is the eCommerce concept of Amazon compared to existing book stores. – This usually requires the establishment of a new enterprise.
Wearable computing systems e.g. in car production or aircraft maintenance offer the mobile worker primarily an ambient support of conversant processes. However, these systems extend also processes by checking e.g. correctness and completeness and collecting data to improve processes using quality assurance systems based e.g. on data mining. Wearable Computing assures especially that processes become more straightforward, the quality increases and idle time decreases. The main technological challenges in any Wearable Computing System exist in the design of user friendly multimodal interfaces supporting primary tasks and robust context detection systems.
2 Wearable versus Mobile Computing Beside Wearable Computing solutions also mobile computing solutions can improve value creation processes. The main difference between wearable and mobile computing (using a laptop, PDA, smart phone) is that mobile computing does not allow the ambient interaction (see fig. 1 left); the user interacts with the real world or the system but cannot do both at a time. Typing-while-driving-a-car is a nice illustrative example.
68
M. Lawo et al.
To ensure an ambient interaction Wearable Computing requires context detection: The system has to be always on and understand the user’s action purpose and environment. Relevant information must be presented in an ambient way (see fig. 1 right). - The integration of sensory devices worn by the user as well as integrated into the user’s environment creates a powerful Wearable Computing system. The “always on” feature requires an efficient energy management essential for any Wearable Computing system.
Fig. 1. Interaction modes of mobile and Wearable Computing systems
Wearable Computing means for the user simultaneous actions: the primary task (e.g. maintaining a machine) happens in the real world, is mobile and basically manual. The secondary task (observing the context dependent information concerning the ongoing action) consists in the observation of a wearable output and eventually the interaction with the wearable application. Here immediately the question arises: How to design this specific task relevant interface? Due to the system restrictions in size, weight, and energy supply in addition to the implicit input (context detection), the developer as well as the user are more challenged as if designing a mobile computing solution. Users and developers have to face the dual task challenge. And that this is not a simple task we know from designing a cockpit or a control station. We have to acknowledge that the cognitive load a user can manage is limited and its extension requires training, sometimes a lot of training. However, when implementing Wearable Computing solutions complex tasks and work processes can be performed in less time and with less effort: the productivity can be increased2 creating a competitive advantage. The approach consists in a 2
E.g. 20 to 50% of an aircraft maintenance task is lost in searching for information. We could improve productivity in our healthcare ward round scenario from 20 minutes at the bedside plus 90 minutes in the back office to 40 minutes at the bedside and no back office work at all.
Empower Mobile Workspaces by Wireless Networks and Wearable Computing
69
complete integration of all components into the clothing reflecting the fact that the primary task is not to work with the computer but to be supported by it in performing primary tasks in the real world. In this case the real advantage arises from wireless networks by the provision of originally stationary IT services independent of time and space by creating smart spaces with the user in the centre taking the full benefit of the infrastructure by the Wearable Computing solution. The benefit increases by decreasing active user interaction with the system. Here we get an optimization problem where the cost for the automatic detection of the context and the user’s intention is the objective and the user acceptance creates the restrictions of the problem and is the key for success. 2.1 Wearable User Interface (WUI) Toolkit The interface between the Wearable Computing system and the user is the key element when designing a system from the user perspective. There are not yet any standards for this technology as we know them from the desktop applications. Similar problems have to be solved as we know them from early designs of graphical user interfaces (GUI). Today a lot of tools are available that facilitate the developer when creating “traditional” applications. In the context of the wearIT@work project a tool for the design of multimodal wearable computing interfaces was developed [Witt08]. This tool allows the developer with little experience in designing Wearable Computing solutions a specification independent of specific input and output devices. This tool supports a multimodal interaction using context sensitive interfaces with an automatic adaptation (fig. 2).
Fig. 2. Wearable User Interface Toolkit [Witt08]
The development is based on reusable components and became a part of the Open Wearable Computing Framework of the wearIT@work project. Further information concerning the Framework and the Toolkit are found in the Technology Repository on the project homepage. Using such tools the software development process is accelerated, technical risks are reduced and the user acceptance can be increased.
70
M. Lawo et al.
2.2 Context Detection To achieve to a large extend an automatic context detection to reduce the user input is a further Wearable Computing specific issue. The context can be detected by sensors in the environment or those integrated into the clothing or attached to the body of the user, where the first has by far higher user acceptance due to our findings. Wearable Computing systems are designed to model and detect user activities [ADOB98], to proactively provide the user with relevant information. Here robust technical solutions are required that reduce the complexity of the user interface. Context information can basically be deducted by sensor information fused with application knowledge. There are different kinds of sensors available fort he context detection like accelerometers, yaw sensors, light sensors as well as microphones, gyroscopes and RFID reader. We differentiate between discrete (events) sensors, e.g. reading a value or identification of a state (RFID-tag), and continuous or steady sensors based on the observation of a signal applying filter techniques; in the simplest case the reach of a threshold is required to detect a specific context. Here machine learning methods are applied [Bish06]. The sensors are applied on different parts of the body or clothing respectively. Thus sensors for indoor localisation are applied in a shoe or a helmet. To detect gestures sensors are integrated into a glove or a wrist band. In general the positioning of the different sensors depends on the context to be detected. For simplicity very often sensors are placed in the user environment instead on the body. However, a sensor cannot be handled separately; the sensor is part of an overall system and requires beside an energy supply usually also an A/D converter, a signal processing unit and a signal transmission unit. We will come later to this point in our findings. 2.3 Energy Consumption and Energy Management Energy efficiency is a critical success factor of Wearable Computing. The user requirements are usually eight hour availability without changing the batteries and hot swappable battery changes (without switching off the device). Especially the wireless data communication has to be energy efficient (e.g. ZIGBEE instead of Bluetooth). Data compression techniques are applied for finding an optimal balance between bandwidth and energy consumption. Cyclic retrievals, reducing the energy requirement for a single operation, the choice of procedures with low energy consumption and the implementation of the respective algorithms on the used processors help to reduce the energy consumption of the usually rechargeable batteries. As weight is a further key issue in Wearable Computing usually Li Polymer batteries are used which provide a high energy density with low weight. Up to date there exists no general approach for an optimal battery. However specific empirical methods and solutions exist e.g. for context detection systems [SLT07]; there a procedure is presented for finding a compromise of energy consumption and detection rate instead of maximising the detection rate while neglecting the required energy.
Empower Mobile Workspaces by Wireless Networks and Wearable Computing
71
The development of efficient batteries during the last couple of years was basically driven by the mass market due to the requirements of cell phones, digital cameras and MP3 players. Although this had already a quite positive effect also for Wearable Computing solutions further efforts are necessary as especially the requirements of multi sensor systems for the context detection are quite different from those of these mobile consumer products.
Fig. 3. Wearable Computing architecture
3 Wearable Computing System Architecture The design space of any Wearable Computing system architecture is determined by the four conflicting criteria of functionality, energy consumption, application domain and configuration. Our research found that there is no universal for all applications suitable system. For the desktop paradigm beside the standard PC we have variations like notebooks, netbooks, PDAs, and smart phones. In Wearable Computing we also have different system designs depending on the criteria above. The architecture (fig. 3) is not a single device but consists beside a main on-body unit with the communication system (taking advantage of the available wireless networks) of a input and output devices (IO devices) as mentioned above, sensors worn by the user and those in the users environment and reach as well as external servers providing application dependent services. Only if these five components are distributed in an appropriate way an optimal solution can be found in the specific design space. To package the hardware of IO devices, main unit and sensors worn by the user specific textile solutions are required to avoid the feeling of a user to be “wired”. A typical solution is the in fig. 4 shown LinkVest that was designed in the wearIT@work project together with the final users for aircraft maintenance but was also evaluated in car production.
72
M. Lawo et al.
Fig. 4. LinkVest
4 Hardware Platform Wearable Computing systems are designed to flexibly support the user under usually changing environmental conditions. Wearable Computing systems therefore require redundancies as even if components fail or features can under the specific circumstances not be used to let the user control the work processes under any condition. To allow a flexible configuration of such systems an open hardware platform is supportive [HLBK06]. This architecture was developed in the context of the wearIT@work project an evaluated in the domains of healthcare, car production, aircraft maintenance and emergency response (see [LTLH07] and [AL09]). Sensors for context detection and localisation are an essential element of the architecture. The open system architecture allows integrating different sensors (see section 2.2) on different levels like functional textiles, embedded microsystems, attachable peripherals and carry on, miniaturized appliances. However, as we wanted to base our applications mainly on COTS3 components two problems became obvious: •
Sensors are in general not energy self-sufficient but require an energy supply; exceptions are passive RFID tags that are inductively provided by the reader. • Sensors usually require different voltage; in case of a sensor cluster that means individual voltage feed with many batteries or an elaborately designed energy management system. At the end of the project we used in different settings 38 hardware components and subsystems that were partially developed during the project. In order to give an objective evaluation of the maturity level of the technologies involved we adopted the Technology Readiness Levels (TRL) from the US Department of Defense [DOD03] where the maturity level is expressed by a numeric value between 1 “basic principles observed and reported” and 9 “actual system proven through successful mission operations”. The above mentioned 38 hardware components and subsystems are found with their initial (at the beginning of the project) and actual TRL in [W@W09]. 3
Commercial-off-the-shelf.
Empower Mobile Workspaces by Wireless Networks and Wearable Computing
73
Fig. 5. Open Wearable Computing Software Framework
5 Software Framework The application development requires not only an openly supporting hardware but also software. Here Wearable Computing Systems require an extension of the middleware (fig. 5). A comprehensive framework has to provide the following services: • • • • • •
Tool kit for the user interface design including speech recognition Interfacing context localisation detection sub systems Supporting mobile audio and video collaboration Allowing seamless communication when switching between the different wireless networks and network cells (WLAN, UMTS, GPRS) Access to complex knowledge as a kind of mobile eLearning (content management) Adaptive workflows reflecting the user needs and desires.
In the wearIT@work project such an open software framework was developed for further use [W@W09]. With this framework applications are configured using the appropriate components. An example of configuring an assembly task application in car production from wearIT@work is shown in fig. 6: a complex assembly task is supported by a wearable system. Beside the direct system support the user can achieve an external expert support. In this example beside the online quantitative and qualitative context detection also the localisation are applied. For the flexible support of the work processes beside the Wearable User Interface Toolkit and speech recognition also the workflow component are utilized. The collaboration module is used to give the user support by a remote expert if necessary.
74
M. Lawo et al.
Fig. 6. Software architecture of an application based on the OWCF
The framework was nearly completely developed from scratch within the project. The TRL definition is used again to evaluate the achieved maturity. At the end the average OWCF maturity level is 6.5 what qualifies the OWCF in between a “system demonstrated in a relevant environment” (maturity level 6), and a “system demonstrated in an operational environment” (maturity level 7). Excluding the Core Framework, half of the components have a maturity level above the average. Four components have a maturity level of 8, indicating that they are completed and qualified through tests and demonstrations. Three out of the five components having a maturity level below average, are ranked 6, indicating that they have been at least demonstrated in a relevant environment. Two components (communication and localization) are ranked 4 indicating that they are still in a somewhat early stage, and that have been validated mostly in a laboratory environment. Excluding the Core Framework, all components went through a significant evolution during the project: eight components have been created within the project itself, and two existed before the project with a low or very low maturity level.
6 Market Situation and Introducing the Technology The Wearable Computing market is already for years an emerging one. The continuous market research [VDC07] shows in the domains of consignment sail and after sales service annual growth rates of around 10%. The development during the last ten years shows especially in the athletics and wellness increasing a trend towards mass customisation. However in our domain of blue collar workers nothing comparable exists beside some remarkable success with pick-by-voice solutions. The hardware became smaller and more powerful with decreasing cost; the hardware cost today
Empower Mobile Workspaces by Wireless Networks and Wearable Computing
75
only one third compared to five years ago and during the next five years we expect a further decrease of 30 to 90% ending with approximately 500 €€. The software cost are however independent of this. The licensing policy and the business models are here more important than the tools and dictate the price. With a project like wearIT@work where large portions of the middleware that creates the competitive advantages are provided as open source or by the mayor software vendors as part of their standard products a great step forward towards standardisation could be done. With the more than fifty systems and components of the wearIT@work Technology Repository indicating the standardised technology readiness level [DoD03]. However, Wearable Computing solutions are not self explaining but require extensive implementation efforts as twenty-five years ago when rolling out CAD solutions revolutionising the design office work or fifteen years ago with the roll out of the PC in the office. Wearable Computing solutions address completely new user groups. Training efforts are necessary to achieve the necessary user acceptance and user efficiency.
7 Conclusion Wearable computing can provide solutions for new user groups e.g. in logistics, maintenance, assembly and production. Wearable Computing systems require the user’s ability to perform two tasks at a time; they support as a dual task primary tasks performed in the real world in an ambient way. They can also serve as intelligent displays and/or speakers. They use personal and environmental sensors providing mobile assistance. The context detection is thus beside the user interface design the second challenge to be mastered. For users simulation systems with scoring were developed to train the dual-task ability. This seems to be a further step towards a practical introduction. With a high level presentation of the project results [PPL09] decision makers get an appropriate publication on the issue into their hands. However, the following lessons were learned within the wearIT@work project when trying to empower mobile workspaces by wireless networks and Wearable Computing [W@W09]: •
• • • •
The simple configuration of COTS sensors to a sensor platform is today (still) not possible; only in exceptional cases (XSens MT) sufficient powerful sensors are at hand. However the sensor cost is often too high avoiding a broader implementation. The different voltage of the COTS sensors requires a specific energy management system. The online activity recognition requires still further research before a successful application in the industrial environment. By context detection the productivity can be increased compared to paper based solutions. The use of RFID tags and readers is actually the state-of-the-art for reliable context detection.
76
M. Lawo et al.
To assure a sustainable exploitation of the project results also after the end of the project in May 2009 the Open Wearable Computing Group (www.owcg.com) was established. The wearIT@work partners herewith pave the way for all those from research, software and hardware, consultants, system integrators and application developers outside the aboriginal consortium.
Acknowledgement The authors acknowledge the European Commission for their support and give gratitude and appreciation to all wearIT@work project partners for their fruitful work and contribution during the development of various ideas and concepts presented in this paper.
References [ADOB98] Abowd, G., Dey, A., Orr, R., Brotherton, J.: Context-awareness in wearable and ubiquitous computing. Virtual Reality 3, 200–211 (1998) [AL09] Amft, O., Lukowicz, P.: From Backpacks to Smartphones: Past, Present and Future of Wearable Computers. IEEE Pervasive Computing (to be published, 2009) [Bish06] Bishop, C.M.: Pattern recognition and Machine Learning. Springer, Heidelberg (2006) [DoD03] Technology Readiness Assessment (TRA) Deskbook (2003), http://www.darpa.mil/STO/solicitations/ sn07-44/pdf/TRA_Desktop.pdf (accessed April 16, 2009) [HLBK08] Herzog, O., Lawo, M., Boronowsky, M., Kenn, H., Nicolai, T., Witt, H., Glotzbach, U.: Mobile City Bremen - Von der Forschung zu innovativen mobilen Lösungen. In: Síeck, J., Herzog, M.A. (Hrsg.) Wireless Communication and Information - New Technologies and Applications, pp. 9–22. Verlag Werner Hülsbusch, Boizenburg (2008) [LTLH07] Lukowicz, P., Timm-Giel, A., Lawo, M., Herzog, O.: WearIT@work: Towards Real-World Industrial Wearable Computing. IEEE Pervasive Computing 6(4), 8–13 (2007) [PPL09] Pezzolo, R., Pasher, E., Lawo, M.: Intelligent Clothing – Empowering the Mobile Worker by Wearable Computing. AKA and IOS Press (2009) ISBN 978-3-89838-612-2 [SLT07] Stäger, M., Lukowicz, P., Tröster, G.: Power and Accuracy Trade-offs in SoundBased Context Recognition Systems. Pervasive and Mobile Computing 3(3), 300– 327 (2007) [VDC07] VDC. Wearable Electronic Systems, 3rd edn., October 2007, vol. 3 (2007), http://www.vdcresearch.com/maw/wearables.asp [W@W09] Technology Repository wearIT@work., http://www.wearitatwork.com/ Technology-Repository.136.0.html (accessed January 7, 2009) [Witt08] Witt, H.: Human-Computer Interfaces for Wearable Computers: A Systematic Approach to Development and Evaluation. Teubner-Vieweg (2008)
Multimodal Interaction with Intelligent Meeting Room Facilities from Inside and Outside A.L. Ronzhin and V.Yu. Budkov St. Petersburg Institute for Informatics and Automation, 39, 14th line, St. Petersburg, Russia {ronzhin,budkov}@iias.spb.su
Abstract. Multimodal user interface capable to perceive speech, movements, poses and gestures of meeting participants in order to determinate their needs provides the natural and intuitively understandable way of interaction with the developed intelligent meeting room. Awareness of the room about spatial position of the participants, their current activities, and roles in a current event, their preferences helps to predict more accurately the intentions and needs of participants. The accomplished integration of Nokia mobile phones with the sensor network and smart services allowed users to control effectors, audio/video and other facilities from outside. Some scenarios of multimodal interaction with the room as well as issues of adaptation of user interface to limitations of mobile phone browser are discussed. Keywords: ambient intelligence, smart space, multimodal interfaces, intelligent meeting room, mobile communication.
1 Introduction An idea of recognition of a current situation and behavior of a user, as well as an unobtrusive satisfaction of his needs underlies the “ambient intelligence” (AmI). These tasks deal with three directions of science and technique: ubiquitous computing, ubiquitous communication, and multimodal interfaces [1]. Integration of diverse computation, information and communication resources into a united framework is one of the important issues at design of ambient intelligence and it identifies the modern tendency to transition from smart devices to an ambient intelligent space. Multimodal interfaces provide natural and intuitively comprehensible interaction between a user and intellectual devices, which are embedded into the environment. All the means should be hidden, thus the user can see only the results of intellectual devices activities and concentrate attention on her/his work. The so-called “smart home” (SH) is the most profoundly studied area from the domain of the ambient intelligence. A smart home can be defined as a dwelling house equipped with computational and informational technologies guessing and reacting upon needs of the inhabitants, working on comfort maintenance, security and good entertainment by means of control of domestic equipment and communications [2]. In spite of the fact that SH idea was well established in the late 1990s, SH technologies still have no popularity among potential users. It is Gann’s opinion [3] that the S. Balandin et al. (Eds.): NEW2AN/ruSMART 2009, LNCS 5764, pp. 77–88, 2009. © Springer-Verlag Berlin Heidelberg 2009
78
A.L. Ronzhin and V.Yu. Budkov
principle barriers to uptake are the following: high cost, difficulties of integration with contemporary domestic appliances and lack of usability. Taking into account functionality available to a user it is necessary to distinguish homes with smart devices and homes providing interactive computing both inside and beyond the home. Five classes of SH were proposed in [3]: (1) home, which contains stand-alone intelligent objects, actions of which are mutually independent; (2) home, which contains network of intelligent objects worked by own rules, but due to exchange information between one another, their functions are enhanced; (3) connected home has internal networks connected to external ones and allows interactive and remote control of home systems, as well as access to services both inside and outside the house; (4) learning home, able to record and store data about activities patterns in order to use them in future; (5) attentive home, constantly recording data of family members localization in order to predict users’ behavior, satisfying their needs and unobtrusively interact with them. The classification takes into account not only functional abilities of SH, but also highlights different levels of communication within and beyond the house, starting from the simplest systems with mechanical toggle switches, and to complicated systems, forecasting user’s behavior and interacting with him. Ambient intelligence paradigm and the subsequent development of ambient intelligent space have led to formation of the so-called cognitive approach become of great importance for designing SH systems. Intelligent agents embedded into the environment, allow AmI distinguishing people behavior and reacting to their needs in an unobtrusive and even invisible way owing to intuitive interfaces [1]. The cognitive approach to design, with the aim of AmI, gives to the developer a way to compare demands and cognitive processes of the end user. This goal can be reached owing to intellectual agents networks, which provide synchronization of information streams, remote control and user notification, in the aggregate with adaptive interfaces and the means for achievement of full situational awareness [4]. The knowledge of a context helps to explain behavior of the user in a concrete situation and to train the system to react adequately. Cognitive and perceptive systems of a human being automatically use a context for identification in everyday life. The context can be defined as a set of relevant conditions and ambient influences which make a certain situation unique and distinguished. The context implies a notion of weight of influencing factors, considered subconsciously by people but being beyond perception of artificial systems. The optimal approach to formalization of contextual information is an iterative two-step procedure: (1) collection of the detailed expert description of situations; (2) check of accuracy of automatic recognition of the certain contexts on the basis of previously created descriptions [5]. One of the basic directions within the AmI paradigm is a development of multimodal user interfaces providing natural and intuitively clear way of human-computer interaction. Speech, gestures, handwriting and other means, which are natural of interhuman dialogue, are applied to management of machines in multimodal interfaces [6]. Audio-visual information processing allows automatic identifying the person, his intentions, speech, movements and the current position. Due to the analysis of single
Multimodal Interaction with Intelligent Meeting Room Facilities
79
modalities and their subsequent fusion on the semantic level it is possible to improve naturalness and robustness of interactions. Multimodal integration combines semantic hypotheses coming from modules of single modalities processing and carries on joint interpretation of a user’s command. For recognition of user’s intention (communicative or practical) it is necessary to use information from different sources, and to consider various types of the contextual information. Three types of contexts are proposed to use during the fusion of multimodal information [7]: (1) context of a subject domain, which contains some a priori knowledge, the user’s preferences, models of situations, descriptions of objects and subjects, their possible activities and their location to each other; (2) conversational context describing possible dialogues with the system and a current condition of a discourse; (3) visual context including the analysis of a direction of vision, gestures, actions of the user in the course of the observable situation. Data and knowledge, accumulated by the system in advance, determine an interpretation of user’s intentions and the current situation. The result of recognition of a command is, theoretically, a multimodal representation of user’s behavior, conclusion about his intentions and a succession of activities which are to be performed by the system. There are more fields for applying areas of AmI besides that of a SH. One may state that requirements for a SH were worked out already, but requirements for meeting and lecture rooms are somewhat insufficiently standardized. Sometimes even users have no idea what they should expect from such intelligent rooms. The user cannot quickly become familiar with a simultaneous use of several wide-screen displays, interactive multimedia support, integration of mobile devices and others new resources. All these means and tools must be intuitively comprehensible. Usually such rooms function in a semi-automatic mode, and all the embedded systems are maintained by experts. Peculiar features of multimodal interfaces exploiting and human behavior in the intelligent meeting room of SPIIRAS are regarded in the present paper. Testing of audio-visual processing is made during laboratory meetings and lectures. Besides, control of the equipment was realized on a basis of mobile-phone, while collecting of data concerning SPIIRAS members was performed with the aim of a mobile robot with a multimodal interface, moving along a corridor of the institute.
2 Technological Framework of Intelligent Meeting Room A premises of 72 square meters located inside the institute building was supplied for intelligent meeting room. Monitoring of the room is performed by 16 video cameras mounted on the walls, ceiling and tables and provides tracking of moving objects, face detection and other functions. Three T-shape 4 channel microphone arrays mounted on the different walls serve for localization of sound sources, far-field recording and following speech processing. Besides video recording the personal web cameras mounted on the tables have internal microphones and are used for recording speech of each meeting participant.
80
A.L. Ronzhin and V.Yu. Budkov
A wide touchscreen plasma panel and multimedia projector (projection screen) are located one under another in the left side of the room and provide output of multimodal information. Operated electro gears are connected to the projection screen and curtain rail. The curtains are made from special lighttight cloth in order to suppress the outside influence on the illumination changing in the room. The processing of recorded audio-visual data, control the multimedia and electro mechanic equipment are performed by six four-cored computers, two multichannel audio boards Presound FirePod, as well as some devices providing cable and wireless network. The referred service equipment is mounted in a rack and located on the adjacent room from the meeting one. Thus, users inside the meeting room could see only appliances for input/output information, but other devices and computational resources are invisible. To provide service and the same time be hidden for a user is one of the main features of ambient intelligence. Figure 1 shows a technological framework of the intelligent meeting room. Availability of multimodal user interface is a distinguishing characteristic of the developed intelligent meeting room. The developed earlier technologies of audio-visual data processing were successfully implemented in the room to provide natural and intuitively understandable way of interaction with the room equipment. Most important technologies are automatic speech recognition, speaker identification, sound source localization, detection of position and tracking moving objects and user faces, detection of user pose. A bimodal audio-visual Russian speech synthesis technology (talking head) is used for developed interactive applications. A method for spectral-spatial analysis of speech activity has been developed. Spatial speaker localization is based on the calculation of phase difference between the signals recorded by different microphones. The energy level of mutual signal spectrum and estimation of acceptable position of a speaker is used for finding the boundaries of speech in a multichannel acoustical stream, recorded in noisy environments. Russian speech recognition is realized by SIRIUS engine with a model of compact representation of extra large vocabularies based on two-level morphophonemic prefix graph (TMPG). Integration of morpheme and phonetic levels into a united tree-based structure of vocabulary provides compact representation of word-forms and their phonemic transcriptions [8]. Usage of the proposed graph while decoding continuous Russian speech provides formation of grammatically correct words at the recognition output and increase of recognition speed. A model of text-independent speaker identification based on assessment of cepstral features distribution of the input speech signal was developed. The minimal length of the speech signal, necessary for speaker pattern training and decision making during verification, is thirty seconds. A multithreaded program model for speaker identification provides simultaneous estimation of several participants of a meeting. Determination of position and face tracking are based on an algorithm for tracking movements of natural markers of human face. Automatic restoring of lost tracking points allows the face tracking system to increase robustness of identifying headposition during quick motions and occasional video noise. An algorithm for cursor controlling is adaptive to velocity of head movements. It provides comfortable cursor controlling on high-resolution screen, using a low-resolution camera. These algorithms have been embedded into the multimodal system ICANDO for assistance to disabled people at hands-free PC-controlling [6].
81
Room state monitor
Talking head
Multimedia stenographer
Smart board
St. Petersburg map
SPIIRAS inquiry
TV set, radio, AV player
Multimodal Interaction with Intelligent Meeting Room Facilities
Fig. 1. Technological framework of the intelligent meeting room
An algorithm for position and video tracking of moving object is based on estimation of difference between adjacent frames and takes into account the measurements of users, velocity and trajectory of their movements, acceptable regions of user appearance. It makes the algorithm robust to sudden changing luminosity and capable to differentiate moving objects closely located to each other. Intel OpenCV library is used for base procedures of video capture and processing. Also two statistic poses (staying and sitting) are detected in intellectual room applications. A criterion for pose recognition is height of an object. The list of recognized poses could be extended by some gestures, which users often apply during meetings [9]: nod, nutation, rise of a hand by sitting user, hand pointing. A model of audio-visual speech synthesis (“talking head”) of Russian texts has been developed jointly with United Institute of Informatics Problems (Belarus) and University of West Bohemia (Czech Republic). Two methods for creation of visual synthesis of facial organs have been elaborated [10]: (1) model-based method; (2) data-based method. The former is used for creation of 3D avatars. The latter approach allows creating a 2D personalized audio-visual TTS-synthesis model. In this model visual facial synthesis is combined with acoustical speech synthesizer. Collaborative work of the referred technologies supplies a room control system with data about current situation, participant’s behavior and provides robust recognition of voice command due to the analysis of spatial-temporal, situational information and user preferences. Distant speech recognition based on microphone array processing allows a user to control light, curtains, projector screen, PTZ functions of video cameras and more complex devices like TV set, radio, audio-video player and other multimodal
82
A.L. Ronzhin and V.Yu. Budkov
applications. In all the applications interactive feedback is realized by a talking head, which shows an awareness of the room about participant’s behavior and pronounces required speech information.
3 Multimodal Interaction with the Room from Inside First scenario, which was experimentally tested, was focused on conjugation of the actuators (light, curtains, and screen) with the distant speech recognition system and other manual controls. A user could give a command by voice, touchscreen panel, remote control panel or via web-interface from a computer or a mobile phone. A scheme of user command processing and possible automatic room control in accordance with a current situation is presented in Figure 2.
Fig. 2. Scheme of user and automatic control of the equipment of the room
Data about current states of all the actuators, presence of users and their positions are recorded every time at the arrival of a user command. Boolean descriptions of actuator states were used for the light, the curtains, the screen, the plasma panel, the projector. The plasma panel and the projector have certainly many other functions, but their execution was realized without actuators in the framework of concrete applications. For example, changing graphical content showed by TV set and projector is processed by a computer, and then corresponding video signal outputs via DVI or VGA cables. A description of situational information was simplified in given scenario too. Time of day was described by four categories (morning, day, evening, night). Number of participants presented inside the room was divided into four groups too (0, 1, 2-10, more than 10). Positions of participants were evaluated in relation to the meeting table, the plasma panel, the window. The chairs located in the right side of the room are used in the case of large number of presented participants. Therefore a special category of participant’s position was defined: participants are distributed throughout the room. A situational map of the meeting room is used for verification of events and analysis of the room state. Table 1 presents structure and states of the map.
Multimodal Interaction with Intelligent Meeting Room Facilities
83
Table 1. The situational map of the meeting room and automatic commands
Position of participants
>10 All the participants are nearby the meeting table There is a participant near the plasma/screen There is a participant near the window Participants are distributed throughout the room
1
2-10
0
Number of participants
Night
Evening
Day
Morning
Projector on
Plasma panel on
Curtains are closed
Screen is lowered down
Group L4
Time of day
Group L3
Light groups on
Group L2
Situational information
Group L1
State of actuator devices
Commands generated automatically, when specific combination of actuator states and events occurs
1. Turn-on the light group L4 2. Open the curtains 3. Lower the screen 4. Turn-off the light group L1 5. Open the curtains 6. Turn-off whole the light 7. Lift the screen 8. Close the curtains
… The black cells corresponding to the turned-on devices and other features of the room indicated a current situation. The right column consists of examples of the commands, automatically generated at occurrence of corresponding combination of events and equipment state. These commands were received during matching the real user command and the state of situational map. For some commands the conditions arisen them are slightly changed and adapted for implementation in automatic mode. For instance, a voice command to turn-off the light is said by user inside the room before she/he goes out. In automatic mode this action is performed when all the participants left the room. By this reason commands 6-8 in Table 1 to turn-off the light, lift the screen and close the curtains are performed when the number of participants is zero. By Figure 2 now we can explain how to control the meeting room in the automatic mode. Changing the situation caused by user behavior or day time or switching the actuators calls the procedure of the map analysis. If a predefined situation is detected a corresponding command for activation or deactivation of actuators is sent. The interaction between various software modules distributed on several computers is accomplished by client-server architecture via TCP/IP protocol. At that command and notification of the actuator states are collected and processed in queue. Scenarios of control of multimedia appliances were based on special dialogues of speech interaction between a user and the meeting room. The multimodal applications “SPIIRAS inquiry” and “St. Petersburg map” were adopted based on similar systems
84
A.L. Ronzhin and V.Yu. Budkov
realized in a multimodal kiosk [11]. Voice control system for TV set and radio implements commands to select a channel by its number or title, to change the settings of sound and picture. In the application “Smart board” voice commands intended for selecting color, width of pen, brush or other instruments for handwrited sketches on the touchscreen. Only examples of voice commands are mentioned above, which could be useful at interaction with the intellectual applications. Moreover, most of the commands could be activated by gestures on the touchscreen. In particular, in map-based applications a direct pointing to a graphical object is more needed, but speech commands are used for operation with objects [6]. Besides the current state of the dialogue the spatial position of a user should be taken into account. In contrast to control of the actuator devices the voice commands aimed for to the multimedia applications are perceived only from a space near the plasma panel (not far than 1.5 meters). This limitation helps to decrease number of false voice commands appeared as a result of background noise and parallel user conversations that increases accuracy of distant speech recognition. Special attention should be paid to influence of activity of controlled devices during transitional stages on the systems of audio-visual data processing. For instance, the speech recognition and sound source localization systems should be notified before starting the gears of screen and curtains, because their noise influences on the system performance. The analogical situation is appeared at turning-on the multimedia projector and condition system. Since these devices will work continuously then a system of spatial filtration of sound signal should suppress noises before the speech recognition phase. Opening the curtains can significantly influence on changing the luminosity in the room and lead to fail of the video tracking systems, for example loss of existent objects or appearance a false object in the room. The same problem happens when turning-on/turning-off the light, so the systems of video processing should be preliminary notified about changing the luminosity in the meeting room. Verification of the technological framework and experimental detection of potential conflicts at the room control appeared owing to uncoordinated work of equipment or unpredicted behavior of users are conducted. Analysis of the extended situational map, as well as discursive information of real dialogues, personified user data will allow us to extract templates of behaviors and preferences of main groups of users, scenarios of man-machine interactions and most important commands, which should be automatically performed to facilitate efficiency of meetings and lectures in the intelligent room.
4 Control of Meeting Room Facilities from Outside Remote control of room facilities and observation of current situation from outside were realized by web-interface adaptive to hardware-software features of a client’s device. A user could change state of four light groups, curtains, screen and multimedia projector, as well as get picture from five different cameras located inside the room. Formatting the web-page containing the controls and a picture from camera is automatically adapted to the peculiarities of client’s mobile device and browser. Table 2 contains a list of the Nokia mobile devices and their characteristics, which are taken into account at adaptation of web-page format and used in the experiments.
Multimodal Interaction with Intelligent Meeting Room Facilities
85
Table 2. The list of Nokia devices verified at control of the room Device
Operation system
Screen resolutions
Nokia N73
S60 3rd Edition (initial release) Symbian OS v9.1
240x320
Nokia N95
S60 3rd Edition, Feature Pack 1 Symbian OS v9.2
Nokia 5800
S60 5th Edition Symbian OS v9.4
320х240 240x320 320х240 360x640 640х360 Nokia E61 Nokia E60 Nokia E90 Nokia N810
S60 3rd Edition (initial release) Symbian OS v9.1 S60 3rd Edition (initial release) Symbian OS v9.1 S60 3rd Edition, Feature Pack 1 Maemo Internet Tablet OS 2008 Edition
320x240 352x416 800x352 800x480
Browser Auto Touchresolution orientation screen 234x277 234x302 314x200 234x277 + 234x302 314x200 360x493 360x640 + + 502x288 640х360 314x200 314x220 346x346 346x386 794x284 800x352 696x362 + 800x480
The window size, which could be used for web page view, is significantly varied owing to different screen sizes and browser options in the tested devices. The maximal resolution of browsers is used in the full screen mode, when service buttons are hidden. In the several devices the orientation of the screen could be changed manually and automatically that also increases the number of possible browser resolutions. As a result 14 different browser resolutions are applied in the referred list of the tested devices. Moreover, motion of cursor is discrete that requires to use sufficiently large buttons in order to select them using joystick or buttons for cursor control. The presence of touch-screen improves the control speed, but more usability is achieved with large buttons too. All these factors were taken into account during development of the web-page for control of the room facilities. Base layout of the web-page included buttons for turn activator states and the window with room picture presenting its current state and provide visual feedback of turning activators. At that buttons as check boxes and room picture were not overlaid. For each browser resolution a specific web-page layout was designed in order to maximally use the browser screen. First of all the changes applied to placement of the check-boxes and the window with room picture relatively each other, as well as the font size of captions is changed concerning the screen resolution. Three examples of web-page layouts are presented in the Figure 3 in real scale. Placement of elements in the web page is specified using the CSS style tables. The resolution and orientation of screen are checked every 500 ms using Java Script. At the changing screen parameters the corresponding layout of the page is automatically selected and generated for a client.
86
A.L. Ronzhin and V.Yu. Budkov
Fig. 3. Examples of web-page layouts
To increase size of the controls and the room picture a pop-up menu overlaid the picture window was designed in the last version of the web-page. As a result the room picture window uses the browser resolution completely, but the menu is appeared when user need to change state of the activators or chose another camera, as it is shown in Figure 4. In the initial mode the menu is hidden and a small icon located in right upper corner of the browser reveals about existence of the menu. Since the cameras send picture, which size has aspect ratio of 4x3, at the wide screen the room picture would be shown incorrectly. To avoid this problem the proportion of the picture is saved, but released space is used for talking head model, using of which increases the naturalness of interaction. The talking head pronounces notifications coming from the room server and confirms the performing of the user commands and other changes occurred in the room. The further research will be focused on introduction the multimodal interfaces into the mobile devices, since control by dozen controllers with different functions via web browser and mobile phone is not so convenient owing to not so large screen. Taking into account limited computational resources of the mobile devices a distributed model of audio-visual processing will be used. The signals recorded by the mobile devices will be transferred to the room server, where speech recognition, head tracking and other technologies will be applied for them. Such approach will allow user to communicate with the room using speech, gestures and other natural modalities.
Multimodal Interaction with Intelligent Meeting Room Facilities
87
Fig. 4. Examples of web-page layouts with pop-up menu and talking head model
5 Conclusion The developed intelligent meeting room is a distributed system with the network of intelligent agents (software modules), actuator devices, multimedia equipment and audio-visual sensors. The main aim of the room is providing of meeting or lecture participants with required services based on analysis of the current situation. Awareness of the room about spatial position of the participants, their activities, role in the current event, their preferences helps to predict the intentions and needs of participants. Context modeling, context reasoning, knowledge sharing are stayed the most important challenges of the ambient intelligent design. Assignment of easy-to-use and well-timed services, at that stay invisible for user, is one of another important feature of ambient intelligent. In the developed intelligent room all the computational resources are located in the adjacent premises, so the participants could observe only microphones, video cameras, as well as equipment for output of visual and audio information. Implementation of multimodal user interface capable to perceive speech, movements, poses and gestures of participants in order to determinate their needs provides the natural and intuitively understandable way of interaction with the intelligent room. Development of a network of intelligent meeting rooms gives the opportunity to organize a videoconference between spatially distributed participants and facilitates to increase collaboration, access to higher knowledge/competence, reduce costs for transport and staff, and increase the quality of education due to automatic immediate monitoring by every student during the lessons. Using the various combinations of multimodal interfaces and the equipment of the intelligent meeting room the fundamental issues of human-machine interaction are studied and applied models in security medicine, robotics, logistics and other scientific areas are investigated now. Acknowledgments. The intelligent meeting room of SPIIRAS was supplied in 2008 at the financial support of the Russian Foundation for Basic Research. Full spectrum of mobile devices was donated by Nokia to study issues of remote user interaction with a smart space.
88
A.L. Ronzhin and V.Yu. Budkov
References 1. Ducatel, K., Bogdanowicz, M., Scapolo, F., Leijten, J., Burgelman, J.-C.: ISTAG - Scenarios of Ambient Intelligence in 2010. European Commission Community Research (2001) 2. Aldrich, F.: Smart Homes: Past, Present and Future. In: Harper, R. (ed.) Inside the Smart Home, pp. 17–39. Springer, London (2003) 3. Gann, D., Venables, T., Barlow, J.: Digital Futures: Making Homes Smarter. Chartered Institute of Housing, Coventry (1999) 4. Masakowski, Y.: Cognition-Centric Systems Design: A Paradigm Shift in System Design. In: 7th International Conference on Computer and IT Applications in the Maritime Industries, pp. 603–607 (2008) 5. Degler, D., Battle, L.: Knowledge management in pursuit of performance the challenge of context. Performance Improvement 39(6), 25–31 (2007) 6. Tzovaras, D. (ed.): Multimodal User Interfaces: From Signals to Interaction. Springer, Heidelberg (2008) 7. Chai, J., Pan, S., Zhou, M.: MIND: A Context-based Multimodal Interpretation Framework. Kluwer Academic Publishers, Dordrecht (2005) 8. Ronzhin, A.: Topological peculiarities of morpho-phonemic approach to representation of the vocabulary of Russian speech recognition. Bulletin of Computer and Information Technologies (9), 12–19 (2008) 9. Wallhoff, F., Zobl, M., Rigoll, G.: Action segmentation and recognition in meeting room scenarios. In: International Conference on Image Processing (ICIP 2004), pp. 2223–2226 (2004) 10. Lobanov, B., Tsirulnik, L., Železný, M., Krňoul, Z., Ronzhin, A., Karpov, A.: AudioVisual Russian Speech Synthesis System. Informatics, Minsk, Belarus 20(4), 67–78 (2008) 11. Karpov, A., Ronzhin, A.: An Information Enquiry Kiosk with a Multimodal User Interface. In: 9th International Conference Pattern Recognition and Image Analysis (PRIA 2008), Nizhny Novgorod, Russia, pp. 265–268 (2008)
Ubi-Check: A Pervasive Integrity Checking System Michel Banˆ atre, Fabien Allard, and Paul Couderc INRIA Rennes / IRISA http://www.irisa.fr/aces
Abstract. Integrity checking is an important concern in many activities, such as logistic, telecommunication or even day to day tasks such as checking for someone missing in a group. While the computing and telecommunication worlds commonly use digital integrity checking, many activities from the real world do not beneficiate from automatic mechanisms for ensuring integrity. RFID technology offer promising perspectives for this problem, but also raises strong privacy concerns as they are usually based on global identification and tracking. In this paper we present an alternative approach, Ubi-Check, based on the concept of coupled physical objects which enable integrity checking relying only on local interactions, without the support of a global information system.
1
Introduction
Integrity checking is an important concern in many activities, both in the real world and in the information society. The basic purpose is to verify that a set of objects, parts, components, people remains the same along some activity or process, or remains consistent against a given property (such as a part count). In the real world, it is a common step in logistic: objects to be transported are usually checked by the sender (for their conformance to the recipient expectation), and at arrival by the recipient. When a school get a group of children to a museum, people responsible for the children will regularly check that no one is missing. Yet another common example is to check for our personal belongings when leaving a place, to avoid lost. While important, these verification are tedious, vulnerable to human errors, and often forgotten. Because of these vulnerabilities, problems arise: E-commerce clients sometimes receive incomplete packages, valuable and important objects (notebook computers, passports etc.) get lost in airports, planes, trains, hotels, etc. with sometimes dramatic consequences. While there are very few automatic solutions to improve the situation in the real world, integrity checking in the computing world is a basic and widely used mechanism: magnetic and optical storage devices, network communications are all using checksums and error checking code to detect information corruption, to name a few. The emergence of Ubiquitous computing and the rapid penetration of RFID devices enables similar integrity checking solutions to work for physical objects. S. Balandin et al. (Eds.): NEW2AN/ruSMART 2009, LNCS 5764, pp. 89–96, 2009. c Springer-Verlag Berlin Heidelberg 2009
90
M. Banˆ atre, F. Allard, and P. Couderc
The purpose of this paper is to present the design of such a system, and one of its application. The paper is organized as follows: in next section, we detail the problem. Then we present the design and implementation of the Ubi-Check system. Finally, some related works and perspectives are discussed.
2
The Problem
Let’s focus on a typical application scenario, which well help identify the key issues of the problem. Consider someone at the airport who is going to cross the security gate. He is required to wear off his jacket, his belt, to put in a container his mobile phone, his music player, to remove from his bag his notebook computer, and may be other objects... All that in hurry, with other people in the queue doing the same. Obviously, personal objects are vulnerable to get lost in this situation: objects can get stuck inside the scanner, can stack up on each other at the exit of the scanner, and it is easy to forget something while being stressed to get a flight. Another vulnerability is to get the object of someone else because, such as a notebook computer of the same model. The vulnerability is introduced because: 1. objects belonging to a common set have to be separated from each other in some occasion 2. getting them back together is not checked by reliable process Consider what’s happen in a computer network: digital objects are fragmented into “packets” which can be transported by independently of each other in the network. When they arrive at a destination point, packets are assembled together to rebuild the original object, which is checked for integrity. For this purpose, packets include additional information enabling error detection. Of course, networks are more complex than this simple view, with multiple encapsulation and fragmentation levels, but for the analogy with real objects and people, the basic principle is sufficient: We can consider a set of physical objects as “data” which are going to be transported and eventually separated at some occasions. At some point where the set of physical object is assumed to be complete, integrity checks will take place. For instance, in our airport security gate scenario, the integrity check would be performed on leaving the zone. Our goal is to propose an integrity checking system that could be integrated at strategic places to warn people when missing objects are detected, or that they are carrying someone’s else object. Such a system would turn some area into smart spaces where people would not have to worry of object lost, which is interesting for trains, hotels, etc. Such a system is only interesting if it can be realistically deployed, given the constraints of the real-world. To this end, some important requirements have to be considered :
Ubi-Check: A Pervasive Integrity Checking System
1. 2. 3. 4. 5.
91
ease of use and as low as possible impact on existing processes low cost for the user scalability and reliability ease of on-site integration privacy respect
We think important to insist on the two latter requirements. Integration issues can lead to death of emerging technologies or experimental systems: the cost of integrating something new into an operational infrastructure is very high, and dependence or impact on existing information systems should be as low as possible for a chance of acceptance. Privacy concerns raise strong resistance to RFID technology [8]. As we will see, a core idea of Ubi-Check is to ensure anonymous operation and no dependence on databases.
3
System Design
The Ubi-Check system is based on the principle of coupled objects. Coupled objects are a group of physical objects that are logically associated together, meaning that they carry digital information referencing other objects from the set, or representing their membership to the group. An important property is that this information is physically stored on the object. Typically, this information will be stored on RFID memory tags embedded on the objects. In our application scenario, this means that users of the Ubi-Check system would have their important objects enabled with tags. Ideally, those tags could be embedded into the object at build time by the manufacturer, but user installed tags could of course be added for objects not ready for the service. Then, there are two procedures in the system. A first one consisting of associating all the objects of a group (ie the objects of a person). And a second one
Fig. 1. Group creation and checking
92
M. Banˆ atre, F. Allard, and P. Couderc
where integrity will be checked at the appropriate places. The figure 1 sums up the process, which we detail in the following. 3.1
Group Creation
At this step, the user presents himself in small area in the range of an RFID writer, and a group is initialized for all its tagged objects: a signature is computed from individual identifiers of the tags. The identifiers can be those attributed at tag construction, or generated for the Ubi-Check system. The latter case is better to protect user’s privacy, because new identifiers can be used each time a group is created, thereby reducing the risk of user tracking by their objects. A user could create a new group a each trip for example. As we can see, a group is made of a set of identifiers. We need to store the group representation somewhere, such that the group integrity could be checked at appropriate places. Storage in a database is a straightforward solution, but it would requires that each checkpoint could access this database, through a communication infrastructure which raises several issues: – Deployment and operating cost – Reliability and scalability, because checking operation would be dependant on the availability of the communication infrastructure and the remote database, and the communication load would increase linearly with the number of users of the service – Privacy, as group representation associated with individuals would we stored in a central database. However, depending on the nature of the object identifiers and the group representation that is used, this issue may be mitigated. These issues conflict with our design goals which motivate an alternative solution, that would not depend on a remote service to operate. In the concept of coupled objects previously mentioned, the logical association between the objects (or the group membership) is part of the physical objects themselves. This can be easily implemented with RFID tags, which in addition to the identifier part, can provide a programmable memory of up to a few kilobit. The group representation will be stored in this memory. Because the size is limited and the integrity check should be fast, the group is represented by a signature, computed by a hash code function. A good discussion of hash functions in the context of RFID is [3]. This approach enables full autonomous operation of both the association points and the checkpoints. An additional property can be stored in a particular object, considered as the owner of the group: the cardinal of the the set. The owner tag would typically be associated with an object that the user would always keep with him, such as his watch. We will see in the next section how it is used. 3.2
Checking Phase
Once a group is formed, the user can move away with his objects. He can separate from his objects, but if he pass a checking point without the complete set, a warning is shown.
Ubi-Check: A Pervasive Integrity Checking System
93
The checking point is made of an RFID reader controlling a double antenna set up arranged close to each other (typically separated by one meter), in order to detect objects crossing the checkpoint. A time frame Δt is set to allow a group of objects to cross the gate. In practice, we have used an interval of 2 to 3 seconds with good success for typical pedestrian flow. A cyclic buffer logs all the identifier i passing the gate, the timestamp of the event ti , and the signature Si of the tag. The integrity check is triggered for each group that is reaching the end of its time frame, that is : ∀i such as t0 ≤ tsi ≤ t0 + Δt and Si = X where t0 is the timestamp when the first identifier of the group of signature X is read. The hash code H(i0 , ..., in ) is checked against the X, and three cases have to be considered : 1. H(i0 , ..., in ) = X, meaning that the set is complete. In Ubi-Check this is shown on as green status at the exit of the checkpoint. 2. H(i0 , ..., in ) = X, and one of the identifier has the owner status described in previous section. This means that there is at least one missing object from the group. Ubi-Check reports a warning of missing objects. The number of missing objects is known as the owner tag includes the cardinal of the set. 3. H(i0 , ..., in ) = X, and no identifier has owner status. Ubi-Check reports that you are carrying one or more objects that do not belong to you. 3.3
Implementation and Experimentation
A prototype of the system has been implemented: it uses FEIG HF 13.56 Mhz readers, controlled by a standard embedded PC, and standard HF read/write tags. A single unit can play both the roles of the group creator (association), and checkpoint. The figure 2 shows a simple set up with an LCD display used to report status. Performance of the readers allow typical rate of capture of 10 tags/s, which in practice translates into checking up to 2 users per second. As we can expect with RFID technology, and especially in this context of free mobility and on-the-fly tags acquisition, mis-read are possible. The occurrence are highly dependant on the placement of the tags in the objects, the nature and the environment of the objects. While some advices can be suggested to the users for optimal tag placement, metal proximity and other perturbation cannot always be avoided. However, as the system is based on a multiplicity of tags, read failures typically lead to false warnings, not missed warning as failing the read all the tags from a user is very unlikely. In case of a false warning, the user can pass again the checkpoint. The system was experimented with the at the “Fete de la science” in November 2008, a two days event where the general public was invited to experiment with recent scientific and technological developments. The feedback was very positive regarding the relevance of service. However, the reading reliability is currently not robust enough to allow operational deployments in environments
94
M. Banˆ atre, F. Allard, and P. Couderc
Fig. 2. A simple Ubi-Check set up
with sustained and continuous flow of people, such as airports. But in other contexts with relaxed constraints, such as in hotels, the system performance could be adequate.
4
Related Works
RFID is a hot topic with many issues given its broad application domain and emerging success in security, accountability, tracking, etc. However, the UbiCheck service and its underlying coupled-objects principle differ than many RFID systems where the concept of identification is central and related to database supported information systems. In some works, the tags memory are used to store semantic information, such as annotation, keywords, properties [1,7]. Ubi-Check is in the line of this idea: RFID are used to store in a distributed way group information over a set of physical artifacts. The concept using distributed RFID infrastructure as pervasive memory storage is due to Bohn and Mattern [2]. Maintaining group membership information in order to cooperate with “friend devices” is a basic mechanism (known as pairing or association) in personal area networks (PAN) such as Bluetooth or Zigbee. Some personal security systems based on PAN for luggages were proposed [5], which enable the owner to monitor some of his belongings, such as his briefcase, and trigger an alarm when the object is out of range. A major drawback of active monitoring is the energy
Ubi-Check: A Pervasive Integrity Checking System
95
power which is required, as well as poential conflicts with radio regulations that can exist in some places, such as in particular in airplanes. Still in the context of Bluetooth, RFID has also been used to store PAN addresses in order to improve discovery and connexions establishment time [9]. It can be seen as storing “links” between physical objects, such as in Ubi-Check, but without the idea of a fragmented group. Yet another variant is FamilyNet [6], where RFID tags are used to provide intuitive network integration of appliances. Here, there is a notion of group membership, but it resides on information servers instead of being self-contained in the set of tags as in Ubi-Check. Probably the closest concept to Ubi-Check is SmartBox [4], where abstractions are proposed to determine common high level properties (such as completeness) of groups of physical artifacts using RFID infrastructures.
5
Conclusion
We presented a service for enabling smart spaces to check that people do not leave a protected area without all their belongings, or with objects of other people. The system is based on completely autonomous checkpoints, and logical group distributed over a set of physical artifacts. The strong points of this solution are its independence of any remote information system support or network support, and user’s privacy respect as it is anonymous and does not relies on global identifiers. As we have seen, RF reading reliability have to be improved for some application scenarios. We have also examined other scenarios, such as checking at home the integrity of a complex medical prescription with a group of medication set up by a druggist. In further research we are investigating other variations of the coupled-objects concept to improve trust and accountability in logistic and e-commerce.
References 1. Banˆ atre, M., Becus, M., Couderc, P.: Ubi-board: A smart information diffusion system. In: Balandin, S., Moltchanov, D., Koucheryavy, Y. (eds.) NEW2AN 2008 / ruSMART 2008. LNCS, vol. 5174, pp. 318–329. Springer, Heidelberg (2008) 2. Bohn, J., Mattern, F.: Super-distributed RFID tag infrastructures. In: Markopoulos, P., Eggen, B., Aarts, E., Crowley, J.L. (eds.) EUSAI 2004. LNCS, vol. 3295, pp. 1–12. Springer, Heidelberg (2004) 3. Feldhofer, M., Rechberger, C.: A case against currently used hash functions in RFID protocols, pp. 372–381 (2006) 4. Floerkemeier, C., Lampe, M., Schoch, T.: The smart box concept for ubiquitous computing environments. In: Proceedings of sOc 2003 (Smart Objects Conference), Grenoble, May 2003, pp. 118–121 (2003) 5. Kraemer, R.: The bluetooth briefcase: Intelligent luggage for increased security (2004), http://www-rnks.informatik.tu-cottbus.de/content/unrestricted/ teachings/2004/
96
M. Banˆ atre, F. Allard, and P. Couderc
6. Mackay, W., Beaudouin-Lafon, M.: Familynet: A tangible interface for managing intimate social networks. In: Proceedings of SOUPS 2005, Symposium On Usable Privacy and Security. ACM, New York (July 2005) 7. Noia, T.D., Sciascio, E.D., Donini, F.M., Ruta, M., Scioscia, F., Tinelli, E.: Semantic-based bluetooth-rfid interaction for advanced resource discovery in pervasive contexts. Int. J. Semantic Web Inf. Syst. 4(1), 50–74 (2008) 8. Peris-Lopez, P., Castro, J.C.H., Est´evez-Tapiador, J.M., Ribagorda, A.: Rfid systems: A survey on security threats and proposed solutions. PWC, 159–170 (2006) 9. Salminen, T., Hosio, S., Riekki, J.: Enhancing bluetooth connectivity with rfid. In: PERCOM 2006: Proceedings of the Fourth Annual IEEE International Conference on Pervasive Computing and Communications, Washington, DC, USA, pp. 36–41. IEEE Computer Society, Los Alamitos (2006)
Towards a Lightweight Security Solution for User-Friendly Management of Distributed Sensor Networks Pentti Tarvainen1, , Mikko Ala-Louko1, Marko Jaakola1, Ilkka Uusitalo1 , Spyros Lalis2 , Tomasz Paczesny3, Markus Taumberger1 , and Pekka Savolainen1 1 VTT Technical Research Centre of Finland, Kaitov¨ ayl¨ a 1, P.O. Box 1100, FI-90571 Oulu, Finland [email protected] 2 Centre for Research and Technology Thessaly (CERETETH), Technology Park of Thessaly, 1st Industrial Area, GR 385 00, Volos, Greece [email protected] 3 Warsaw University of Technology (WUT), Pl. Politechniki 1, 00-661 Warszawa, Poland [email protected]
Abstract. Wirelessly networked sensors and actuators are a cornerstone of systems that try to introduce smartness in a wide range of outdoor and indoor environments. The robust and secure operation of such systems is of central importance and will definitely play a key role in terms of their adoption by organizations and individuals. However, the commonly applied security mechanisms based on current standards are either resource consuming, inconvenient to use, or both. In this paper, a generic, lightweight and user-friendly approach to implement secure network management and confidentiality in a Distributed Sensor Networks (DSNs) is proposed. Keywords: information security, distributed sensor network, wireless network, network management.
1
Introduction
A new era of applications programming is coming, triggered by the appearance of the new generation of wireless embedded microcontroller and sensoractuator devices. Besides mobile phones and PDAs (Personal Data Assistants), programmable nodes with sensing and actuating capabilities are already embedded in household appliances and cars; in the recent years compute-sense-actuate units also come as separate, small, multipurpose and ad-hoc wireless devices (aka motes) which can be deployed in a straightforward way. With the increasing power-efficiency of embedded systems and ad-hoc wireless technologies, we can expect wireless distributed monitoring and control systems and applications spreading from high-end to more commodity use.
Corresponding author.
S. Balandin et al. (Eds.): NEW2AN/ruSMART 2009, LNCS 5764, pp. 97–109, 2009. c Springer-Verlag Berlin Heidelberg 2009
98
P. Tarvainen et al.
Such systems, henceforth referred to as Distributed Sensor Networks (DSNs), are expected to have a strong connection to our everyday life, and at least economic losses are expected in case of failures during their operation. The additional value of wireless DSNs is largely in their flexible deployment and communication capability. However, this also opens up most of the security1 vulnerabilities of such systems. There are known attack methods to compromise the ad-hoc communication channel between the nodes of a network. There are also precaution mechanisms built into the communication standards, to support the generation of network keys as well as the encryption and decryption of the messages. But increased security usually translates to increased cost, both in terms of development effort and run-time resource consumption. At the same time, the system must be maintenance-free or at least easily manageable even by non-technical users. To make matters worse, most of the security aids provided to the application developer are specific to the networking technology being used, i.e., there is the risk of having to re-define security-related protocols and implementations as part of the system’s evolution. Hard application coding and the lack of portability of the security logic – or low security and reliability – can turn out to be an obstacle for the DSN technology penetration to the mass market. Motivated by these challenges, this paper presents a lightweight and portable security solution for DSNs designed by the authors in the POBICOS project [1]. The solution enables the user to manage such a system in a straightforward way and achieves sufficient data confidentiality at the application level. The rest of this paper is organized as follows: Section 2 briefly discusses the related work. Section 3 provides a short introduction to the DSNs and their inherent vulnerabilities, security threats and requirements. In section 4 we propose two novel approaches to manage security in the DSNs, thus providing practical mechanisms and security protocols for implementation in typical DSN context. Section 5 discusses implementation issues of the security abstraction and, finally, section 6 summarizes the paper and delineates the future work.
2
Related Work
Existing security methods are available for many of the threats present in sensor networks but they are usually too resource consuming to be used in DSNs [2,3,4]. Actuator nodes, when present in the DNS, introduce increased risk. However, most of the research done in the field of DSNs does not consider that actuator nodes could exist in the network. A security related work that takes actuator nodes into account has been published in [5,6,7]. DSN security issues and a public key cryptography based key establishment scheme for DSNs are given in [7]. A cluster based communication architecture around each actuator of a DNS could be used to reduce the key management overhead. To suit this demand, a suchlike, scalable, energy efficient routing algorithm is introduced in [5]. 1
In this paper, the term security is used to refer to information security as well as protection against unauthorized access and control.
Towards a Lightweight Security Solution
99
Sensor networks, such as the ZigBee [8] and the WirelessHART [9], are based on the IEEE 802.15.4 [10] standard. For example, in the ZigBee there are two different modes defined for security-related operations; the “commercial mode” and the “residential mode”. In the “commercial mode”, three different types of encryption keys are used to ensure network security. Hence, the memory required for the ZigBee Trust Center grows with the number of devices in the network and it also requires implementation of key establishment using specific protocol and entity authentication. Due to these issues, the “commercial mode” is not suitable for purposes of the DSNs. In the “residential mode”, only one network key can be used. All devices using this shared key encrypt and decrypt information. This approach requires little resources from the ZigBee Trust Center. The memory required for the Trust Center does not grow with the number of devices in the network. Due to that, this mode can be exploited in the DSN security approach. However, the network protocol is not encrypted during the first steps of delivery of the network key and the “residential mode” lacks in security until the network key is spread to all the devices in the network. Moreover, in the “residential mode” the IEEE 802.15.4 based sensor networks are vulnerable to “insider attacks”, i.e. a malicious node knowing the network key can attack the system from inside.
3
Distributed Sensor Networks (DSNs)
Distributed sensor networks consist of large amounts of spatially distributed, selforganizing, low power, low cost sensor nodes deployed to monitor and possibly control certain physical variables such as temperature, humidity or luminosity. The capabilities of these sensor nodes include computing, sensing, wireless communication and optionally actuating. Each node is also responsible of routing in a multi-hop fashion, constituting effectively an ad-hoc network with other nodes. Earlier it was common to deploy large numbers of similar sensor nodes to a sensor field to monitor usually a single physical variable such as temperature [2]. Nowadays DSNs are becoming more common in the domain of private homes as well, in the spirit of the Internet of Things or cooperating objects. Not only does this cause significant heterogeneity, also in many application scenarios DSNs handle confidential data that has to be kept secure and private from outsiders. This creates the need for security and privacy mechanisms [3,11]. New approaches are required that allow to manage DSNs intuitively while maintaining security and privacy of the users home and data. However, the implementation of these features on networks without infrastructure, consisting mostly of low-resource nodes, is much more troublesome than in infrastructure supported, high-resource networks. The amount of overhead must be kept small to minimize the energy consuming radio transmission time among the sensor nodes, memory space for key storage is limited, and the computational power used for security functions has to be kept small. In addition, the security mechanisms have to be scalable to support networks with varying amounts of nodes [6]. These constraints limit how extensive the used security features can eventually be.
100
P. Tarvainen et al.
Due to the wireless nature of communication the DSNs are vulnerable to several security attacks such as eavesdropping, denial-of-Service (DoS), tampering, selective forwarding, sinkhole attack, wormhole attack, and sybil attack. The DSNs have also specific security requirements such as data confidentiality, data authenticity, data integrity, and data freshness. Detailed descriptions of these threats and requirements can be found for example in [2,4,12,13].
4
Security Protocols and Mechanisms for the DSN
At the network level, our proposed security solution for the DSN Devices offers two levels of security: (1) a One Network Key Approach ( Figure 1) and (2) on utilization of the public and private key pairs, i.e. an Identity-based Cryptography Approach (Figure 2). These two approaches are alternative and selectable by application developer. The security of the DSNs is based on utilization of a security card (S-Card) as a network-specific trust centre, bearing responsibility over all the security-related configuration actions. The S-Card is like a safe deposit box of the network, withholding the network’s sensitive configuration data. The S-Card is networkspecific, thus it does not interfere with other (e.g. next-door neighbours’) networks beside the one it belongs to. User access to the S-Card is secured by a personal security code. Delivery of the network keys into the devices can be performed by means of the S-Card with close proximity. A Trust Center distributes network keys for the purpose of the DSN and end-to-end application configuration management. The Trust Center software is running on the S-Card. 4.1
One Network Key Approach
In the following we focus on the functionality of the S-Card in the case of the One Network Key Approach. (Figure 1). In this approach, the DSN Device securely communicates with other devices using one 128-bit network key and network identification code. For purposes of trust management, the device accepts an active network key originating from the Trust Center via unencrypted key transport with close proximity. The Trust Center maintains a list of devices and network keys of all the devices in the network and controls policies of network admittance. Network key can be updated periodically. Furthermore, it is assumed that “insider attacks” are not possible due to the limitations of the One Network Key Approach, i.e. no rogue device is assumed to get hold of the network key. In user authentication, the DSN security solution exploits the S-Card with integrated keyboard and user-configurable security code. In addition, each SCard contains an extended security code which can be used to reset devices in the network in the cases of security code or S-Card getting lost. In following steps, close proximity is used in communication between the S-Card and the DSN Devices during key distribution.
Towards a Lightweight Security Solution
101
DSN Network 1 Security-Card1: Network Key1, Network ID1 Trust Center Security Code1, Extended Security Code1 Ext
Close Proximity: NWK Key1, NWK ID1 Close Proximity: NWK Key1, NWK ID1
NWK Key1 + NWK ID1 Blocked
Outside Attacker
DSN Network 2 User
Security-Card2: Network Key2, Network ID2
Close Proximity: NWK Key2, NWK ID2 N
Inside Attacker Unblocked NWK Key2 + NWK ID2
Trust Center Security Code2, Extended Security Code2
Close Proximity: NWK Key2, NWK ID2
Fig. 1. One Network Key Approach
A new S-Card is taken into use with the steps as follows: 1. User acquires a S-Card from genuine dealer and brings the S-Card home, 2. The S-Card comes with a manual containing the extended security code for the S-Card. The user stores the extended security code in a safe place, 3. User activates the S-Card by pressing a button on it and is prompted to type in a security code of choice, 4. User types in a security code and is asked to confirm it by retyping, 5. S-Card generates a network key and network ID from the S-Card’s 64-bit hardware address. As a result, the S-Card now contains the user’s security code, extended security code, network key and network ID, 6. User takes the S-Card into use and activates DSN Devices in his/her home A user can manage several DSNs by acquiring additional S-Cards which can each be configured to use the same security code. Each S-Card contains the security credentials of the corresponding network in addition with the user specified security code. The user can move DSN Devices between different DSNs by using a single security code and the S-Cards of each network respectively. Adding a new device to the network can be performed as follows: 1. User brings the S-Card close to a DSN Device, 2. Near Field Connection (NFC) between the S-Card and the DSN Device can now be established,
102
P. Tarvainen et al.
3. User enters the security code (in the case of a new S-Card the security code is entered twice), 4. User is verified by the security code known by the S-Card (the S-Card is personal to the user), 5. The S-Card sends the user’s security code, S-Card’s extended security code, network key and network ID to the DSN Device. In addition, the S-Card sends an up-to-date list of other DSN members, the Member Table, to the new DSN Device, 6. If the user’s security code and the S-Card’s extended security code correspond to the ones previously transmitted to the DSN Device, network key, network ID and Member Table are accepted by the Device. If this is the first security code that the device hears, it learns it and stores it into its memory along with the S-Cards extended security code, network key, network ID and Member Table, 7. Having the required security credentials, the DSN Device attempts to join the network, 8. At this point, the new DSN Device gets its DSN address that is calculated from the underlying network’s network address at the hardware layer (HAL), 9. The new DSN Device broadcasts exactly once, with unreliable transport messaging, a new Member Table after it has added its own address to it, 10. The S-Card and all the other DSN Devices in the network update their Member Tables with the one that contains the new Device After the steps above the DSN IS READY TO OPERATE IN SECURE MANNER, i.e. air interface is secured against “outsider attacks”. Autonomous Device leaving from the network happens as follows: 1. When in range of the DSN, the S-Card monitors the network state by receiving updated Member Tables that are being broadcasted when changes in the network membership occur. For this purpose, the S-Card utilizes similar communication technology used by normal DSN Devices, i.e. it is a member of the DSN when not in NFC-mode. If the S-Card notices that a single DSN Device has dropped out from the Member Table permanently, a global network key change is issued in the network by the S-Card, 2. The S-Card receives an updated Member Table from the DSN that indicates that a DSN Device has left the network, 3. The S-Card waits for a constant time period during which it pings the missing device periodically. If the DSN Device that was missing reappears, i.e. answers to ping, no operations are needed and the S-Card is no longer in the waiting mode, 4. The constant time period ends and the DSN Device hasn’t reappeared in the network, 5. The S-Card generates a new network key from its 64-bit hardware address and issues a global network key change by sending the new network key to each node explicitly by using reliable communications, 6. Once every device in the DSN has acknowledged the new network key transport message, the S-Card changes its own network key to the newly generated
Towards a Lightweight Security Solution
103
one. If a single DSN Device is not in range of the network during the network key change it is dropped out and must be added again to the network as specified in section “Adding a new device to the network” Removing a Device from the network is performed as follows: 1. The user wants to remove a malfunctioning Device X from the DSN via the S-Card, 2. The S-Card initiates a global network key change as specified in section “Autonomous Device leaving from the network” in such a way that the new network key is sent to all devices in the DSN except the Device X which is to be left out of the DSN, 3. After the network key change, the S-Card broadcasts a new Member Table where Device X has been dropped out, 4. The remaining devices in the network communicate now with the new network key encryption and the malfunctioning Device X is left out of the DSN If the user forgets his/her security code, the extended security code of the S-Card can be used to reset a DSN Device into a state where it accepts a new security code, i.e. the step described in section “Adding a new device to the network” can be performed. The device reset must be performed on every device separately with NFC-connection. Similarly, the S-Card’s user-configurable security code is reset when using the extended security code. If the user has lost the extended security code of the S-Card, a genuine dealer must be contacted. If the user loses the S-Card of the DSN, a new S-Card must be acquired from a genuine dealer. Having the new S-Card, the user can reset all his/her DSN Devices by typing the extended security code of the lost S-Card while in NFCconnection mode. This action must be performed on every Device respectively. After this, the new S-Card can be used to reconfigure the network as specified earlier. If both S-Card and the corresponding extended security code are lost, the user is required to get the memories of his/her DSN Devices flashed by a dealer or technician. 4.2
Identity-Based Cryptography Approach
In Identity-Based Cryptography Approach, the users’ or devices’ identity information, such as unique names, can be used as public keys for encryption and/or signature verification [14,15,16]. Identity-based cryptography reduces the complexity and management cost of public-key cryptography systems by eliminating the need for a key directory and public-key certificates. Identity-based cryptography allows any party to generate a public key from a known identity value. A trusted third party, called the Private Key Generator (PKG), included in the Trust Center, generates the corresponding private keys. To operate, the PKG first publishes a master public key, and retains the corresponding master private key. Given the master public key, any party can compute a public key corresponding to the identity ID by combining the master public key with the identity
104
P. Tarvainen et al.
DSN Network 1 Security-Card1: Master Public Key1, Master Private Key1 Trust Center + PKG
Close Proximity: Master Public Key
Public Key1 = Master Public Key1 + Node ID1
Node ID Private Key = Master Private Key + Node ID
Close Proximity: Security Code1, See the steps Extended Security Code1 a above
Encryption with Public Key + Node ID Blocked Outside Attacker Inside Attacker
Decryption with Private Key
User
Security-Card2: Master Public Key2, Master Private Key2
DSN Network 2 Close Proximity: See the steps above
Encryption with Public Key + Node ID
Trust Center + PKG Security Code2, Extended Security Code2
Blocked
Close Proximity: See the steps above
Decryption with Private Key
Fig. 2. Identity-based Cryptography Approach
value. To obtain a corresponding private key, the DSN Device with identity ID contacts the PKG, which uses the master private key to generate the private key for identity ID. As a result, parties may encrypt messages (or verify signatures) with no prior distribution of keys between individual participants. In [17] the authors propose to assign a descriptive identity to nodes, meaning the identity will represent the function of the node in the network. If there is more than one node with the same function, the identity will include additional information to differentiate those nodes, e.g. location or serial information. When DSN Device A wants to send information to another DSN Device B, it will first obtain the identity information of B from device’s own local database, and then use this information, together with master public key, to encrypt the information. Each DSN Device will also be granted a private key associated with its public identity information and will use that key to decrypt information destined to it. The Trust Center generates the key pair for each DSN Device in the network. The private key would be deployed by means of close proximity (Figure 2). The working steps in more detail is as follows: 1. The Trust Center generates a master public key, Master-key, and transfers it to the DSN Device with close proximity, 2. A DSN Device joins the network and authenticates with the Trust Center by using its identity information, Node ID. The membership approval can be done manually by means of close proximity,
Towards a Lightweight Security Solution
105
3. Being approved, the DSN Device is granted with a private key, associated with its identity information by means of close proximity, 4. Devices publish their Node ID in cleartext along with their Node ID and address encrypted with their private key. If Device A wants to communicate with Device B, it decrypts B’s encrypted contact info using the master public key and the published Node ID of B. If the decrypted Node ID matches the published Node ID, A knows that the decrypted address corresponds to B. The information sent to B can be encrypted using A’s private key, and B can decrypt it using the same (symmetrical) process, based on A’s published (or piggybacked) contact info, 5. The encrypted message passes through multiple hops to reach the destination. The message is secure since only the designated DSN Device can decrypt and read the information using its private key In a system of N nodes, N key pairs and N node IDs are needed to retrieve and maintained, so the implementation would not be too complex. Also the Trust Center would need only a bit more resources to generate and distribute the key pairs.
5
Implementation of the Security Abstraction
At network level, the DSN is wireless and it can be located for example on top of star, tree, and mesh topologies. Security related issues are hidden from end-users as far as possible. At node level, the DSN security solution exploits a stack architecture principle made up of a set of layers (Figure 3). Each layer performs a specific set of services for the layer above. The architecture builds on this foundation by providing a Network and Security Adaptation Layer and a framework for a Core Middleware (CMI) Layer. The Core Middleware (CMI) Layer includes a Communications Module (CommM) Layer that allows frame security and device authentication to be based on network and cryptography keys. Furthermore, the layer is responsible for the processing steps needed to securely transmit outgoing frames, securely receive incoming frames, and securely establish and manage network and cryptography keys. In addition, the layer is responsible for network management, device authentication, and software data encryption. The Network and Security Adaptation Layer acts as “glue” between hardware and the core middleware. The layer includes Best-effort Transport, Reliable Transport, and Fragmentation services. The layer is responsible for (1) processing steps needed to securely transmit outgoing frames, and (2) securely receive incoming frames. The underlying network technology (i.e. the Medium Access Control (MAC) Layer and the Physical (PHY) Layer) below the Network and Security Adaptation Layer may vary. The basic principle in our security approach is that the underlying layers adopt the security mechanisms from the used sensor network technology if available. In addition, a software encryption scheme is introduced to support platforms with no prior security mechanisms.
P. Tarvainen et al.
SECURITY SUITES
Data Entity Service Access Point
Management xME-SAP Entity Service Access Point
DSN Defined
EntityHALI
Communications Module (CommM) DatagramTransportI
ReliableTransportI
NetworkMgmtI
Network and Security Adaptation Layer MLDE-SAP
Layer Interface xDE-SAP
API-SAP Generic Instructions
Core Middleware (CMI) Layer
S-Card including Trust Center One Network Key Approach or Identity-based Cryptography Approach
APPLICATION
API-SAP Non-Generic Instructions
MLME-SAP
Medium Access Control (MAC) Layer PLDE-SAP
PHY Layer, HARDWARE
PLME-SAP Air Interface
Underlying NWK Defined
106
Fig. 3. DSN security architecture at node level
In proposed security solution of the DSNs the actual physical network technology below the Network and Security Adaptation Layer may vary and the underlying layers adopt the security mechanisms from the used technology. As an example, in following we focus on implementation of the security abstraction in the case of the ZigBee based networks. To avoid the lack in the “residential mode” the DSN security solution exploits the proposed S-Card approach in close proximity delivery of the network keys to all the devices in the network. After the delivery is completed, the DSN Devices are able to communicate via interface in a secure manner. The proposed One Network Key Approach is able to protect the DSN against the “outsider attacks”, but not against the “insider attacks”. However, there are possibilities to achieve this, although the system will get a bit more complex as the number of keys and the complexity of algorithms grows. A solution against “insider attacks” is to provide node and message authentication by using the proposed Identity-based Cryptography Approach. The Datagram Transport and Reliable Transport use the HW encryption provided by the ZigBee radio processor (instead of the optional SW encryption), so the encryption takes place underneath the Network and Security Adaptation Layer. The encryption key is initialized during the joining into the network. Functionality of the Network Management is as follows: – Network key changes (for hardware encryption) are implemented by configuring the proper key in, and then committing a ZigBee stack restart, – A global network key change sends the new key for every network member explicitly; after this every node reboots into the new security scheme after a coarse synchronization period,
Towards a Lightweight Security Solution
107
Table 1. Commands and events of the NetworkMgmtI -interface
Command or Event
Short Description
Attaches the node into a DSN, which is idencommand int joinNetwork (idtype NetworkID, keytype tified with NetworkID and encrypted with NetworkKey, addrtype *Addr) NetworkKey. After a successful joining, the node’s new device address is in Addr. command int leaveNetwork ()
Removes the node from the current DSN.
command int Returns the current local knowledge of other getNetworkMembers (addrtype network members into Addr in a list form. Total count of the other members is stored in **Addr, int *len) len. This event’s purpose is to update the node’s event void knowledge of other network members. networkMembersChanged (addrtype **Addr, int *len) command int banNodeFromNetwork (addrtype Addr)
Used to ban a node from the network.
command void Writes the node’s DSN address in Addr. getMyNetworkAddr (addrtype *Addr) command int isReachable (addrtype *Addr)
Queries the liveliness of a node.
event void reachableResp (int result)
Denotes the result of liveliness query.
command int Used for setting the network ID locally. The setNetworkIDLocally (idtype command is needed in both ”initial” joining into the DSN, and also during a network NetworkID) change. command int setNetworkKeyLocally (