Lecture Notes in Computer Science Commenced Publication in 1973 Founding and Former Series Editors: Gerhard Goos, Juris Hartmanis, and Jan van Leeuwen
Editorial Board David Hutchison Lancaster University, UK Takeo Kanade Carnegie Mellon University, Pittsburgh, PA, USA Josef Kittler University of Surrey, Guildford, UK Jon M. Kleinberg Cornell University, Ithaca, NY, USA Alfred Kobsa University of California, Irvine, CA, USA Friedemann Mattern ETH Zurich, Switzerland John C. Mitchell Stanford University, CA, USA Moni Naor Weizmann Institute of Science, Rehovot, Israel Oscar Nierstrasz University of Bern, Switzerland C. Pandu Rangan Indian Institute of Technology, Madras, India Bernhard Steffen TU Dortmund University, Germany Madhu Sudan Microsoft Research, Cambridge, MA, USA Demetri Terzopoulos University of California, Los Angeles, CA, USA Doug Tygar University of California, Berkeley, CA, USA Gerhard Weikum Max-Planck Institute of Computer Science, Saarbruecken, Germany
6294
Sergey Balandin Roman Dunaytsev Yevgeni Koucheryavy (Eds.)
Smart Spaces and Next Generation Wired/Wireless Networking Third Conference on Smart Spaces, ruSMART 2010 and 10th International Conference, NEW2AN 2010 St. Petersburg, Russia, August 23-25, 2010 Proceedings
13
Volume Editors Sergey Balandin Nokia Research Center Office of CTO Itamerenkatu 11-13 00180 Helsinki, Finland E-mail:
[email protected] Roman Dunaytsev Tampere University of Technology Department of Communications Engineering Korkeakoulunkatu 10 33720,Tampere, Finland E-mail:
[email protected] Yevgeni Koucheryavy Tampere University of Technology Department of Communications Engineering Korkeakoulunkatu 10 33720, Tampere, Finland E-mail:
[email protected]
Library of Congress Control Number: 2010931611 CR Subject Classification (1998): C.2, H.4, D.2, K.6.5, D.4.6, H.3 LNCS Sublibrary: SL 5 – Computer Communication Networks and Telecommunications ISSN ISBN-10 ISBN-13
0302-9743 3-642-14890-5 Springer Berlin Heidelberg New York 978-3-642-14890-3 Springer Berlin Heidelberg New York
This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, re-use of illustrations, recitation, broadcasting, reproduction on microfilms or in any other way, and storage in data banks. Duplication of this publication or parts thereof is permitted only under the provisions of the German Copyright Law of September 9, 1965, in its current version, and permission for use must always be obtained from Springer. Violations are liable to prosecution under the German Copyright Law. springer.com © Springer-Verlag Berlin Heidelberg 2010 Printed in Germany Typesetting: Camera-ready by author, data conversion by Scientific Publishing Services, Chennai, India Printed on acid-free paper 06/3180
Preface
We welcome you to the joint proceedings of the anniversary 10th NEW2AN 2010 (Next Generation Teletraffic and Wired/Wireless Advanced Networking) and Third ruSMART 2010 (Are You Smart) conferences held in St. Petersburg, Russia during August 23–25, 2010. Originally the NEW2AN conference was launched by ITC (International Teletraffic Congress) in St. Petersburg in June 1993 as an ITC-Sponsored Regional International Teletraffic Seminar. The first implementation was entitled “Traffic Management and Routing in SDH Networks” and was hosted by R&D LONIIS. In 2002 the event received its current name, NEW2AN. In 2008 NEW2AN was joined by a new counterpart in smart spaces, ruSMART, hence boosting interaction between researchers, practitioners and engineers from different areas of ICT. This year, in its 10th implementation, NEW2AN/ruSMART is an established conference with a unique cross-disciplinary mix of telecommunications science in Russia. NEW2AN/ruSMART has always featured outstanding keynotes from universities and companies from Europe, USA and Russia. The 10th NEW2AN technical program addressed various aspects of nextgeneration network architectures. New and innovative developments for enhanced signaling protocols, QoS mechanisms, cross-layer optimization, traffic characterization were also addressed within the program. In particular, issues of QoS in wireless and IP-based multiservice networks were dealt with, as well as financial aspects of future networks. It is also worth mentioning that emphasis was place on wireless networks, including, but not limited to, cellular networks, wireless local area networks, personal area networks, mobile ad hoc networks, and sensor networks. The Third Conference on Smart Spaces, ruSMART 2010, provided a forum for academic and industrial researchers to discuss new ideas and trends in the emerging area of smart spaces that creates new opportunities for making fully customized applications and services for the users. The conference brought together leading experts from top affiliations around the world and there was active participation and strong interest from Russian attendees with a good reputation for high-quality research and business in innovative service creation and applications development. This year the technical program of both conferences benefited from two keynote speakers: – Vladimir Gorodetsky, St. Petersburg Institute for Informatics and Automation of Russian Academy of Sciences, Russia – Ian Oliver, Nokia Research Center, Finland – Alexey Vinel, St. Petersburg Institute for Informatics and Automation of Russian Academy of Sciences, Russia
VI
Preface
We wish to thank the Technical Program Committee members of both conferences and associated reviewers for their hard work and important contribution to the conference. This year the conferences were organized in cooperation with the FRUCT Program, ITC (International Teletraffic Congress), IEEE, Tampere University of Technology, Popov Society, and supported by NOKIA, Nokia Siemens Networks and Sententia Ltd. The support of these organizations is gratefully acknowledged. Finally, we wish to thank many people who contributed to the organization. In particular, Jakub Jakubiak (TUT) carried a substantial load of the submission and review process, the website maintenance, did an excellent job on the compilation of camera-ready papers and in the interaction with Springer. Many thanks go to Natalia Avdeenko, Ekaterina Antonyuk and Ekaterina Akimova (Monomax Meetings & Incentives) for their excellent local organization efforts and the conference’s social program preparation. We believe that the work done for the 10th NEW2AN and Third ruSMART conferences provided an interesting and up-to-date scientific program. We hope that participants enjoyed the technical and social conference program, the Russian hospitality and the beautiful city of St. Petersburg. June 2010
Sergey Balandin Roman Dunaytsev Yevgeni Koucheryavy
Organization NEW2AN International Advisory Committee Nina Bhatti Igor Faynberg Jarmo Harju Andrey Koucheryavy Villy B. Iversen Paul K¨ uhn Kyu Ouk Lee Mohammad S. Obaidat Michael Smirnov Manfred Sneps-Sneppe Ioannis Stavrakakis Sergey Stepanov Phuoc Tran-Gia Gennady Yanovsky
Hewlett Packard, USA Alcatel Lucent, USA Tampere University of Technology, Finland ZNIIS R&D, Russia Technical University of Denmark, Denmark University of Stuttgart, Germany ETRI, Republic of Korea Monmouth University, USA Fraunhofer FOKUS, Germany Ventspils University College, Latvia University of Athens, Greece Sistema Telecom, Russia University of W¨ urzburg, Germany State University of Telecommunications, Russia
NEW2AN Technical Program Commitee TPC Chair Roman Dunaytsev Mari Carmen Aguayo-Torres Ozgur B. Akan Khalid Al-Begain Sergey Andreev
Tampere University of Technology, Finland
University of Malaga, Spain METU, Turkey University of Glamorgan, UK State University of Aerospace Instrumentation, Russia Tricha Anjali Illinois Institute of Technology, USA Konstantin Avrachenkov INRIA, France Francisco Barcelo UPC, Spain Sergey Balandin Nokia, Finland Thomas M. Bohnert SAP Research, Switzerland Torsten Braun University of Bern, Switzerland Chrysostomos Chrysostomou University of Cyprus, Cyprus Nirbhay Chaubey Institute of Science and Technology for Advanced Studies and Research (ISTAR), India Ibrahim Develi Erciyes University, Turkey Eylem Ekici Ohio State University, USA Sergey Gorinsky Washington University in St. Louis, USA Markus Fidler NTNU Trondheim, Norway Giovanni Giambene University of Siena, Italy Stefano Giordano University of Pisa, Italy Ivan Ganchev University of Limerick, Ireland Vitaly Gutin Popov Society, Russia
VIII
Organization
Andreas Kassler Maria Kihl Tatiana Kozlova Madsen Yevgeni Koucheryavy Jong-Hyouk Lee Vitaly Li Leszek T. Lilien Saverio Mascolo Maja Matijaˇsevic Paulo Mendes Pedro Merino Ilka Miloucheva Dmitri Moltchanov Edmundo Monteiro Se´an Murphy Marc Necker Nitin Nitin Jaypee Mairtin O’Droma Evgeni Osipov George Pavlou Veselin Rakocevic Simon Pietro Romano Alexander Sayenko Dirk Staehle Sergei Semenov Burkhard Stiller Weilian Su Arvind Swaminathan Dmitry Tkachenko Vassilis Tsaoussidis Christian Tschudin Andrey Turlikov Kurt Tutschku Alexey Vinel Lars Wolf
Karlstad University, Sweden Lund University, Sweden Aalborg University, Denmark Tampere University of Technology, Finland (Chair) INRIA, France Kangwon National University, Republic of Korea Western Michigan University, USA Politecnico di Bari, Italy University of Zagreb, FER, Croatia INESC Porto, Portugal University of Malaga, Spain Salzburg Research, Austria Tampere University of Technology, Finland University of Coimbra, Portugal University College Dublin, Ireland University of Stuttgart, Germany University of Information Technology, India University of Limerick, Ireland Lulea University of Technology, Sweden University of Surrey, UK City University London, UK Universita’ degli Studi di Napoli “Federico II”, Italy Nokia Siemens Networks, Finland University of W¨ urzburg, Germany Nokia, Finland University of Z¨ urich and ETH Z¨ urich, Switzerland Naval Postgraduate School, USA Qualcomm Inc., USA IEEE St. Petersburg BT/CE/COM Chapter, Russia Demokritos University of Thrace, Greece University of Basel, Switzerland State University of Aerospace Instrumentation, Russia University of Vienna, Austria SPIIRAN, Russia Technische Universit¨ at Braunschweig, Germany
NEW2AN Additional Reviewers V.V. Govindaswamy C. Hoene J. Jakubiak
N. Nitin S. Poryazov M. Smirnov
N. Vassileva
Organization
ruSMART Executive Technical Program Committee Sergey Boldyrev Nikolai Nefedov Ian Oliver Alexander Smirnov Vladimir Gorodetsky Michael Lawo Michael Smirnov Dieter Uckelmann Cornel Klein Maxim Osipov
Principle Scientist, Nokia Research Center, Helsinki, Finland Senior Research Engineer, Nokia Research Center, Zurich, Switzerland Principle Scientist, Nokia Research Center, Helsinki, Finland Head of laboratory, SPIIRAS, St. Petersburg, Russia Head of laboratory, SPIIRAS, St. Petersburg, Russia Professor, Center for Computing Technologies (TZI), University of Bremen, Germany Fraunhofer FOKUS, Germany LogDynamics Lab, University of Bremen, Germany Program Manager, Siemens Corporate Technology, Germany Siemens CT, Embedded Linux, Russia
ruSMART Technical Program Committee Juha Laurila Sergey Balandin Alexey Dudkov Didem Gozupek Kim Geunhyung Reto Krummenacher Prem Jayaraman Michel Banˆ atre Sergei Bogomolov Gianpaolo Cugola Dimitri Konstantas Markus Taumberger Bilhanan Silverajan Aaron J. Quigley Pedro Merino Victor Govindaswamy
Nokia Research Center, Switzerland Nokia Research Center, Finland University of Turku, Finland Bogazici University, Turkey Dong Eui University, Republic of Korea STI Innsbruck, Austria Monash University, Australia IRISA, France LGERP R&D Lab, Russia Politecnico di Milano, Italy University of Geneva, Switzerland VTT, Finland Tampere University of Technology, Finland University College Dublin, Ireland University of Malaga, Spain Texas A&M University, USA
ruSMART Additional Reviewers M. Baqer J. Jakubiak
Y. Koucheryavy
IX
Table of Contents I
ruSMART
Smart Spaces Use Cases Extending Context Spaces Theory by Proactive Adaptation . . . . . . . . . . . Andrey Boytsov and Arkady Zaslavsky
1
Context-Awareness Enabling New Business Models in Smart Spaces . . . . Boris Moltchanov, Christian Mannweiler, and Jose Simoes
13
A Middleware Architecture for Safety Critical Ambient Intelligence Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Antonio Coronato and Giuseppe De Pietro
26
Ranking Algorithm by Contacts Priority for Social Communication Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Natalia Dudarenko, Juwel Rana, and K˚ are Synnes
38
CrysP: Multi-Faceted Activity-Infused Presence in Emerging Social Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Saguna, Arkady Zaslavsky, and Dipanjan Chakraborty
50
Smart-M3 Platform Multichannel System of Audio-Visual Support of Remote Mobile Participant at E-Meeting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Andrey L. Ronzhin, Victor Yu Budkov, and Alexey A. Karpov On-the-Fly Ontology Matching in Smart Spaces: A Multi-model Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Alexander Smirnov, Alexey Kashevnik, Nikolay Shilov, Sergey Balandin, Ian Oliver, and Sergey Boldyrev
62
72
Access Control in Personal Localized Semantic Information Spaces . . . . . Alexey Koren and Alexander Buntakov
84
Integration of an Answer Set Engine to Smart-M3 . . . . . . . . . . . . . . . . . . . . Vesa Luukkala and Jukka Honkola
92
A Video Monitoring Model with a Distributed Camera System for the Smart Space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Andrey L. Ronzhin, Maria Prischepa, and Alexey Karpov
102
XII
Table of Contents
Smart Spaces Solutions Physical Approach in Smart Homes: A Proposition and a Prototype . . . . Michele Dominici, Giulio Zecca, Fr´ed´eric Weis, and Michel Banˆ atre SMEO: A Platform for Smart Classrooms with Enhanced Information Access and Operations Automation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Akshey Jawa, Sudip Datta, Sujeeth Nanda, Vishal Garg, Vasudeva Varma, Suresh Chande, and Murali Krishna Punaganti Venkata Creative Approach to the Design and Prototyping of Experimental Smart Spaces, Case Studies from the Interactive Environments Minor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Tomasz Jaskiewicz, Walter A. Aprile, and Aadjan van der Helm Event Recognition via Energy Efficient Voting for Wireless Sensor Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Mohamed Baqer
II
111
123
135
148
NEW2AN
Performance Evaluation Evaluation of Multi-service CDMA Networks with Soft Blocking . . . . . . . Villy B. Iversen Design of MPLS over DWDM Architecture for Unicast and Anycast Flows . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Jakub Gladysz and Krzysztof Walkowiak Comparative Analysis of Sleep Mode Control Algorithms for Contemporary Metropolitan Area Wireless Networks . . . . . . . . . . . . . . . . . Alexey Anisimov, Sergey Andreev, Olga Galinina, and Andrey Turlikov Determining Normalized Measure of Dispersion for Evaluating the Performance of IEEE 802.11 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Faisal Iradat and Sayeed Ghani Development of a System to Promote Eco-Driving and Safe-Driving . . . . Ryosuke Ando, Yasuhide Nishihori, and Daisuke Ochi
160
172
184
196 207
Performance Modeling Oligopoly Game Modeling for Cognitive Radio Environments . . . . . . . . . . Ligia C. Cremene, D. Dumitrescu, and R´eka Nagy
219
Table of Contents
Reducing the Number of States for Markovian Model of Optical Slotted Ring Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Mateusz Nowak and Piotr Pecka Performance Modelling of Automatic Identification System with Extended Field of View . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Troels Laursen, Hans Peter Mortensen, Nikolaj Bisgaard Pedersen, Ulrik Wilken Rasmussen, Tatiana K. Madsen, and Jens Dalsgaard Nielsen Performance Measures Computation for a Single Link Loss Network with Unicast and Multicast Traffics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Irina Gudkova and Olga Plaksina Realization of a Method of Matrix-Geometric Progression . . . . . . . . . . . . . Yuri Ryzhikov
XIII
231
242
256
266
Delay-/Disruption-Tolerant Networking and Overlay Systems Mobility Pattern Based Routing Algorithm for Delay/Disruption Tolerant Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ManKyu Park, SeokHong Min, SangHo So, DeockGil Oh, ByungChul Kim, and JaeYong Lee
275
Heuristic Congestion Control for Message Deletion in Delay Tolerant Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Lertluck Leela-amornsin and Hiroshi Esaki
287
The FDPOO: A Flexible Delivering Platform with at-One-time and On-time guarantees in Opportunistic Networks . . . . . . . . . . . . . . . . . . . . . . Yuan-Tse Yu, Chung-Ming Huang, and Hsing-Cheng Wang
299
A Distributed Telematics Peer-to-Peer (D-TP2P) Networking System for Service Discovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Chung-Ming Huang, Chia-Ching Yang, and Chun-Yu Tseng
311
Empirical Predictor of TCP Throughput on a Multihop Wireless Path . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Anna Chaltseva and Evgeny Osipov
323
A Framework for Connectivity in Inter-working Multi-hop Wireless Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Oladayo Salami, Antoine Bagula, and H. Anthony Chan
335
XIV
Table of Contents
Integrated Wireless Networks Practical Vertical Handover Mechanisms between Satellite and Terrestrial Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . MinSu Shin, ManKyu Park, DeockGil Oh, ByungChul Kim, and JaeYong Lee Geo-aware Handover of Mission Agents Using Opportunistic Communication in VANET . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Edison Pignaton de Freitas, Tales Heimfarth, Fl´ avio Rech Wagner, Armando Morado Ferreira, Carlos Eduardo Pereira, and Tony Larsson
353
365
Comparison and Evaluation of the Most Efficient Mobility Management Protocols for Integrated Wireless Networks . . . . . . . . . . . . . . . . . . . . . . . . . . Asimakis Lykourgiotis and Stavros Kotsopoulos
377
Functional Model and Service Scenarios for QoS Enabled Mobile VoIP Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Kyu Ouk Lee, Sang Soo Lee, and Tae Whan Yoo
389
Heuristic Approaches to the Multi-objective Network Design and Optimization for Wireless Data Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . Chutima Prommak and Naruemon Wattanapongsakorn
398
Resource Management Call Admission Control in Single and Two-Tier Cellular Networks . . . . . . Manfred Schneps-Schneppe and Janis Sedols
411
Joint Admission Control and Resource Allocation for Multiuser Loading in LTE Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . V. Venkatkumar, T. Haustein, and M. Faulkner
421
Data Traffic Scheduling Algorithm for Multiuser MIMO-OFDM System with the Throughput Maximization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Jerzy Martyna
436
Multimedia Communications Scalable Video Coding Based on Three-Dimensional Discrete Pseudo Cosine Transform . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Eugeniy Belyaev, Timofey Sukhov, and Liu Kai
448
NTRULO: A Tunneling Architecture for Multimedia Conferencing over IP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A. Amirante, T. Castaldi, L. Miniero, and S.P. Romano
460
Table of Contents
XV
Seamless Streaming Service Session Migration Support Architecture for Heterogeneous Devices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Geun-Hyung Kim and Bong-Hwan Lee
473
Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
485
Extending Context Spaces Theory by Proactive Adaptation Andrey Boytsov and Arkady Zaslavsky Department of Computer Science and Electrical Engineering Luleå University of Technology, SE-971 87 Luleå {Andrey.Boytsov,Arkady.Zaslavsky}@ltu.se
Abstract. Context awareness is one of the core features of pervasive computing systems. Pervasive systems can also be improved by smart application of context prediction. This paper addresses subsequent challenge of how to act according to predicted context in order to strengthen the system. Novel reinforcement learning based architecture is proposed to overcome the drawbacks of existing approaches to proactive adaptation. Context spaces theory is used as an example of how existing context awareness systems can be enhanced to achieve proactive adaptation. This recently developed theory addresses problems related to sensors uncertainty and high-level situation reasoning and it can be enhanced to achieve efficient proactive adaptation as well. This article also discusses implementation options and possible testbed to evaluate the solutions. Keywords: Context awareness, context prediction, context spaces theory, pervasive computing, reinforcement learning, proactive adaptation, act-ahead adaptation.
1 Introduction Pervasive computing is a paradigm where computing systems are integrated into the everyday life in a non-intrusive, graceful and transparent manner. Some implementations of pervasive computing paradigm include smart homes, elderly care systems, smart mobile devices, GPS navigators, RFID tracking systems, social networks. Context awareness is one of the basic features of pervasive computing. Context prediction is also recognized as a challenge and an opportunity. However, for context prediction and especially for acting on predicted context there is a definite lack of universal approach to the problem. This paper proposes and motivates the approach for contextual act-ahead adaptation – proactive adaptation to predicted context. The strengths and challenges of proposed approach are discussed. The article is structured as follows. Section 2 describes related work and current challenges of proactive adaptation to predicted context. Section 3 further elaborates identified challenges and proposes and explains the solution approach. Section 4 describes the essentials of context spaces theory and introduce ECORA framework. Sections 5 and 6 discuss the integration of proposed approach into context spaces S. Balandin et al. (Eds.): ruSMART/NEW2AN 2010, LNCS 6294, pp. 1–12, 2010. © Springer-Verlag Berlin Heidelberg 2010
2
A. Boytsov and A. Zaslavsky
theory and introduce CALCHAS – Context Aware aCt aHead Adaptation System. Section 7 introduces a sample testbed scenario. Section 8 makes a summary, provides future research plans and concludes the paper.
2 Context Prediction and Acting on Predicted Context Most of context prediction approaches use machine learning techniques. Predictive models that were applied to context prediction task include sequence predictors [5], neural networks [22], Bayesian networks [17], branch predictors [16], decision tree learning [11], time series prediction [19], Markov models [7], trajectory extrapolation [1]. Some authors actually do recognize acting on predicted context as a specific problem ([3], [4]), but still there is not much research done for the challenge of proactive adaptation. Generally most of the ways to act on predicted context can be classified in two groups. 1. Rule-based engines. For every particular prediction result there is a rule that defines an action. Improving behavior of the system is achieved by improving efficiency of context prediction. 2. Learning by example. That approach was applied mostly in smart home environments. Pervasive computing system tracks user actions and then it starts executing the actions for the user. On the basis of studying and analyzing existing approaches, summary representation can look like expression set (1). state(t)=SenseAndProcessData(t) prediction(t)=PredictionModel(state(t),history(t),prediction(t-1 ) history(t+1)=addToHistory(state(t),history(t) ) action(t)=actOnPrediction(state(t),prediction(t),history(t)) .
(1)
Where state(t) is entire context state at the time t, including the results of sensor data acquisition, validation and processing. Entire aggregated history up to time t, but not including time t, is stored in history(t). Prediction results at time t for time t+1 and maybe subsequent steps are referred to as prediction(t). Usually they depend on the current state and on the dependencies learned from history. Sometimes previous prediction results can influence them as well, if time horizons of prediction attempts do overlap. System is acting on predicted context, so action at time t ܽܿ݊݅ݐሺݐሻ depends on prediction results and current state. In case learning by example is implemented, action also has some learning mechanisms and therefore depends on history. The model presented in (1) has a very serious drawback – it cannot handle mutual dependency between system actions and prediction results. Those drawbacks and solution opportunities will be addressed in section 3. Notable exceptions that do not fall in that model are the applications of Markov decision processes and Q-learning based approaches to context prediction [24],[6],[13]. They will be mentioned further in section 3 when discussing reinforcement learning solutions.
Extending Context Spaces Theory by Proactive Adaptation
3
3 Proactive Adaptation as Reinforcement Learning Task Sometimes the system has to provide results of context prediction in quite specific conditions. Here are some motivating use cases. Consider elderly care system. Users need some assistance in accomplishing everyday tasks. System is able to give an advice, call a caregiver or just do nothing if the user is performing quite well. The task is to predict the probability of successful outcome of the activity and to act to maximize that probability, but not at the cost of wasting the time of the caregiver or annoying the user with unnecessary advices. Use case is partially based on [10]. Another motivating use case can be a smart home, where user satisfaction can be estimated according to both direct questioning and the number of user manual interventions into a configuration of home parameters. The task is to predict user’s cumulative satisfaction and act to increase it. Also the system should do its best to behave in non-intrusive manner and not to annoy the user with too many questions. One more example might be prediction and prevention of memory shortage in pervasive system. If memory shortage was predicted with the certainty above the threshold, the system takes actions to avoid memory shortage condition (e.g. searches for additional resources to share the work). But if those preventive actions are successful, the predictor would have learning problem. Formally if there was no memory shortage, it is a prediction failure. In reality most likely it is not, but we cannot claim it for sure. Actually, when the system starts decision making, predictor becomes unable to learn properly - predictor cannot distinguish between prediction failure and successful prevention. For all those cases, neither rule-based system, nor learning by example can solve the problem completely. The solution for this and many similar use cases is to take decision making process into account while making predictions. The dependencies for those use cases can be described like the expressions (2). state(t)=SenseAndProcessData(t) prediction(t)=PredictionModel(state(t),history(t),prediction(t-1) , action(t)) history(t+1)=addToHistory(state(t),history(t) ) action(t)=actOnPrediction(state(t),prediction(t),history(t)) .
(2)
The meaning of state(t), history(t), action(t) and other elements in (2) is the same as in expressions (1). The difference between (2) and (1) is marked in bold. The additional dependency makes most current proactive adaptations methods inapplicable. System acts on predicted context, but in turn predicted context depends on system actions. Most of the context prediction approaches we mentioned previously use predictive models, which do not take into account the mutual dependency between system actions and prediction results. Usually this problem is avoided by completely splitting the context in two independent parts: the one that is affected by actions and the one that is predicted. For example, if the system intends to proactively open the door before the user comes into the room, the choice whether the system opens the door or leaves it closed will not change user intentions and, therefore, prediction results. Learning by example is also the way to avoid mutual dependency problem, but this approach has very limited applicability: it works only if possible user actions and
4
A. Boytsov and A. Zaslavsky
system actions significantly overlap and only with the assumption that imitating user actions grants acceptable effectiveness of the system. We propose an enhanced method to solve mutual dependency problem between actions and predictions. That problem can actually be viewed as reinforcement learning task. Reinforcement learning problem is a problem faced by an agent that should find an acceptable behavior in a dynamic environment and learn from its trial and errors [12]. Reinforcement learning in application to pervasive computing task is almost not researched. Notable exceptions are Markov decision processes and Q-learning approaches [24],[6],[13]. However, as it will be discussed further in this section, Markov decision processes have very limited applicability due to the features of pervasive computing systems. Recent advancements in reinforcement learning include the approaches, which to the best of our knowledge were not applied to pervasive computing at all, including predictive state representation [23], most cases of neural network control [8], different applicable cases of actor-critic model (like [9]), selforganizing maps and Q-learning integration [20] and many more. To the best of our knowledge, there was no attempt to address particular features of reinforcement learning in pervasive computing area. Those features are: • Continuous action spaces. Actuators sometimes provide the choice from practically continuous range of values. • Continuous state spaces. Many kinds of sensors produce values from practically continuous range as well. However, this problem can be resolved by the means of situation awareness. • Mobility of nodes. Mobility of nodes in pervasive computing system can cause the loss of connection to certain sensors and actuators. Or, on the contrary, new connections to different sensors and actuators can be established. • High dimensionality. The count of sensors and actuators in pervasive computing system can be very high. • High heterogeneity of data. Discrete-valued, continuous-valued or event-based sensors and actuators can appear in any combination. • Ability to incorporate prior knowledge. Sometimes common sense or expert estimations can give some sketches of good acting strategies. Ability to incorporate them can significantly reduce learning time. • Explicit prediction result can also provide some insight into the problem. • Limited exploration capabilities. Pervasive system cannot just do a random thing to see what happens. • Limited time for decision making. • Goals of the user can change instantly. The features mentioned above seriously limit the scope of reinforcement learning solutions that we can try. In particular, those features mean that Markov decision process – the dominating model for reinforcement learning – can be used only in limited set of special cases. The main reason here is MDPs’ discrete state and action spaces, which might be not suitable for pervasive computing solutions. In sections 4-6 we will discuss in more details, how we can address pervasive computing challenges presented in the list above.
Extending Context Spaces Theory by Proactive Adaptation
5
4 Context Spaces Theory – Main Concepts Some features of proactive adaptation in pervasive computing can be addressed by improving context spaces theory. The theory of context spaces [14] is a recently developed approach for context awareness and reasoning which addresses the problems of sensors uncertainty and unreliability. It also deals with situation reasoning and the problems of context representation in a structured and meaningful manner. Context spaces theory is designed to enable context awareness in clear and insightful way. This theory uses spatial metaphors for representing context as a multidimensional space. To understand context spaces theory we need to introduce several new terms. Any kind of data that is used to reason about context is called context attribute. Context attribute usually corresponds to a domain of values of interest, which are either measured by sensors directly or calculated from other context attributes. It can be either numerical value or a value from pre-defined set of non-numerical options. Context state represents the set of all relevant context attributes at a certain time. A set of all possible context states constitutes application space. Therefore, application space can be viewed as a multi-dimensional space where the number of dimensions is equal to the number of context attributes in the context state. The state of the system is represented by a point in the application space and the behavior of the system is represented by a trajectory moving through the application space over time. Situation space is meant to represent real life situation. It can be defined as subspace of the application space. So if context state is in the subspace representing situation S, it means that situation S is occurring. Situation S has the level of occurrence certainty. It depends on the probability of the context state to be within S and it also depends on the subspace within S. See Figure 1 for simple illustration.
Fig. 1. Context spaces theory
In context spaces theory several methods were developed for reasoning about the context. Bayesian reasoning [21] or Dempster-Schafer algorithm [18] are used to get overall confidence in the fact that a situation is occurring. Algebraic operations on situations and some logic-based methods were developed for reasoning in terms of situations [14].
6
A. Boytsov and A. Zaslavsky
Some solutions were developed to integrate various context prediction methods into the theory of context spaces [2]. However, those prediction methods are based on pure forecasting and do not take into account decision making aspects. Context spaces theory was implemented in ECORA [15] – Extensible Context Oriented Reasoning Architecture. ECORA is a framework for development of contextaware applications. That framework provides its functionality as a set of Java classes to be integrated into the prototypes of context-aware systems. The presented research introduces proactive adaptation methods that can be used to extend the context spaces theory and enhance ECORA-based applications.
5 Integrating Proactive Adaptation into Context Spaces Theory Context spaces theory has all the necessary capabilities to ensure context awareness. However, when it comes to proactive adaptation, some enhancements have to be made. At first, the actuators need to be introduced into the model. Actually, those actuators can constitute a separate space in the manner much like the context space itself. That actuator space can be treated as action space for reinforcement learning. Considering different meaning of sensors and actuators, it seems that the better solution is to separate the spaces for sensors and actuators. Situation spaces in the space of actuators are not likely to have any value (or, in case a set of discrete actions can be elicited, those situations will just look like points, not spaces). In case of the lost connection to actuator, the corresponding axis is removed. Most likely, it will take some time for the system to learn how to act in new conditions, but previously learnt data can provide a good starting point and reduce adaptation time. Similar refers to introducing new actuators – combining old data with exploration can provide adaptation to new capabilities. Another question is how to define reinforcement learning state space for context spaces theory. Having discrete state space is likely to make the task simpler. One solution is to take situations as states for reinforcement learning and trigger the state if the certainty of situation occurring is above some threshold. To fit the Markov property (that is essential for different kinds of Markov decision processes), situation decomposition algorithm presented in [2] can be used. That kind of situation decomposition approach can also help to overcome node mobility problem from sensor point of view: if the sensor loses the link or if new sensor is introduced, situation properties are recalculated (see [14] for the details), but it is transparent for the upper layer, including the proactive adaptation. Also taking context space as state space for reinforcement learning is an option. One more option is to take the degrees of certainty in different situations as continuous state space. To summarize, context spaces theory can be enhanced to incorporate proactive adaptation solutions. Those context spaces theory enhancements can deal with one of the main features of pervasive computing – the mobility of sensors and actuators – using geometrical metaphors and situation awareness. Addressing other features of proactive adaptation depends mostly on exact reinforcement learning method and the architecture of proactive adaptation solutions of upper layers.
Extending Context Spaces Theory by Proactive Adaptation
7
6 CALCHAS Prototype The proposed approach can be implemented by creating a general-purpose middleware for context prediction and proactive adaptation based on context spaces theory. CALCHAS (Context Aware Long-term aCt aHead Adaptation System) prototype is now under development. Integrated with ECORA, it will allow achieving efficient proactive adaptation for context spaces theory. The architecture was designed to address all the features of pervasive computing area that were mentioned in section 3. That middleware solution should incorporate a library of different context prediction and proactive adaptation methods. Middleware development is in progress and the prototype is being implemented. We propose the following architecture for CALCHAS (see Figure 2).
Fig. 2. CALCHAS general architecture
The architectural blocks have following purpose. Run-time goal and efficiency engine translates user input into exact goals for pervasive system in terms of context and timing. Sensor fusion and low-level validation block provides preliminary low-level processing of context data. Retraining database provides bulk of recent low-level data on request, in case there is a need to retrain the model (e.g. due to goal change) or
8
A. Boytsov and A. Zaslavsky
Fig. 3. CALCHAS adaptation engine
provide faster model start. Feedback is implicit – every actuator action will affect future state of the system, which will in turn be tracked by sensors. Adaptation and prediction engine is responsible for inferring adaptation sequences – sequences of actions for the actuator to do. Also it is responsible for providing explicit prediction results. Adaptation engine has the following structure (see Figure 3). All the translation blocks are used to translate between the format of internal model (e.g. states of Markov decision process, its reward functions) and the format of the outer system (e.g. vectors of values of sensor data and exact commands for actuators). The adaptation core is responsible for generating the actions according to the model and learning engine is responsible to adapt the model to new data. All translation blocks are heavily dependent on specific task, but for most of other blocks general purpose solutions can be developed. Supplied with a comprehensive library of reinforcement learning methods, the system can facilitate the development of pervasive computing applications, which will be aware of current and predicted context and which will be capable of smart proactive adaptation to predicted context.
7 Sample Scenario for Proactive Adaptation Methods In order to validate proactive adaptation solutions, we introduce “Moonprobe” model. “Moonprobe” is the result of subsequent development of application scenario presented in [2]. The “Moonprobe” model is an imitation of a vehicle in 2D environment. The aim of the vehicle is to reach destination point safely and not crash or run out of fuel on its way. “Moonprobe” was developed using XJ Technologies AnyLogic modeling tool [25].
Extending Context Spaces Theory by Proactive Adaptation
9
Although not representing pervasive system on its own, the “Moonprobe” model deals with the problems that are common for every pervasive system like sensor uncertainty, sensor failures, acting under uncertainty, fusing heterogeneous sensor data. The moonprobe has a set of specific situations to be aware of. These are sensor failure conditions, progress of going to the destination point and safety conditions. “Moonprobe” uses around 20 context parameters to reason about the situations. Also the moonprobe has some actuators to trigger: engines can help the probe to land safely or to keep the speed. However, working engines require fuel, and the amount of fuel is limited. The actions based on actuators can be considered continuous (direction and power of engine) as well as discrete (like “do nothing”, “keep velocity”, “dampen the fall” etc.). In a very simplified manner the model can be described by the following set of equations (see formula (3)).
N
If probe is in the air: 0 If probe is on the ground: projection of M g on the axis perpendicular to the ground
Fen
(3)
|Fen|
Where t is time, X is probe coordinate, V is probe velocity, M is mass of the probe (known system parameter), N is support reaction (directed perpendicular to the ground), Fl – fuel level remained, g – gravity, FCR – fuel consumption rate (known system parameter), Fen – engine force vector (value and direction set by probe). In some cases probe can leave the ground. When probe lands, it will experience ground collision. Ground collisions are dangerous for the probe. Engines can be used to slow down the landing and avoid the crash.
Fig. 4. “Moonprobe” system architecture
The architecture of the model is following (see Figure 4). System consists of several components. •
“Environment and probe” component. It implements the physical model of the system. Also it monitors what happens to the probe (crash, sensor outage etc.).
10
• • •
A. Boytsov and A. Zaslavsky
“Sensors” component. Sensors measure the data from the probe in certain moments of time and introduce some errors. “Controlling engine” component. It takes sensed data and provides control information. All proactive adaptation logic is embedded there. “Main” component provides overall assessment of experimental results.
The design of exact learning approaches in CALCHAS is in progress. Rule-based decision making engine with quantified actions was introduced into “Moonprobe” model to act as a benchmark for different reinforcement learning solutions. Rules were defined based on common sense. Screen dump of running “Moonprobe” is depicted on Figure 5.
Fig. 5. "Moonprobe" system working
8 Conclusion and Future Work In this work we addressed the task of proactive adaptation to predicted context and proposed reinforcement learning based architecture, which will allow pervasive computing systems to address more complicated use cases. The features of proactive adaptation task in pervasive computing systems were identified. The architecture of proactive adaptation solution – CALCHAS system – was developed to address them. Context spaces theory was used as particular example of integration between context awareness system and proactive adaptation solution. Context spaces-based testbed for proactive adaptation solution was implemented. Our future plans are to implement the entire middleware with a comprehensive library of available reinforcement learning approaches. Also the plan is to extend reinforcement learning solutions methods to suit the core challenges pervasive computing area.
Extending Context Spaces Theory by Proactive Adaptation
11
References 1. Anagnostopoulos, C., Mpougiouris, P., Hadjiefthymiades, S.: Prediction intelligence in context-aware applications. In: Proceedings of the 6th international conference on Mobile data management, Ayia Napa, Cyprus, pp. 137–141. ACM, New York (2005) 2. Boytsov, A., Zaslavsky, A., Synnes, K.: Extending Context Spaces Theory by Predicting Run-Time Context. In: Balandin, S., Moltchanov, D., Koucheryavy, Y. (eds.) ruSMART 2009. LNCS, vol. 5764, pp. 8–21. Springer, Heidelberg (2009) 3. Cook, D.J., Augusto, J.C., Jakkula, V.R.: Ambient intelligence: Technologies, applications, and opportunities. Pervasive and Mobile Computing 5, 277–298 (2009) 4. Cook, D.J., Das, S.K.: How smart are our environments? An updated look at the state of the art. Pervasive and Mobile Computing 3, 53–73 (2007) 5. Das, S.K., Cook, D.J., Bhattacharya, A., Heierman III, E.O., Lin, T.-Y.: The role of prediction algorithms in the MavHome smart home architecture. IEEE Wireless Communications 9(6), 77–84 (2002) 6. Feki, M., Lee, S., Bien, Z., Mokhtari, M.: Context Aware Life Pattern Prediction Using Fuzzy-State Q-Learning. Pervasive Computing for Quality of Life Enhancement, 188–195 (2007) 7. Gellert, A., Vintan, L.: Person Movement Prediction Using Hidden Markov Models. Studies in Informatics and Control 15(1) (2006) 8. Hagan, M.T., Demuth, H.B.: Neural networks for control. In: Proceedings of the 1999 American Control Conference, San Diego, CA, pp. 1642–1656 (1999) 9. van Hasselt, H., Wiering, M.: Reinforcement Learning in Continuous Action Spaces. In: Proceedings of IEEE International Symposium on Approximate Dynamic Programming and Reinforcement Learning (ADPRL’07), Honolulu, HI, USA, pp. 272–279 (2007) 10. Hoey, J., Von Bertoldi, A., Poupart, P., Mihailidis, A.: Assisting persons with dementia during handwashing using a partially observable Markov decision process. In: Proceedings of International Conference on Vision Systems, p. 66 (2007) 11. Hong, J., Suh, E., Kim, J., Kim, S.: Context-aware system for proactive personalized service based on context history. Expert Systems with Applications 36(4), 7448–7457 (2009) 12. Kaelbling, L.P., Littman, M.L., Moore, A.W.: Reinforcement learning: A survey. Journal of Artificial Intelligence Research 4, 237–285 (1996) 13. Mozer, M.C.: Lessons from an adaptive home. In: Smart environments: technologies, protocols, and applications, pp. 273–294. Wiley, Chichester (2004) 14. Padovitz, A., Loke, S.W., Zaslavsky, A.: Towards a theory of context spaces. In: Proceedings of the Second IEEE Annual Conference on Pervasive Computing and Communications Workshops, pp. 38–42 (2004) 15. Padovitz, A., Loke, S.W., Zaslavsky, A.: The ECORA framework: A hybrid architecture for context-oriented pervasive computing. Pervasive and Mobile Computing 4(2), 182–215 (2008) 16. Petzold, J., Bagci, F., Trumler, W., Ungerer, T.: Context Prediction Based on Branch Prediction Methods. Technical Report, Institute of Computer Science, University of Augsburg (2003) 17. Petzold, J., Pietzowski, A., Bagci, F., Trumler, W., Ungerer, T.: Prediction of Indoor Movements Using Bayesian Networks. In: Strang, T., Linnhoff-Popien, C. (eds.) LoCA 2005. LNCS, vol. 3479, pp. 211–222. Springer, Heidelberg (2005) 18. Shafer, G.: A Mathematical Theory of Evidence. Princeton University Press, Princeton (1976)
12
A. Boytsov and A. Zaslavsky
19. Sigg, S., Haseloff, S., David, K.: Minimising the Context Prediction Error. In: Proceedings of IEEE 65th Vehicular Technology Conference VTC 2007, Dublin, Ireland, pp. 272–276 (Spring 2007) 20. Smith, A.J.: Applications of the self-organising map to reinforcement learning. Neural Networks 15, 1107–1124 (2002) 21. Russell, S.J., Norvig, P.: Artificial Intelligence: A Modern Approach, 2nd edn. Prentice Hall, Englewood Cliffs (2003) 22. Lin, T., Wang, C., Lin, P.-C.: A neural-network-based context-aware handoff algorithm for multimedia computing. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMCCAP) archive 4(3), 1–23 (2008) 23. Wolfe, B., James, M.R., Singh, S.: Approximate predictive state representations. In: Proceedings of the 7th international joint conference on Autonomous agents and multiagent systems, vol. 1, pp. 363–370. International Foundation for Autonomous Agents and Multiagent Systems, Estoril (2008) 24. Ziebart, B.D., Maas, A.L., Dey, A.K., Bagnell, J.A.: Navigate like a cabbie: probabilistic reasoning from observed context-aware behavior. In: Proceedings of the 10th international conference on Ubiquitous computing, Seoul, Korea, pp. 322–331. ACM Press, New York (2008) 25. How to Build a Combined Agent Based / System Dynamics Model Logic in AnyLogic (2008), http://www.xjtek.com/file/161
Context-Awareness Enabling New Business Models in Smart Spaces Boris Moltchanov1, Christian Mannweiler2, and Jose Simoes3 1
Telecom Italia S.p.A., via Reiss Romoli 274, 10148-Torino, Italy
[email protected] 2 University of Kaiserslautern, Wireless Communications and Navigation Group 67663 Kaiserslautern, Germany
[email protected] 3 Fraunhofer Institute FOKUS, Kaiserin-Augusta-Alle 31, 10589 Berlin, Germany
[email protected]
Abstract. This paper describes how emerging context-awareness technologies will act as a driving force for introduction of intelligently personalized services in smart spaces. The according context-awareness framework is taken from the results of the EU FP 7 project “C-CAST”. Major novelties include a highly scalable context management architecture as well as a context prediction methodology. The paper considers business models and financial structures as well as service and application examples that might be realized in smart spaces enriched with environmental, personal, social, and network context information. From a commercial perspective on context-aware applications in smart spaces, these examples differ from traditional Telecom and IT services and demonstrate how Telecom and IT companies may strengthen and leverage their business based on context information about customers, services and their environments that is abundantly available in smart spaces. Moreover, this paper includes the description of a field trial with new applications in quasi-smart space environments. Keywords: Business model, smart space, sensor, context, prediction.
1 Introduction The number of mobile user terminals permanently connected to the web over wireless technologies as potential context sources (e.g., embedded sensors, cameras) and the sensors creating large interworking Wireless Sensor Networks (WSN) interconnected with Building Sensor Network and Automotive Sensor Networks is expected to increase significantly creating really ubiquitous systems. All these information sources create heterogeneous and very pervasive environments collectively identifiable as Smart Spaces. Users surrounded by this technology expect more sophisticated and useful services and richer multimedia content. S. Balandin et al. (Eds.): ruSMART/NEW2AN 2010, LNCS 6294, pp. 13–25, 2010. © Springer-Verlag Berlin Heidelberg 2010
14
B. Moltchanov, C. Mannweiler, and J. Simoes
In parallel, the advent of Web 2.0 technologies and their consequent application deployments revealed that user generated content, when tied to a certain geographical location and part of the augmented reality and virtual social communities became a trend in what concerns user most used services. Leveraging smart space technology with a context-aware framework will allow objects, devices, persons and entities to be characterized and interact in real time. Among all players involved in the business value chain of the innovative services, users themselves are the ones demanding for appealing services. To satisfy this demand, a mentality shift needs to occur, as the technological one occurred with the emergence of context-awareness and WSN technologies that enabled new business models and revenue schemes. The context management technology used in this work, has been designed in the European research project C-CAST [1], which aims at combining mobile distribution techniques (both unicast and multicast) and context-awareness in order to optimize multimedia content distribution. This papers shows how it can be used for new business models in smart spaces. The remainder of the paper is organized as follows. Section 2 presents the state-of-the-art in context-awareness for smart spaces. The technical and business aspects of our context management framework are presented in sections 3 and 4, respectively. Then, section 5 describes some context-aware commercial services and trials built within Telecom Italia (TI) infrastructure. Finally, section 6 concludes the paper and introduces future work.
2 State of the Art This section introduces the terminology and provides an overview of related research activities in the areas of context management. 2.1 Definitions and Terminology Context – Ryan et al. [2] referred to context as the user’s location, environment, identity and time. Dey [3] defines context as the user’s emotional state, focus of attention, location and orientation, date and time, as well as objects and people in the user’s environment. Hull et al. [4] describe context as the aspects of the current situation. Brown [5] defines context to be the elements of the user’s environment, which the computer knows about. The notion of situation is closely related to the concept of context. Zimmermann [6] defines it as “the state of a context at a certain point (or region) in space at a certain point (or interval) in time, identified by a name”. Smart Space – a smart space is any real or virtual location equipped with passive and active artifacts. These artifacts can be any kind of sensors and actuators, mobile devices, software agents, autonomous vehicles, management systems, and also human beings. In smart spaces, these artifacts have the processing and communication capabilities to interact with each other in a (mutually) beneficial way. Examples include smart homes, smart factories, smart transportation, smart grid or ambient assisted living environments [7], [8].
Context-Awareness Enabling New Business Models in Smart Spaces
15
2.2 Types of Context for Smart Spaces Context information can be derived from the physical environment or a physical state of an entity (e.g., temperature, humidity, light, noise levels, etc.) and logical (e.g., activity, friend, parent, preferences, etc.). Moreover, context may be of primary origin or derived origin, which effectively is reasoned out of primary context data. There is also meaningful context, a type considered in the literature as linguistic context. For this work, we will basically abstract from the context type, i.e. consider all types. 2.3 Context-Aware Systems for Smart Spaces The history of context-aware systems started when Want et al. [9] introduced the Active Badge Location System, considered to be one of the first context-aware applications. Ailisto et al., [10] and Dey and Abowd [11] proposed a layered conceptual architecture that augments layers for detecting and using context by adding interpreting and reasoning functionalities. In middleware-based systems, the Service-Oriented ContextAware Middleware (SOCAM) project [12] introduced an architecture for building and rapid prototyping context-aware mobile services. One further extensible centralized middleware approach designed for context-aware mobile applications is a project called Context-Awareness Sub-Structure (CASS) presented in [13]. Chen [14] presents a Context Broker Architecture (CoBrA), which is an agent-based architecture, for supporting context-aware systems in smart spaces (e.g., intelligent meeting rooms, smart homes, and smart vehicles). Other well-known examples for broker-based approaches are the SOUPA, and GAIA [15]. Many EU projects have explored contextawareness aspects, for example, SPICE [16] and MUSIC [17]. One of the main reasons of little context-awareness usage, despite the mentioned efforts, was lack of end user devices that could host the context-aware applications for users. With the wide scale availability of such devices, including the WSN sensors, it is now possible to harness the possibilities of context-awareness. There are many types of the services where context-aware smart-spaces could play a significant role, namely advertisement, e-recommendations, e-tourism, gaming, etc, opening the path for personalized content creation and distribution, enabling new business models of contextualized applications and service delivery over smart spaces.
3 Technical Aspects of Context-Aware Smart Spaces Before focusing on the business aspects, it is important to understand the technical requirements of smart spaces and how context is managed and represented, using our framework, throughout the entire value chain. 3.1 Context Management Requirements of Smart Spaces The vision of smart spaces includes intelligent environments and context-aware services that react in a sensitive and adaptive way to the presence of humans and/or objects. Telecom related applications as well as energy saving, gaming, chatting, info and payment services are important constituent parts of such smart spaces. The design of such context management system should therefore consider objectives and
16
B. Moltchanov, C. Mannweiler, and J. Simoes
requirements from various domains, like data management, communication, user ID management, system reliability, privacy, security, etc. The novelty here lies in the utilisation of information from smart spaces for supporting conventional, as well as community created context applications, in a way that has not been attempted before. This section gives an overview of the most common high-level objectives and requirements [7],[18]. 1) Enhance and leverage context data through • filtering mechanisms as well as evaluating and improving its (multidimensional) quality; • aggregation, fusion, inference, and prediction. 2) Efficiently exchange data and provide context services within and between different smart spaces by • establishing rules for context discovery, storage, and distribution; • designing common interfaces for communication within and across smart spaces; • specifying communication mechanisms between smart space artifacts • establishing management procedures for smart space entities owning context data. 3) Ensure end-to-end data security by • monitoring and assuring the non-violation of privacy, security, and trust within and across smart spaces; • guaranteeing the enforcement of rules and policies. These overall objectives yield a set of design requirements for the desired smart space system functionalities. The system needs to be capable of obtaining and representing context data from a wide range of sources, as well as processing and storing it according to rules and policies defined by the artifacts (users, objects, etc.) present in a smart space. Moreover, mechanisms for computation of semantically more abstract context, obtained from simple types of contexts (aggregation, fusion, reasoning, prediction, and inference) have to be made available. This includes the usage of widely used, standardized communication protocols and interfaces. A smart space has to dispose of a (preferably distributed, as described in [19]) context service directory that stores different types of information. Namely, what context services are available and at which location within a smart space. This will allow autonomous reconfigurations depending on the load and the capabilities of the smart space artifacts. Hence, a protocol for handling the arrival of new context sources joining a smart space needs to be available. Finally, mechanisms for identification, authentication and authorization guaranteeing end-to-end security and privacy of user context within and across smart spaces have to be included. 3.2 Context Management Architecture Context, “embedded” into our world and around our lives, can be exploited to enrich communication systems. The architecture proposed in this section is based on a topdown, service scenario-to-architecture component analysis. It is further scrutinized by a bottom-up synthesis of available underlying technologies for service creation and
Context-Awareness Enabling New Business Models in Smart Spaces
17
deployment to potential customers within a mobile environment, which possesses already context information regarding its customers. The proposed architecture is based on the producer consumer role paradigm. The system is built upon three basic functional entities: Context Consumer (CxC), Context Broker (CxB) and Context Provider (CxP), where a CxP is a source of the context information, a CxB is an arbitral node handling the context data as a reference point for the overall context in a smart space and CxCs are consumers of context information. The decoupling of provisioning and consumption is important as it impacts on the overall scalability of the system. Further details about this implementation can be found in [20] and [21]. 3.3 Context Modeling and Representation Context information needs to be represented and modeled for being machine interpretable and exchangeable using well-defined interfaces. The goals are to support easy manipulation on many devices, including resource constrained mobile devices, achieve low overhead in keeping the model up-to-date, easy extension, efficient search and query access, having cheap and simple mechanisms for adding new types of information and scalability. In the literature, different approaches for representing contextual knowledge and semantic information can be found [22]. To improve the limitations identified, our work proposes an appropriate option for representation and exchange of context information in smart space environments is ContextML [23], which is based on the XML format. ContextML can handle several layers of abstraction in a context-aware system, as illustrated in Fig. 1, and any context-aware middleware or architecture must therefore be capable of building representations and models of these layered abstractions.
Fig. 1. Layered Context Model
18
B. Moltchanov, C. Mannweiler, and J. Simoes
The main context management features are context acquisition, context aggregation & fusion, context dissemination, discovery and lookup. Adapting such a structure to existing infrastructures could allow a context handling interface to be embedded into a service exposure layer as a SOA-like web service, allowing secure and controlled access to context information. Nevertheless, privacy issues need to be addressed efficiently in order for such services to become a reality. To accomplish such vision, the work done in [24] addresses some of the most relevant issues.
4 The Business Aspects of Context-Awareness in Smart Spaces This chapter shows how the outcome of the proposed context-aware system can enable new business models in smart spaces. 4.1 Types of Context Before getting into the possible business models, it is important to understand what kind of context can be made available. In this sense, we will separate context into three different categories: real-time, historical and reasoned context. Nevertheless, only real-time information is considered as context in the real meaning of this term. Table 1 provides some examples of these context types. Table 1. Types of Context Available within a Smart Space Real-Time Location Orientation
Historical Previous locations Previous orientation
Temperature Humidity Brightness Noise level
Previous temperatures Previous humidity Previous brightness Previous noise levels
Proximity
Previous proximity
Call/VAS Log Phone profile settings HW version, FW release and SW packages installed Battery level
Previous Call/VAS Logs Previous profile settings Previous devices settings
…
Reasoned Movement Movement, destination Weather Weather Weather, Time User activity, user location Friends, colleagues, customers Relationship Proximity Service type, capability, content format Service usage
4.1.1 Real-Time Context This type of context represents the data available in the cache of the CxB, meaning that the context retrieved is still valid (has not expired). This is particularly useful to trigger events based on real time actions. It can be requested per user, per context or per object (e.g., retrieve users that viewed a specific content in a determined location).
Context-Awareness Enabling New Business Models in Smart Spaces
19
4.1.2 Context Prediction for Generating Higher-Level Knowledge Reasoning refers to information that can be inferred from analyzing data and combining different context information. This is usually more complex to obtain and should have a confidence level associated with it. One example for reasoning includes context prediction [25] Context prediction deals with the issue of generating awareness for future context. This implicitly includes the proactive control and adaptation of a system such as a smart space according to (expected) future user behavior. Generally, an entity of a context management system (e.g. context consumer, CxC) can query the future value of the specified context. An exhaustive prediction contains the following parameters: • • •
Name of context scope to be predicted Name of entity the scope belongs to Future point in time the context should be computed for
On the one hand, current context information of the specified entity will be queried in order to get the starting state for the prediction algorithm. On the other hand, historical data of the specified scope and as well as other (related) scopes of that entity will then be retrieved from other entities, e.g. from other CxPs or from context history databases. For example, in case of a location prediction, additional interesting context data could be speed, acceleration and orientation. For the actual prediction process, a set of algorithms for analysis of historical data are available. Suitable candidates include tendency predictors, first or higher order Markov models, hidden Markov models, or Bayesian networks. Depending on the prediction request, the most appropriate algorithm is selected. Again, relevant criteria include time horizon of prediction, type of context (i.e. expected data characteristics), performance of previous predictions for specific applications (e.g. service execution in smart space environments) as well as real time requirements. Another important aspect is the degree to which the algorithm needs supervision, i.e. human interaction. In principal, for smart space environments, prediction should be performed completely unsupervised. As such, the combination or parallel execution of different algorithms shall provide predictions with higher confidence and reliability. 4.2 Relevant Applications and Business Models Context-aware smart spaces may be used by many applications and services. The most interesting application classes that can benefit from context-awareness in smart paces are the following: • • •
sustainability – including assisted living, smart cities, traffic management, intelligent energy management, tele-working, smart remote (business) process control machine to machine communications – including smart factories, logistics, and vehicular communication e-health – including smart hospitals, personalized fitness programs, elderly support from remote, advanced national health and wellness program
20
B. Moltchanov, C. Mannweiler, and J. Simoes
• •
emergency management, recovery and rescue teams support – fast recovery from catastrophes, remote monitoring, support for rescue forces and efficient material allocation Gaming – including distributed and remote entertainment, education and tourism systems.
Any type of company from both the Telecom and IT industries may leverage on the potential of the presented context-aware applications in smart spaces. Some of the most promising business models applicable in this area will be described in the following sections. 4.2.1 The In-House Model One of the best ways to avoid churn is to offer users what they want and need. Using context intelligence to improve services customization and personalization became a trend and operators are starting to use context to improve or create, existing or new services, respectively. A great example of such deployment is provided by some Telcos where a service merges a simple address book into an immersive social experience. Similar ideas can be employed for enhancing user experience in smart spaces. Although very often third party applications (e.g., Facebook, Twitter) are used, there is not any sort of revenue sharing. With this approach, they improve not only customer satisfaction but also revenue generation resulting from the associated data consumption from the operators' network. Furthermore, as these services are usually associated with fidelity plans, this helps operators to rethink long-term strategies. 4.2.2 The Revenue Sharing Model This particular method can exist in two variants. The first occurs when the Telco improves its offer by using context information from external sources (e.g., social networks, traffic or news provider). This is similar to the outside-in methodology or to a white labeling product/service where the operator leverages its customer base to a new offering and earning a percentage out of it. The second approach resembles the inside-out approach, where third party applications are built not only based on operator infrastructure (using APIs) but also its context information. In this situation, although the Telecoms do not own the end user of these applications; they still earn a part of the revenue generated from this service. 4.2.3 The Advertising Model Advertising can be leveraged by a smart space infrastructure in a myriad of ways. It can be done in a non-intrusive way when the customer requests a specific service (e.g. in a shopping mall), which is then sponsored by advertising. In any case, advertisement can be targeted according to a user’s preferences and context. Therefore, advertisers are no longer limited to little customer segmentation options. In a way, this model is a mix of revenue sharing and context exposure selling as they earn money by using both their infrastructure (e.g., SMS or MMS) and their context information (e.g., location). The main challenge of this model is to provide advertisements in an unobtrusive way, simultaneously considering users’ interests for a specific moment in time. These targeted events will consequently improve the probability of service sales since the offer will match his or her needs, interests and context.
Context-Awareness Enabling New Business Models in Smart Spaces
21
4.2.4 Context Exposure Selling Model Operators can extend the usage of their customers’ context to outside their domain in order to improve their long-tail experience and creating revenues from businesses build by third parties. In this model there is no strict correlation between the service offered by third party providers and the operator. In this sense, the Telco exposes context through well-defined APIs. Then, depending on what is necessary the context can be accessed by: • •
•
Request – can be done per user, per group of users, or per context. In this case, the billing is performed on a per request basis or per response basis (both models could work). Subscription – for some services, in case an application needs to keep an updated status of the context (e.g., to track movement you need to update location regularly). For this case, a fee can be applied for the subscription to certain context, to certain users, group of users or a combination of all as the updates can occur without any specific pattern and therefore is difficult to estimate a price per notification. Bundling – bundling context decreases the offer price when compared to single purchases while significantly improving or completing the service offer.
4.2.5 Self-service Model In today’s interactive networks, users are no longer just consumers but also producers. Although for now this concept is mainly associated with content, it is predictable that in the future the same tendency is reflected on service composition and execution, where people can build their own applications based on intuitive graphical interfaces that represent logical interfaces of a myriad of context enablers. In this scenario the operator would also need a service orchestrator based on context. This would pave the way to a whole new way of service creation and consequently revenue generation. In a first instance, it would allow customers to engage with the service and network providers and would accelerate new service development paying back the network operator’s investments in customer tailored services. In a way, this can be seen as the “Application Stores” business model, where customers could then publish their services, using both the operators’ infrastructure and context information. For this to happen, a revenue sharing model together with context exposure billing model should be defined to charge an intermediate or an end user. 4.3 Limitations Despite the simplicity and flexibility in which these business models can be implemented, the processing and management of personal data obviously raises security and privacy issues that need to be addressed. In this sense, users need to stay in control of the whole process, enabling them to specify when, what, why, who, where and how the data is or can be accessed. In other words, user context should be disclosed according to contextual privacy policies and settings. Even the most primitive model should allow users to opt-in and opt-out from such profiling. Currently, on an average, only 3% of an operator’s costumer base opts-in for such services. Therefore, it is necessary to improve these business models in order to attract more end users. Otherwise, largescale context exposure and distribution will never be available.
22
B. Moltchanov, C. Mannweiler, and J. Simoes
In addition, it is necessary to achieve a consensus and agreement among different stakeholders so that information can be provided in a standardized way. This will allow for compatibility within and across different smart spaces and it will facilitate the reusability of applications in different smart spaces. Examples include developerindependent software APIs, as well as manufacturer-independent “hardware APIs”. Moreover, context shall be efficiently acquired, encoded and transformed into a determined high-level context, distributed and provided to requesting entities for consumption, while simultaneously maintaining its real-time properties. This is a very complex and resource-consuming task requiring appropriate context sources, very efficient supporting context management systems with embedded intelligence, employment of interoperable and lightweight standards, fast communication channels and secured interfaces for context retrieval and distribution. The main difficulty in the creation of intelligent and comprehensive smart spaces is “filling” the spaces with myriads of the sensors, which require huge investments. Therefore the users’ mobile terminals should be considered as sensor devices in smart spaces.
5 Trials of New Applications and Services Quasi-smart Space Environments TI is already providing some context-aware services tied to a certain location or a territory to its customers especially for mobile users in the eTourism field, where the B2B2C model is implemented with the following elements: • • • • •
the development of a model of integrated and aggregated offer on a local basis, or "under the patronage" of public administrations responsibility; the involvement of the PAs (Public Administrations) for the sponsorship of the model and the involvement of the proper actors active on the business value chain; the collection and delivery of content (e.g., information for the enhancement of heritage artists); a strong leverage on mobile devices for the end customer and on TI’s infrastructure based on assets (connectivity, data centers, Telco Capabilities) solutions; the hypothesis of conditional business to the establishment of a collaborative model based on a local public-private integration.
The different roles and their interactions in the TI’s eTourism scenario are shown in Fig. 2 where the CA&SA Provider is responsible for integrating context information and self-adapts it into its services. To some actors in the network this context information has an economic value. Therefore, it is assumed that actors do not give away data or services without an adequate compensation. Consequently, the analysis here has been done in terms of value exchanges between the different actors. By playing the role of context enabler, TI limited its involvement to providing network data that could enable context reasoning. In any case, it could be an important distributor of context data in the market of CA&SA services. By playing the role of context
Context-Awareness Enabling New Business Models in Smart Spaces
23
provider, TI would increase the involvement in the organization of these services and would trade context data from different sources. There are no sensor networks involved in the described services as abovementioned, however, mobile terminals and location-aware services can be considered as sensors. Therefore, the described environment is a first step towards a field trial smart-space facilitated by a Telecom operator. In the presented scenario, it plays the role of network operator, service provider and seller. This combination of roles puts TI in a privileged position to collect user and network context data.
CA&SA Service Provider
Fig. 2. View of different roles and their interactions in a general eTourism service scenario
The business model used in the aforementioned services was based on the bundling of a customized platform/service solution to third parties interested in a promotion of certain initiatives and events at large national, regional and local scales. These entities, (e.g., City Hall, Government, Event sponsors) interested to involve and to attract, for any reason (e.g., advertisement, social visibility, tourism), as much people as possible, have now a way to target this offer within a specific and contextualized scope. A prepared ad-hoc system usually involves context management platforms, preparation and customization of the website and its continent, where users can register and upload their content. Then different interactions are offered, voting, rating, the best content, the most popular place, as well as interactive game scenarios where every participant is invited and potentially involved. All these web portals [26] leverage User Generated Content (UGC) upload and sharing capabilities, mainly based on the user location, including the proximity to objects and other people, presence status, willingness proposition, preferences, and social contexts.
24
B. Moltchanov, C. Mannweiler, and J. Simoes
6 Conclusions and Future Work In this work we have presented the business potential of context-aware systems, demonstrated by a technical solution implemented at TI premises for trial concepts and scenarios. Moreover, we have highlighted context management requirements for smart spaces. As seen, most use cases involve more than two entities (a smart space consists of many active entities), therefore disrupting the traditional business models. Despite the presented limitations, we have shown that small variations to well-known models can generate positive financial flows. Furthermore, it became clear that the application of context-awareness as wide spread technology, oriented to the social usage and treating personal data, shall be further analysed for potential issues regarding privacy, business models and law regulations before it can be widely adopted and deployed in smart spaces. Nevertheless, when context-awareness and context-data are exposed as a business enabler to third parties, in a guaranteed and protected way, all entities within the value chain will be able to deliver attractive solutions. Still lacking is the definition of the right pricing structure with an appropriate revenue sharing scheme. Altogether, this work will leverage context exposure over Web2.0 interfaces and other new business propositions.
Acknowledgments This work is supported by the European ICT project “C-CAST” - Context Casting (Contract-No. ICT-216462) as well as by the project “G-Lab” (funding reference 01 BK 0808) funded by the German Ministry for Education and Research.
References 1. Context Casting (C-CAST), FP7-ICT-2007-1, http://www.ict-ccast.eu/ 2. Ryan, N., Pascoe, J., Morse, D.: Enhanced reality fieldwork: the context-aware archaeological assistant. In: Proceedings of 25th Anniversary Computer Applications in Archaeology (1997) 3. Dey, A.K., Abowd, G.D.: Towards a better understanding of context and contextawareness. In: Proceedings of Work on the What, Who, Where, When and How of Context-Awareness. ACM Press, New York (2000) 4. Hull, R., Neaves, P., Bedford-Roberts, J.: Towards situated computing. In: Proceedings of the First International Symposium on Wearable Computers (ISWC ’97), p. 146 (1997) 5. Brown, P.J.: The stick-e document: a framework for creating context-aware applications. In: Proceedings of Electronic Publishing, Palo Alto, pp. 259–272 (1996) 6. Zimmermann, A.: Context Management and Personalisation, p. 260 (2007) 7. Oh, Y., et al.: A Context Management Architecture for Large-Scale Smart Environments. IEEE Communications Magazine (March 2010) 8. Casdell-Oliver, R., Liu, W.: Representation and recognition of Situations in Sensor Networks. IEEE Communications Magazine (March 2010) 9. Want, R., Hopper, A., Falcao, V., Gibbons, J.: The active badge location system. ACM Transactions on Information Systems 10(1), 91–102 (1992)
Context-Awareness Enabling New Business Models in Smart Spaces
25
10. Ailisto, H., Alahuhta, P., Haataja, V., Kylloenen, V., Lindholm, M.: Structuring context aware applications: Five-layer model & example case. In: Proceedings of Workshop on Concepts & Models for Ubiquitous Computing, Sweden (2002) 11. Dey, A.K., Abowd, G.D.: A conceptual framework and a toolkit for supporting rapid prototyping of context-aware applications. HCI Journal 16(2-4), 7–166 (2001) 12. Gu, T., Pung, H.K., Zhang, D.Q.: A middleware for building context-aware mobile services. In: Proceedings of IEEE Vehicular Technology Conference (VTC), Milan, Italy (2004) 13. Fahy, P., Clarke, S.: CASS – a middleware for mobile context-aware applications. In: Workshop on Context Awareness, MobiSys 2004 (2004) 14. Chen, H., Finin, T., Joshi, A.: An Intelligent Broker for Context-Aware Systems. In: Article, Adjunct Proceedings of Ubicomp 2003 (2003) 15. Roman, M., et al.: GAIA: A Middleware Infrastructure to Enable Active Spaces. IEEE Pervasive Computing 1(4), 74–83 (2002) 16. Service Platform for Innovative Communication Environment (SPICE), FP6 EU Project, http://www.ist-spice.org 17. Salis, N., Licciardi, C., Hallsteinsen, S., Geihs, K.: Telecom Italia’s MUSIC Exploitation: Mobile eTourism, D13.12, FP6 EU Project MUSIC (December 2009) 18. Mannweiler, C., et al.: A Robust Management Platform for Multi-Sensor Location Data Interpretation. In: Proceedings of the 19th Future Network and Mobile (2010) (to be published in June 2010) 19. Mannweiler, C., Amann, B., Schneider, J., Klein, A., Schotten, H.D.: A Distributed Context Service Architecture for Heterogeneous Radio Environments. To be published in ITG-Fachbericht der 15. ITG Fachtagung Mobilkommunikation, Osnabrück, Germany (May 2010) 20. Moltchanov, B., Knappmeyer, M., Licciardi, C.A., Baker, N.: Context-Aware Content Sharing and Casting. In: Proceedings of ICIN 2008, Bordeaux, France (October 2008) 21. Liaquat, S., Knappmeyer, M., Baker, N., Moltchanov, B.: A Federated Broker Architecture for Large Scale Context Dissemination. To be published in Proceedings of The 2nd International Symposium on Advanced Topics on Scalable Computing, in conjunction with The 10th IEEE International Conference on Scalable Computing and Communications (ScalCom-2010), Bradford, UK, June 29-July1 (2010) (to be published in July 2010) 22. Baldauf, M., Dustdar, S., Rosenberg, F.: A survey on context-aware systems. International Journal of Ad Hoc and Ubiquitous Computing 2(4), 263–277 (2007) 23. Knappmeyer, M., Liaquat, S., Frà, C., Baker, N., Moltchanov, B.: ContextML A LightWeight Context Representation and Context Management Schema. In: Proceedings of IEEE International Symposium on Wireless Pervasive Computing, Modena, Italy, May 5-7 (2010) 24. Simoes, J., Weik, P., Magedanz, T.: The Human side of the Future Internet. In: Towards the Future Internet, pp. 183–192. IOS Press, Amsterdam (2010) doi:10.3233/978-1-60750539-6-183 25. Mayrhofer, R.M.: An Architecture for Context Prediction. Dissertation, Johannes Kepler University Linz (2004) 26. http://teamlife.telecomitalia.it/lucca, http://teamlife.telecomitalia.it/lecce, http://teamlife.telecomitalia.it/venezia, http://beta.teamlife.it/bs
A Middleware Architecture for Safety Critical Ambient Intelligence Applications Antonio Coronato and Giuseppe De Pietro Institute of High Performance Computing and Networking ICAR-CNR Via P. Castellino 111, 80131, Naples - Italy
[email protected],
[email protected] Abstract. Ambient Intelligence (AmI) technologies are more and more often adopted in scenarios, like healthcare, environments monitoring and control, ambient assisted living, etc, which are critical for human safety and assets preservation. This requires that such a kind of applications be designed taking into account stringent dependability requirements. This paper presents a service-oriented middleware architecture for safety critical Ambient Intelligence applications, which provides services that enable the designer to develop runtime verification mechanisms; that is, it is possible to continuously monitor the verification of correctness properties defined at design time.
1
Introduction
Ambient Intelligence (AmI) is the emerging computing paradigm, which derives from Pervasive Computing, that refers to electronic environments that are sensitive and responsive to the presence of people [4]. In the past few years, many pervasive computing environments have been developed and deployed to realize smart rooms, active spaces and context-aware applications. All these environments have been devised with the aim of enabling users to derive benefit from technology without being distracted by it, in order to augment productivity, collaboration, and access to technology. A recent trend is to use these technologies in new domains like hospitals, old people’s homes, emergency scenarios, etc. But, such kinds of environment present a new criticality especially concerning the issues of expressing requirements, verifying and validating them, ensuring functional correctness, etc. These applications must be considered safety critical and, consequently, should be designed taking into account such a criticality. As an example, let us consider the application described in [14]; it mainly consists in a tele-monitoring system that processes the stream of physiological data coming from an Electrocardiogram (ECG) sensor monitor of a cardiopathic patient at home. Data must be processed in real-time; i.e. by granting ”results” within a certain temporal period. Moreover, in the case of critical values, alerts must be raised and processed by a temporal threshold. As also emphasized by the authors, functional incorrectness or run time failures may result in catastrophic consequences for the patients monitored. As a consequence, such considerations require that this class S. Balandin et al. (Eds.): ruSMART/NEW2AN 2010, LNCS 6294, pp. 26–37, 2010. c Springer-Verlag Berlin Heidelberg 2010
Safety Critical Ambient Intelligence Applications
27
of systems be designed and developed taking into account intrinsic and stringent dependability requirements. In this paper, we present a service oriented middleware architecture that provides a set of basic mechanisms for rapidly prototyping safety critical Ambient Intelligence applications. In addition to other middleware infrastructure, the one proposed in this paper presents a specific section (set of services) that enables the specification of correctness properties and their check while the system is running. So called runtime verification techniques are adopted both to tune the system in its early running and to monitor its correct behavior once deployed definitively. The rest of this paper is organized as follows. Section 2 presents our motivation and related work. Section 3 describes the middleware architecture. Section 4 focuses on runtime verification mechanisms for correctness properties. Section 5 presents a case study. Finally, Section 6 concludes the paper.
2
Motivation and Related Work
A remarkable example of safety critical AmI applications is the class of Ambient Assisted Living (AAL), which regards the realization of Smart Environments to monitor elderly and ill people. A middleware infrastructure for AAL applications is the PERSONA project (Perceptive Spaces prOmoting iNdepentent Aging) [15], which aims at developing a scalable open standard technological platform for building a broad range of Ambient Assisted Living Services. This is an ongoing project funded by the European Commission. The main technical challenge is the design of a self-organizing middleware infrastructure allowing the extensibility of component/device ensembles. The communication patterns of the infrastructure are based on distributed coordination strategies for service discovery and utilization. The current prototype of the PERSONA middleware, implemented on the OSGi platform [12], uses connectors based on UPnP, Bluetooth, and R-OSGi [18]. However, the development of such a kind of applications presents new criticalities depending on their intrinsic characteristics, as for an example the dependency of the system behavior from the user and resources location and movements. Such criticalities can partly be faced by means of formal methods [17]. On the other hand, the use of formal methods in the field of pervasive computing is a relatively new trend and most of existing applications rely on Ambient Calculus (AC ) [3] that provides interesting primitives for the designer of pervasive systems. As an example, Campbell and Ranganathan [2] have argued the need for the formal specification of pervasive computing environments and have proved the correctness of some elements of their system GAIA by means of AC. In previous papers, we have defined a methodology that enables the designer to specify formally an Ambient Intelligence or a Pervasive Computing application [5] [6]. However, formal methods are typically not confined to the specification phase. Indeed, they can be adopted to prove the correctness of the specification or the
28
A. Coronato and G. De Pietro
implementation. In particular, Model Checking, that along with Theorem Proving and Static Analysis is a technique aiming at static program verification [13], which is concerned with checking if all possible traces derived from a program (or abstract model) satisfy a property of interest. In contrast to these techniques, this paper focuses on runtime verification which is concerned with checking traces of events generated from the program run against properties described in the proposed logic. When a property is violated or validated, the program can take actions to deal with it. The technique can be used both for testing before deployment of the software, and for monitoring after deployment. In the first case, one must come up with test cases [1] that might uncover a bug. In this setting, runtime verification is considered as an auxiliary tool to automate the creation of oracles that detect errors. In the second case, runtime verification is used to monitor a program during its execution so to take actions in response to the violation of properties. With this perspective in mind, a runtime verification tool may be used to define how the program reacts to bugs, possibly steering it to the correct behavior [11]. It is useful to note that runtime verification techniques can be adopted also to tune the system. This is particularly true for pervasive applications whose behavior can strongly depend on operating conditions. As an example, an RFID reader can fail to detect a tagged user depending on i) the orientation of the antenna, ii) the power of the antenna itself, iii) the position of the tag on body, iv) the existence of an object between the tag and the reader at the time of reading, etc. All these conditions may be very difficult to verify with static techniques. In contrast, runtime verification of the prototype system in the real domain of application can make possible the collecting of traces that can be used to tune the system. It is important to note that runtime verification techniques typically concern real-time properties [16], but in this paper we extend the verification to properties related to physical locations.
3
Middleware Architecture
In this section, we describe the overall middleware architecture. Figure 1 shows the key services, layers and modules of the proposed middleware. The bottom-left section regards Human Computer Interaction. It includes mechanisms for handling natural and advanced interactions in a smart environment. Indeed, one of the grand challenges for future smart environment applications is making interaction with technology-rich environments intuitive, natural, unobtrusive, intelligent, and distraction free. Currently, two services have been developed, a TextToSpeech Service, which is in charge of synthesizing vocal indications, and a Messaging Service that is in charge of sending textual messages to the user’s mobile device [7]. The bottom-right section is about Sensing and Controlling. This section includes hardware sensors and actuators. These two sections are structured in different layers. The lowest layer includes hardware devices. The heterogeneity of the components at this layer is handled by means of software components,
Safety Critical Ambient Intelligence Applications
29
Fig. 1. Middleware architecture
namely facilities, which hide the hardware components to upper layers. For example, the RFID Facility provides support to handle an RFID antenna; thus, each RFID antenna has its own facility that will handle interactions between the hardware component and the rest of the software system. Upon this layer, a Facility Service handles the set of homogeneous facilities that stand below; i.e. a facility service catches information coming from the set of facilities that are related to the same kind of hardware device, RFID antenna for the case of RFID. These services are in charge of capturing data coming from hardware components and handling them as an integrated set of information for the upper levels. Thus, the RFID Facility Service (one of the facility services not shown in the figure) collects information coming from any RFID antenna deployed within the environment. The middle section offers communication functionalities. In particular, in addition to RPC, an asynchronous communication mechanism is exposed by the Event Service. This service is in charge of dispatching events between software components. It supports the publish-subscribe mechanism. Further details can be found in [8]. The topmost section concerns the context. The services defined in this layer are: – Topology Service - This service provides a unique and uniform representation for the topology of the environment. In particular, locations are classified as semantic locations (buildings, floors, rooms, an area in front of a wall monitor, etc) and physical locations (the area sensed by an RFID antenna, the area covered by a WiFi access point, etc.) [9]. – Tracking Service - This service offers a mechanism to track all movements of a mobile resource or user.
30
A. Coronato and G. De Pietro
– Location Service - This component is in charge of handling physical location information for mobile resources and users. In particular, it locates mobile users by means of the RFID technology and WiFi enabled mobile devices by means of classic location techniques [9]. – User Service- This service provides basic authentication mechanisms for users and handles the list of all users active in the environment. – Resource Service - This component extends standard mechanisms for registering and monitoring resources, like laptops, PDAs and sensors. Finally, the correctness section includes mechanisms for runtime verification. The correctness properties are defined with respect to the state of an ambient, which is a physically bounded entity of the domain. An ambient may be i) a location of the environment (e.g. a room in which only authorized people can enter); ii) a resource (e.g. a constrained mobile device that can execute only lightweight software agents); or iii) a person (e.g. a user that can invoke some services only after having been identified); etc. The correctness properties can also be defined with respect to temporal properties (i.e. real-time constraints). The set of services defined for runtime verification consists of: – RunTimeStateHolder Service - This is the component that holds and exposes the state of an ambient. – RunTimeChecker Service - This is the component that checks, on behalf of an ambient, the properties of hosted, entering and leaving ambients. – Timer Service - This is the component that holds and verifies temporal constraints. – Monitoring Service- This service provides basic mechanisms for monitoring the entire system.
4
Runtime Verification
In this section, the services for runtime verification are described. Runtime verification is divided into two parts: instrumentation and event observation. A monitor receives events from the executing program, emitted by event generators inserted during instrumentation, and dispatches them to a collection of algorithms, each of which performs a specialized trace analysis. In the following, both the structural and behavioral models are presented. The structural model is reported in figure 2 and consists of a set of software components that instrument the running system in order to be monitored by an external monitor, the Monitoring Service. It is important to note that in addition to real-time properties we are interested in correctness properties that are related to locations. 4.1
Structural Model
The detailed architecture is: RunTimeStateHolder This component supports the following structure:
Safety Critical Ambient Intelligence Applications
31
– S - This is the state of the current ambient; – ES - This is the list of all admissible states for an entering ambient; i.e. another ambient must verify one of these states in order to enter the current ambient; – LS - This is the list of all admissible states for a leaving ambient; – HS - This is the list of all admissible states for a hosted ambient; – getState - This operation returns the current state; – setState - This operation updates the current state to a new value; – getHostedStates - This operation returns the list of admissible states for a hosted ambient; – setHostedStates - This operation sets the list of admissible states for a hosted ambient; – getEnteringStates - This operation returns the list of admissible states for an entering ambient; – setEnteringStates - This operation sets the list of admissible states for an entering ambient; – getLeavingStates - This operation returns the list of admissible states for a leaving ambient; – setLeavingStates - This operation sets the list of admissible states for a leaving ambient; RunTimeChecker This component supports the following operations: – canEnter - This is the operation that verifies whether an ambient (say for example b) can enter or not the current ambient (a); i.e., when the ambient b tries to enter into a, this method verifies whether the current status of b is admitted by a, or not; – canLeave - This is the operation that verifies whether an ambient (b) can leave or not the current ambient (a); – canHost - This is the operation that verifies whether an ambient can be hosted or not by the current ambient. Timer This component supports the following structure: – TimeToExpire - This is the time the timer has to wait before expiring; i.e., this is the temporal range to complete a real-time operation; – TimeToAlert - This is the time the timer waits before raising an alert. It holds the property (TimeToAlert < TimeToExpire). This parameter, which is optional, is used to send an alert a certain time before the timer expires; – setTimer - This is the operation that sets the timer’s parameters; – startTimer - This is the operation that starts the timer; – ariseAlert - This is the operation that raises an alert before the timer expires; – expire - This is the operation that sends a message when the timer expires.
32
A. Coronato and G. De Pietro
Fig. 2. Pattern Structural View
4.2
Behavioral Models
The case of an action of type enter in is detailed. In particular, the ambient b wants to enter the ambient a. For this case, two alternative interaction models are specified. In figure 3(a), the ambient b itself is in charge of verifying that the condition Enabled for entering the ambient a is satisfied. Eventually, the ambient b is considered trustworthy. In contrast, in figure 3(b), the ambient a is in charge of verifying incoming ambients, but, it must be notified of this action in some way.
(a) Incoming centered responsibility
(b) Hosting centered responsibility
(c) Change of the State of a Hosted Ambient Fig. 3. Capability in - Pattern Behavioral Views
Finally, in figure 3(c) the case of change of the state for a hosted ambient is reported. In this case, the hosted ambient (b in the figure) must notify its new state to the hosting one (a). After that, the hosting ambient verifies that the new state is compliant with the states admissible for hosted ambients.
Safety Critical Ambient Intelligence Applications
5
33
Case Study
In this section, we present a case study. The system being designed is a Pervasive Healthcare Application for the Department of Nuclear Medicine of the University Hospital ”Secondo Policlinico” of Naples. 5.1
Informal Description
Within the Department, patients are injected with a small amount of a radioactive substance in order to undergo specific examinations (e.g. Blood Volume Study, Bone Scan, Brain Scan, etc.). Once injected, patients emit radiation (gamma rays) and have to stay in a specific room to wait for the examination to be performed. The time patients have to wait depends on the kind of examination that has to be performed and the time the radioactive substance takes to propagate -within the body- and to decay to the right level. In fact, examinations can be executed only if the radiation level is in a certain range. After the examination, patients return to the waiting room until the level of radiation becomes less than a specific threshold and, so, harmless. The department consists of the following rooms: 1) Acceptance Room (AcceptanceRoom) -This is the room where patients are accepted within the department and wait for injection; 2) Injection Room (InjectionRoom) -This is the room where patients receive the injection; 3) Hot Waiting Room (HotWaitingRoom) -This is the room where patients wait for the examination after having been injected and until the radiation level reaches the correct range; and 4) Diagnostic Room (DiagnosticRoom) -This is the room where examinations are performed. Currently, patients are accompanied by nurses when moving within the department. The goal is to have an automatic system that would lead them. The system will adopt RFID readers and tags to identify and locate patients within the department. Mobile devices will also be used to send messages to patients in order to invite them to move (e.g. ”Move to the injection room”, ”exit the department”, ”you are not allowed to enter this room”, etc.). In this way, nurses will no longer be necessary to guide patients within the department. This will have two benefits: lower costs and fewer people exposed to radiation. 5.2
A Peace of the Formal Specification
First, the following classes of ambients are concerned: {Department, Room, Patient, RFIDReader, RFIDTag, MobileDevice, Message}
It is possible to specify the status of each ambient. The injection room and diagnostic room can be either ’WORKING’ or ’NON WORKING’. When ’WORKING’, it can be ’IDLE’ -no patient inside- or ’BUSY’. The patient is ’WAITING TO BE INJECTED’ (W) before having the injection and ’INJECTED’ (I) after the injection. Next, when the radioactive substance decays to the right level for the examination, he is ’READY’ (R) to be examined. After the examination his status
34
A. Coronato and G. De Pietro
is ’EXAMINED’ (E). Finally, when the radioactive substance becomes harmless, he is ’DECAYED’ (D). It is also possible to assume that other classes of ambients are stateless. The following formulas specify these possible states: Room ∈ {NON WORKING,WORKING*IDLE,WORKING*BUSY}
(1)
Patient ∈ {W,I,R,E,D}
Now, let us focus on some dynamic aspects. As an example, it is possible to formalize the path of a patient who has to move from the hot waiting room to the diagnostic room in order to undergo the examination. Before this, some constraints regarding this movement are expressed. Indeed, concerning the hot waiting room, a patient can enter only when he is just injected or examined and can exit only when ready to be examined or decayed. Therefore, the following constraints must be guaranteed: Patient canEnter(HotWaitingRoom), iff Patient ∈ {I,E}
(2)
–A patient can enter the hot waiting room only if he is INJECTED or EXAMINED Patient canLeave(HotWaitingRoom), iff Patient ∈ {R,D}
(3)
–A patient can leave the hot waiting room only if he is READY or DECAYED HotWaitingRoom canHost(Patient), iff Patient ∈ {I,R,E,D}
(4)
–A patient can stay in the hot waiting room only if he is INJECTED, READY, EXAMINED or DECAYED
It is worth noting that, on one hand, the specification proposed so far describes in an unambiguous way part of the application domain and behavior of its users; on the other hand, constraints and properties (e.g. canEnter ) will affect the software system. One of the most important constraints for the application regards the minimum and maximum time to wait for the examination after the patient has received injection. This is specified by the following formula: T MIN
(5)
where TMIN is the time to wait until the radioactive substance is sufficiently decayed to enable examination; TMAX is the maximum time to wait after the patient has been injected; Tk is the time when the patient is injected; and, Tj is the time when the examination is completed. 5.3
System Architecture and Runtime Verification
In this section, an implementation for runtime verification is provided. For the sake of brevity, the focus is on correctness properties related to the hot waiting room. Monitoring and runtime verification is accomplished by means of the following services: – PatientStateHolder - This is the component that holds and exposes the state of a patient by means of the operations getState and setState. Such operations act accordingly with what defined previously (1);
Safety Critical Ambient Intelligence Applications
35
– HotWaitingRoomStateHolder - The ambient hot waiting room is stateless; but, it must specify which are the states admissible for patients who try to get in/out of, or are hosted by the room. Then, it supports all the features of the RunTimeStateHolder component (figure 2) with the exception of S, getState and setState; – HotWaitingRoomChecker - This is the component that checks, for the hot waiting room, the properties of hosted, entering and leaving patients. Specifically, it supports the operations: canEnter, canLeave and canHost. Thus, the operation canEnter is in charge of verifying the correctness property (2), the operation canLeave refers to the property (3), whereas the operation canHost verifies the correctness property (4). – Timer - This is the component that verifies the temporal constraint (5); – EventService - This service is in charge of dispatching events between software components; – LocationService - This service is able to identify and locate RFID tagged entities; – MessagingService - This service offers functionalities in order to send messages to patients through their mobile devices; – Graphic User Interfaces - Two user interfaces are available: InjectionRoomInterface and DiagnosticRoomInterface. Both enable a doctor to select a patient from a waiting list and next to update the patient status. For an example, in the case of injection, a bottom Injection allows to change the status of the selected patient to ’INJECTED’. Now, let us analyze three possible scenarios. Scenario 1. The operator in the hot waiting room uses the InjectionRoomInterface to select and call a patient for the injection (see figure 4), let us call him Massimo Rossi. Next, the software interface component invokes the MessagingService to send a message to the patient by means of the operation MoveToRoom. Subsequently, the message is received and shown by the Massimo Rossi’s mobile device. Next, after the injection has been performed, the operator pushes the bottom ’INJECTED’, which produces three events: 1) the state of the patient is changed to ’INJECTED’ by means of the setState operation; 2) the Timer is set by the setTimer operation; and, 3) the patient is required to move to the hot waiting room by the operation MoveToRoom. Scenario 2. Let us suppose that after having received the message that requires to move to the injection Room, by mistake Massimo Rossi moves to the hot waiting room. Once at the door, he is located and identified by the LocationService, which alerts the HotWaitingRoomChecker. The HotWaitingRoomChecker asks the PatientStateHolder for the state of the patient. After that, it verifies that the state of Massimo Rossi is not compatible with any admissible by the hot waiting room since he has not yet been injected. Then, a message is sent to the Massimo Rossi’s mobile device in order to require he move away.
36
A. Coronato and G. De Pietro
Fig. 4. Case Study - A piece of system architecture
Scenario 3. Let us suppose the case that the Timer expires while the patient is within the hot waiting room and he has not undergone the examination yet; i.e. the correctness property (15) is not verified. In this case an event is generated by the Timer and notified to the EventService. Next, the Timer updates the state of the patient to ’DECAYED’ by invoking the setState operation. Successively, the HotWaitingRoomChecker, which periodically verifies the state of the patients in the room, detects the new state for Massimo Rossi and concludes that he can no longer stay in the room. Consequently, a message is sent to his mobile device.
6
Conclusions
Ambient Intelligence is one of the most exciting fields of application for pervasive, wireless and embedded computing. Indeed, promising prototypes are continuously being proposed by both academia and industry. However, the design and realization of real -non prototype- systems that may affect the safety of their users must be conducted by means of software engineering approaches and taking into account dependability issues. This paper has presented some middleware mechanisms for runtime verification of Ambient Intelligence applications. Such mechanisms are part of a methodology that relies on the use of formal methods for i) specifying AmI applications in an unambiguous way; ii) statically verifying the correctness of the specification; and iii) dynamically verifying the correctness of the implementation. This
Safety Critical Ambient Intelligence Applications
37
choice is due to the consideration that the correctness of specifications does not imply the correctness of implementation. In contrast, runtime verification of the prototype system in the real domain of application can make possible the collecting of traces that can be used to tune the system or to monitor the system itself in order to execute recovery strategies whenever a correctness proprty is violated.
References 1. Artho, C., et al.: Combining test case generation and runtime verification. Theor. Computer Science 336 (2005) 2. Campbell, R., Ranganathan, A.: Provably Correct Pervasive Computing Environments. In: Proc. of PerCom 2008 (2008) 3. Cardelli, L., Gordon, A.D.: Mobile Ambients. Theor. Computer Science (2000) 4. Cook, J., Augusto, J.C., Jakkula, V.R.: Ambient Intelligence: Technologies, applications, and opportunities. Pervasive and Mobile Computing 5(4) (2009) 5. Coronato, A., De Pietro, G.: Formal Specification of Wireless and Pervasive Healthcare Applications. ACM Transactions on Embedded Computing Systems (to appear in the SI on Wireless Healthcare) 6. Coronato, A., De Pietro, G.: Formal Specification of Dependable Pervasive Applications. In: The Proc. of APSCC 2009 (2009) 7. Coronato, A., De Pietro, G.: Middleware mechanisms for supporting multimodal interactions in smart environments. Computer Communications 31(18) (2008) 8. Coronato, A., De Pietro, G.: MiPeG: A Middleware Infrastructure for Pervasive Grids. Journal of Future Generation Computer Systems 24(1) (2008) 9. Coronato, A., De Pietro, G., Esposito, M.: A Multimodal Semantic Location Service for Intelligent Environments: An Application for Smart Hospitals. Personal and Ubiquitous Computing 13(7) (2009) 10. Crisler, K., Anneroth, M., Aftelak, A., Pulil, P.: The human perspective of the wireless world. Computer Communications 26(1) (2003) 11. d’Amorim, M., Havelund, K.: Event-based runtime verification of java programs. In: Proc. of WODA 2005 (2005) 12. Hall, R.S., Cervantes, H.: An OSGi implementation and experience report. In: Proc. of Consumer Communications and Networking (2004) 13. Karlsson, D., Eles, P., Peng, Z.: Formal verification of component-based designs. Journal of Design Automation for Embedded Systems 11(1) (2007) 14. Lee, R.G., Chen, K.C., Hsiao, C.C., Tseng, C.L.: A Mobile Care System With Alert Mechanism. IEEE Transactions on Information Technology In Biomedicine 11(5) (2007) 15. PERSONA, http://www.aal-persona.org/ 16. Rosu, G., Havelund, K.: Rewriting-based techniques for runtime verification. Automated Software Engineering 12(2) (2005) 17. Saha, G.K.: Software fault avoidance issues. ACM Ubiquity 7(46) (2006) 18. Wu, J., Huang, L., Wang, D., Shen, F.: R-OSGi-based architecture of distributed smart home system. IEEE Trans. on Consumer Electronics 54(3) (2008)
Ranking Algorithm by Contacts Priority for Social Communication Systems Natalia Dudarenko, Juwel Rana, and Kåre Synnes Department of Computer Science and Electrical Engineering Luleå University of Technology, SE-971 87 Luleå, Sweden {natalia.dudarenko,juwel.rana,kare.synnes}@ltu.se
Abstract. The paper presents the ranking algorithm by contacts priority with application for social communication services. This algorithm makes it possible to rank contacts in different social communication services by their priority for the user, and find a preferable communication tools for every contact in the social graph. It is proposed to determine priorities of the contacts using communication history. The ranking algorithm is based on the Markov chains theory and social strength calculation approach. The paper exploits the opportunities for measuring social strength for the contacts and also prioritizes communication tools for simplifying communication. The results are supported by an example. Keywords: Ranking algorithm, Markov chains, Social strength, Communication service, Contacts priority.
1 Introduction The social networking and communication services (Facebook, MySpace, Twitter, etc.) have a large impact on our daily life due to enormous applications and large number of users (e.g., Facebook have 400 Milion active users) [1, 6]. In general, the services form social graph among the contacts, provide online interactions among the contact individuals, facilitate content sharing mechanism, and also offer different kind of group communication tools. In a way, this forwards the traditional communication services (e.g., Email, SMS) to new directions (e.g., Google Wave) where Web technologies and mobile devices are playing a very important role [2]. However, that increased numbers of communication services (social networking and communication services) generates huge amount of contents (e.g., picture, video, presence information, etc), which add new challenges for maintaining contacts, and contents, and initiating group communication [4, 10]. Therefore, this paper addresses to the problem of social ranking of the contact individuals by providing an algorithm and discusses possible application scenarios for that. The paper proposes the ranking algorithm by contacts priority (CP-Rank algorithm) that allows classifying the contacts by their priority for the user and finding a preferable communication tools for every contact. The proposed method prioritizes the contacts based on communication history. However, aggregating communication history log is S. Balandin et al. (Eds.): NEW2AN/ruSMART 2010, LNCS 6294, pp. 38–49, 2010. © Springer-Verlag Berlin Heidelberg 2010
Ranking Algorithm by Contacts Priority for Social Communication Systems
39
not addressed in the paper. The focus of the paper is to provide sophisticated algorithm for ranking social contacts and communication tools. The CP-Rank algorithm is based on Markov chain theory. The social strength calculation algorithm [20] generates the values for the transition probability matrix with respect to the CP-rank algorithm. The rest of the paper is organized as follow. Section 2 provides related work of the research problem. Section 3 describes social strength as a building block of this research. Section 4 proposes CP-Ranking followed by the algorithm. Section 5 includes an example. Section 6 provides evaluation. Finally, Section 7 contains the discussion and future work. The conclusion is in Section 8.
2 Related Work Reto et al., do a survey for group communication and collaboration in mobile settings where they come to the result that contact ranking is an important problem [16]. They apply the CONGA algorithm for detecting group or community of contacts. In our work, the contacts are ranked in more generalized manner to form groups based on social strength rather detecting communities from the communication log. Ranking algorithms for example Pagerank or hypertext induced topic search [17, 18] are specialized for Web link searching. In the paper, we also develop an algorithm called CP-ranking for ranking the contacts and their communication tools in the online social networks. HITS or Pagerank algorithms consider in-links and out-links to form the utility functions used in the algorithm, while in our proposed algorithm we considered number of interactions using different communication tools in online social networking and communication services to measure social strength. Bao et al., propose two algorithms SocialSimRank and SocialPageRank for capturing the popularity of Web pages [7]. The algorithms take feedback of the user in form of social annotations. In our work, we consider frequency of communication with particular tools to identifying the popularity of particular communication tools. We also consider different threshold values based on popularity of tools to get good result. Different approaches for dealing with social strength are described in [5, 9, 16, 20]. In those papers different kind of quantitative methods are applied to capture social strength. The Markov chain theory is applied for finalizing social strength. As opposed to Zhang X. et al., who take into account the importance of the in-links and out-links and the different link weights, we use the social strength to form transition probability matrix [19].
3 Social Strength Calculation This section discusses how social strength can be calculated from different communication and social networking services. In the paper, social strength defines as numerical values to rank the contacts of a particular user’s social graph. For example, if a user has a social graph of three contacts (A, B and C) then their corresponding social strengths can be represented as 0.30, 0.50 and 0.20, which exposes that the user is
40
N. Dudarenko, J. Rana, and K. Synnes
highly connected with the contact B. Since the algorithm was described in an earlier work [20], we limit ourselves to a short description for the self-containment of this paper. Fig. 1 shows a conceptual overview of how social data (e.g., calendar data, sensor data, etc) is collected from different communication and social networking services and processed through aggregated social graph framework. When the user interacts in social media the interactions are captured by the framework [3]. Then the framework applies social strength calculation algorithm to generate rank of contacts in the social graph. For instance, Strength measurement block in the Fig. 1 applies social strength calculation algorithm and provides strongest network of contacts.
Fig. 1. Conceptual Architecture
Communication history log is processed (see Table 1) by following the approach of [20]. According to the approach, each of the frequency parameter f includes three dimensions where, i implies to communication service, j implies to communication tools and k implies to contacts (Fig. 2). In the Table 1, the field where i=1, j=1, k=1 means that Johan, a contact of the social graph interacts with the user through the Facebook messaging tool. The overall social strength of the user U (U is the main user which forms the aggregated social graph for all of his/her contacts) is n
S = ∑ sk = 1 , k =1
(1)
Ranking Algorithm by Contacts Priority for Social Communication Systems
41
k(Contact) j(Communication tool)
f
i(Communication service)
Fig. 2. Frequency Data Type
In the proposed CP-Ranking algorithm, the sum of strength of all values in a row of transition matrix is 1. In [20], utility functions are used to formulate social strength calculation algorithm. The utility function for social strength calculation is q
si , a = ∑ j =1
f i , j , a thresij n
∑ k =1
fi, j , k
,
(2)
where parameter si,a in equation (2) provides the strength of the user U with the contact a, considering all interactions in particular communication service i; q is the number of communication tools of the service i. The parameter thresij is the threshold value estimated over the usage on communication tool j of the communication service i. For instance, if the user has 4 contacts, 4 communication services, in which each services has 2 different communication tools and each of the communication tools has threshold to signify the effects of that particular communication tool, then the equation will provide 4 different strengths for the 4 contacts. The overall sum of the strength will be 1, as mentioned before. p
sa = ∑ thresi si , a .
(3)
i =1
Equation (3) accumulates strengths collected from Si,a to measure global rank of the contacts. Here, the parameter sa provides the overall strength of the user U with the contact a. The parameter thresi is the threshold value estimated over the usage on communication service i. Utility based approach is able to tune threshold values for the different communication tools and services, which is discussed more in [20]. Usage of the tools and services are taken into consideration for tuning threshold parameter dynamically.
42
N. Dudarenko, J. Rana, and K. Synnes Table 1. Frequency logs
Name
Johan (k=1) Natalia (k=2) Kare (k=3) Juwel (k=4)
FB/Message (i=1,j=1) 3 2 1 3
FB/Photo (i=1,j=2) 0 2 0 4
FB/Comments (i=1,j=3) 0 1 6 2
Mobile-phone/Call (i=2,j=1) 5 1 2 2
In the next section the proposed CP-ranking algorithm allow to aggregate the result, rank the contacts and find the preferable service/tools for the user/contacts using social strength.
4 The Proposed CP-Rank Let us consider the communication process in any communication system as a finite Markov chain [13], where S={Si: i=1÷n} is the states set of stochastic process with interactions between the user and his contacts, and between the contacts and communication tools of the different services; P={Pij: i=1÷n, j=1÷n} is the set of probabilities Pij (i=1÷n, j=1÷n) of moving from the state Si to the state Sj. P is a stochastic matrix, the Markov chain is a random walk on the graph defined by the interactions structure of the contacts. The probabilities set satisfies the equation [13,14] n
∑ P =1, i =1
i
(4)
that P = P1
P2 … Pi … Pn = row{Pi : i = 1, n} ;
(5)
Ptr =col{|row(Pij): j=1÷n|, i=1÷n} is the transition probability matrix from the state Si into the state Sj (i,j=1÷n); K={k:0≤k<∞} is the finite number of discrete sample time such that the stochastic process is in the state S(k+1) with probability vector P(k+1) and in the state S(k) with probability vector P(k) satisfy the equation P(k + 1) = P(k ) Ptr (k ) , P(0) = P{S (0)} ,
(6)
where P(0)=P{S(0)} is the initial probability vector of stochastic process being in the initial state S(0). In our problem statement the transition probabilities matrix is based on the values of the social strengths of interactions between the user and contacts which is discussed in the previous section. Let us consider the aggregated social graph as the graph representation of the communication process. For instance, the user U has 3 contacts k1, k2, k3. The
Ranking Algorithm by Contacts Priority for Social Communication Systems
43
connections between the user and his contacts can be realized with 2 services. One of the services has two communication tools (f11, f12), and another one has only one communication tool (f21). The contact k1 apply the communication service 1 with tools f11 or f12; the contact k2 likes to use the communication tools f11 of the service 1 or the tool f21 of the service 2. The third contact k3 prefers to connect by one of the all possible tools. Notice that every contact has individual preference about communication service and communication tool for interaction with the user. The social graph of the system S is shown in Figure 3. Using the social graph we get the next transition probabilities matrix 0 0 0 0 0 0 0 ⎡0 sk (i, f ,k1) sk (i, f , k2) sk (i, f ,k3) ⎤ ⎢0 ⎥ 0 0 0 sk (1, i f 1, k1) sk (1 i , f 2, k1) 0 0 0 0 0 ⎢ ⎥ ⎢0 ⎥ 0 0 0 0 0 sk (1 i , f 1,k2) sk (i2, f 1,k2) 0 0 0 ⎢ ⎥ 0 0 0 0 0 0 0 sk (1, i f 1,k3) sk (1, i f 2,k3) sk (i2, f 1,k3)⎥ ⎢0 ⎢0 ⎥ 0 0 0 1 0 0 0 0 0 0 ⎢ ⎥ Ptr = ⎢0 0 0 0 0 1 0 0 0 0 0 ⎥ ⎢0 ⎥ 0 0 0 0 0 1 0 0 0 0 ⎢ ⎥ 0 0 0 0 0 0 1 0 0 0 ⎢0 ⎥ ⎢ ⎥ 0 0 0 0 1 0 0 0 0 0 ⎢0 ⎥ ⎢0 ⎥ 0 0 0 0 0 0 0 0 1 0 ⎢ ⎥ 0 0 0 0 0 0 0 0 0 1 ⎣0 ⎦
(7)
Fig. 3. Social graph
Let us find the solution of equation (8) using recurrent method [15] ⎧ P (1) = P ( 0 ) Ptr , ⎪ 2 ⎪ P ( 2 ) = P (1) Ptr = P ( 0 ) Ptr Ptr = P ( 0 ) Ptr , ⎨ 2 3 ⎪ P ( 3) = P ( 2 ) Ptr = P ( 0 ) Ptr Ptr = P ( 0 ) Ptr . ⎪ ⎩…
(8)
44
N. Dudarenko, J. Rana, and K. Synnes
So, the solution of (8) is
P ( k ) = P ( 0 ) Ptr k .
(9)
The final probability vector Pf = lim { P ( k ) = P ( 0 ) Ptrk } satisfies the next equation k →∞
[13] Pf = Pf Ptr .
(10)
The solution of equation (10) with respect to the final probability vector Pf allows ranking the probabilities of stochastic process being in the state Si; then we obtain priorities of the user’s contacts and priorities of the contacts with respect to the communication tools. Now, we can describe CP-Rank algorithm.
5 The CP-Rank Algorithm The CP-Ranking algorithm follows the steps mentioned below: 1. Build the graph of communication process, where states are the user, his contacts, and communication tools. 2. Using the social strength calculation algorithm, find the social strength of all interaction between the user and his contacts, and calculate the social strength of every communication tool that are used by the contacts. 3. Form the transition probability matrix Ptr in form (7). 4. Solve the matrix equation (10). 5. Sort the user’s contacts in descending order with respect to the final probabilities. 6. Sort the communication tools in descending order with respect to every contact of the user by the final probabilities.
6 Example Let us consider the graph representation of communication process as graph in figure 3, where the social strengths for every user’s contact and every communication tool are the next values sk (i, f ,k1) =6/16; sk (i, f ,k2) =3/16; sk (i, f ,k3) =7/16; sk (1, i f 1,k1) = 2/6; sk (1, i f 2,k1) = 4/6; sk (1, i f1,k2) = 2/3; sk (i2, f1,k2) =1/3; sk (1, i f 1,k3) =1/7; sk (1, i f 2,k3) = 4/7; sk (i2, f 1,k3) =2/7.
Ranking Algorithm by Contacts Priority for Social Communication Systems
45
In this case, we get the transition probability matrix in the form ⎡0 6/16 3/16 7/16 0 0 ⎢0 0 0 0 2/6 4/6 ⎢ ⎢0 0 0 0 0 0 ⎢ 0 0 0 0 ⎢0 0 ⎢0 0 0 0 1 0 ⎢ Ptr = ⎢0 0 0 0 0 1 ⎢0 0 0 0 0 0 ⎢ 0 0 0 0 ⎢0 0 ⎢ 0 0 0 0 0 0 ⎢ ⎢0 0 0 0 0 0 ⎢ 0 0 0 0 ⎣0 0
0 0 2/3 0 0 0 1 0 0 0 0
0 0 0⎤ 0 0 0 ⎥⎥ 0 0 0⎥ ⎥ 1/7 4/7 2/7⎥ 0 0 0⎥ ⎥ 0 0 0⎥ 0 0 0⎥ ⎥ 0 0 0⎥ ⎥ 1 0 0⎥ 0 1 0⎥ ⎥ 0 0 0 1⎦
0 0 1/3 0 0 0 0 1 0 0
Using (16), we obtain the final probability vector Pf Pf 1 = 6 / 16 Pf 2 + 3 / 16 Pf 3 + 7 / 16 Pf 4 ; Pf 2 = 2 / 6 Pf 5 + 4 / 6 Pf 6 ; Pf 3 = 2 / 3Pf 7 + 1/ 3Pf 8 ; Pf 4 = 1/ 7 Pf 9 + 4 / 7 Pf 10 + 2 / 7 Pf 11 ; Pf 5 = Pf 5 ; Pf 6 = Pf 6 ; Pf 7 = Pf 7 ; Pf 8 = Pf 8 ; Pf 9 = Pf 9 ; Pf 10 = Pf 10 ; Pf 11 = Pf 11 ; Pf 1 + Pf 2 + Pf 3 + Pf 4 + Pf 5 + Pf 6 + Pf 7 + Pf 8 + Pf 9 + Pf 10 + Pf 11 = 1. The priorities for the user’s contacts can be found from the first row of the transition probability matrix and satisfy the next sequence: P3, P1, P2. The solution for searching the prevalent communication tools is Pf = 0 0 0 0 0.1111 0.2222 0.2222 0.1111 0.0476 0.1905 0.0952 . It is obvious that contact k1 prefers the communication tool f12; the contact k2 like to use the communication tool f11; the contact k3 usually connects with the user by the communication tool f12.
7 Evaluation We run an experiment over a simulated data-set to test the algorithm. The data-set is generated using random interaction generator and it is represented in MXML form of data-set to apply mining operations [11]. The generator follows Weibull distribution to produce more realistic interactions [12]. Using the CP ranking method, the ranks of the contacts is measured considering the user’s all interactions stored in the data-set. After that, we made different queries to form groups. To form the groups, group generation factors are used as search-key, for example, communication tools or locations. We preset counters in the interaction generator to log total number of properly
46
N. Dudarenko, J. Rana, and K. Synnes
generated groups using the group generation factors. Also, at the query time, we get total number of properly formed group and total number of returned groups. Using these numbers, we calculate precision and recall, which is quite standard method to evaluate performance of search techniques. The precision and recall satisfy the next form Precision= (number of properly formed group)/total formed group Recall= (number of properly formed group)/total number of properly formed group The result is not fully-completed, yet it is found that the measured precision and recall for group formation factors using different factors are satisfactory, except for the cases where context data have direct influence on group formation. To increase performance for that case, it is needed to have sophisticated method for capturing realtime context, which is not focus of the current paper.
8 Discussion and Future Work The paper exploits Markov chain theory to address the problem of social ranking. In the proposed approach, social strength calculation mechanism provides a strong background to apply the Markov chain theory in the problem space. The problem targets initiation of group communication by exploiting social strength, social rank and finally social contexts in the paper. It investigates how Markov chain theory provides the strength of current state based on probability transition matrix. The transition matrix is formed on the calculated social strength of the previous states. It is not defined yet how the approach could be utilized for predicting suitable communication tools to initiate a group communication or predicting communication patterns. Therefore, the current contribution can be motivated by a simple scenario of group initiation problem where the user would like to have a decentralized social networks (i.e., personalized social network from user perspective) to maintain the contacts and communication tracks. The user wants to have support to discover micro-community to initiate communication and also be able to identify the tools through which reachability to the contacts are higher. For example, an application which groups the Tweet messages (i.e., presence information) from different social networks is based on social strength and able to filter the information overloads could be interesting. Fig. 4 illustrates such a proof-of-concept prototype, where the user’s social interest is considered to group contacts in (high, medium, and low) levels and illustrates the presence information about the interested group. The work is not addressed the problem of social data collection and aggregation in unified manner [8], rather it aims at providing sophisticated algorithm for ranking contacts and predicting preferable communication tools of the contacts. Markov chain approach is comparatively new to apply in social strength problem domain which opens up new challenges. For example, how to prove reliability of such approach in context of other possible approaches of social ranking or is Markov chain based approach scalable enough to deal with large size of transition matrix. All these questions will be addressed in future work.
Ranking Algorithm by Contacts Priority for Social Communication Systems
47
Fig. 4. Twittering Client
9 Conclusion The proposed ranking algorithm by contacts priority with application for social communication services is based on the using social strength calculation algorithm and Markov chains theory. The algorithm allows grouping the contacts in different social communication services with respect to priorities of the individual user. The CP-Rank algorithm enables prediction of user's preferences for different communication services. The CP-Rank algorithm is effective in practice for applications made aware of the social strength between users. However, there are the challenges with respect to data about real-world interactions limited by ensuring privacy. Acknowledgments. This work was partly funded by the Research Area Multimedia Technologies of Ericsson Research, where Stefan Håkansson and Dr Johan Kristiansson have been vital for the direction and execution of the research. The work was also supported by the Centre for Distance-spanning Technology (CDT) at Luleå University of Technology, the Norrbotten Research Council and by the Satin-II research project, funded by the European Regional Funds (mål-2), the Swedish Agency for Economic and Regional Growth, the County Administrative Board of Norrbotten, Norrbotten County Council, and the City of Luleå.
References 1. Banerjee, N., Chakraborty, D., Dasgupta, K., Mittal, S., Nagar, S., Saguna: R-u-in? - exploiting rich presence and converged communications for next-generation activity-oriented social networking. In: MDM ’09: Proceedings of the 2009 Tenth International Conference on Mobile Data Management: Systems, Services and Middleware, pp. 222–231. IEEE Computer Society, Washington (2009)
48
N. Dudarenko, J. Rana, and K. Synnes
2. Rana, J., Kristiansson, J., Josef, H., Synnes, K.: Challenges for mobile social networking applications. In: EuropeComm 2009: Proceedings of the First International ICST Conference Communications Infrastructure. Systems and Applications in Europe, pp. 275–285. Springer, Heidelberg (2009) 3. Campbell, A.T., Eisenman, S.B., Fodor, K., Lane, N.D., Lu, H., Miluzzo, E., Musolesi, M., Peterson, R.A., Zheng, X.: Cenceme: Injecting sensing presence into social network applications using mobile phones (2009) (demo abstract), http://citeseerx.ist.psu.edu/viewdoc/summary? doi=?doi=10.1.1.136.6257 4. Jones, Q., Moldovan, M., Raban, D., Butler, B.: Empirical evidence of information overload constraining chat channel community interactions. In: CSCW ’08: Proceedings of the 2008 ACM conference on Computer supported cooperative work, pp. 323–332. ACM, New York (2008) 5. Ankolekar, A., Szabo, G., Luon, Y., Huberman, B.A., Wilkinson, D., Wu, F.: Friendlee: a mobile application for your social life. In: MobileHCl ’09: Proceedings of the 11th International Conference on Human-Computer Interaction with Mobile Devices and Services, pp. 1–4. ACM, New York (2009) 6. Messinger, P.R., Stroulia, E., Lyons, K., Bone, M., Niu, R.H., Smirnov, K., Perelgut, S.: Virtual worlds - past, present, and future: New directions in social computing. Decis. Support Syst. 47(3), 204–228 (2009) 7. Bao, S., Xue, G., Wu, X., Yu, Y., Fei, B., Su, Z.: Optimizing web search using social annotations. In: WWW ’07: Proceedings of the 16th international conference on World Wide Web, pp. 501–510. ACM, New York (2007) 8. Amitay, E., Carmel, D., Har’El, N., Ofek-Koifman, S., Soffer, A., Yogev, S., Golbandi, N.: Social search and discovery using a unified approach. In: 18th International World Wide Web Conference, WWW2009 (2009), http://data.semanticweb.org/conference/www/2009/paper/186 9. Aleman-Meza, B., Nagarajan, M., Ding, L., Sheth, A., Arpinar, I.B., Joshi, A., Finin, T.: Scalable semantic analytics on social networks for addressing the problem of conflict of interest detection. ACM Trans. Web 2(1), 1–29 (2008) 10. Baatarjav, E.-A., Dantu, R., Phithakkitnukoon, S.: Privacy management for facebook. In: Sekar, R., Pujari, A.K. (eds.) ICISS 2008. LNCS, vol. 5352, pp. 273–286. Springer, Heidelberg (2008) 11. Rozinat, A., Wynn, M.T., van der Aalst, W.M.P., ter Hofstede, A.H.M., Fidge, C.J.: Workflow simulation for operational decision support. Data Knowl. Eng. 68(9), 834–850 (2009) 12. Pinder III, J.E., Wiener, J.G., Smith, M.H.: The weibull distribution: A new method of summarizing survivorship data. Ecology 59(I), 175–179 (1978), http://www.jstor.Org/stable/l936645 13. Romanovsky, I.V.: Discrete analysis. Spb: Nevsky Dialect; BHV Peterburg (2003) (in Russia) 14. Akunov, T.A., Alisherov, S., Omorov, R.O., Ushakov, A.V.: Matrix equations in research of discrete processes over the finite and infinite fields. Ilim, Bishkek (1993) (in Russia) 15. Ushakov, A.V., Habalov, V.V., Dudarenko, N.A.: Mathematical base of systems theory: elements of theory and practical work. SPbSU ITMO, SPb (2007) (in Russia) 16. Grob, R., Kuhn, M., Wattenhofer, R., Wirz, M.: Cluestr: mobile social networking for enhanced group communication. In: Proceedings of the ACM 2009 international Conference on Supporting Group Work, GROUP ’09, Sanibel Island, Florida, USA, May 10-13, pp. 81–90. ACM, New York (2009)
Ranking Algorithm by Contacts Priority for Social Communication Systems
49
17. Page, L., Brin, S., Motwani, R., Winograd, T.: The PageRank citation ranking: Bringing order to the web. Technical report, Stanford Digital Library Technologies Project (1998) 18. Liu, B.: Web Data Mining - Exploring Hyperlinks, Contents and Usage Data ISBN 354037881 19. Zhang, X., Fan, X., Liu, X., Hongyu.: A Ranking Algorithm via Changing Markov Probability Matrix Based on Distribution Factor. In: Fifth International Conference on Fuzzy Systems and Knowledge Discovery, pp. 3–7 (2008) 20. Rana, J., Kristiansson, J., Josef, H., Synnes, K.: Enriching and Simplifying Communication by Social Prioritization. In: ASONAM 2010: Proceedings of the The 2010 International Conference on Advances in Social Networks Analysis and Mining, Odense, Denmark, August 9-11 (2010)
CrysP: Multi-Faceted Activity-Infused Presence in Emerging Social Networks Saguna1 , Arkady Zaslavsky1,2, and Dipanjan Chakraborty3 1 2
CaSIT, Monash University, Melbourne, Victoria, Australia Lule˚ a University of Technology, SE-971 87, Lule˚ a, Sweden 3 IBM Research, India Research Lab, New Delhi, India
[email protected],
[email protected],
[email protected]
Abstract. This paper presents Crystal Presence (CrysP) approach where we fuse presence and activity in the realm of social networking. CrysP functions on the basis of the activity-infused multi-faceted presence information. This information is derived from user’s involvement in multiple activities at any point in time. We propose and develop the CrysP System (CrysPSys) to determine CrysP from the user and his/her environmental context information (mainly relating to user activity, moods/emotions and cognition/thoughts) and then sharing the relevant CrysP with the appropriate devices, services and social-networks (corporate, personal and interest networks) and blogs (text, micro, video, photo and voice). We also highlight the benefits and challenges faced in developing such a system. This paper proposes the architecture and its test-bed and prototype implementation. Keywords: Presence, activity analysis, social networks, context, prototype.
1
Introduction
Activity analysis is an area of extensive research in domains such as healthcare, industry/workshop environments and sports. Current research focuses on lower granularities of activities where activities such as walking and running reveal information about user fitness and encourages user to be more active [8]. In this paper our focus is on inferring higher level activities which the user can share with other people, devices or services in the form of user presence. Presence is defined as the availability of a person for communication in the instant messaging (IM) domain [2]. This presence has later been extended as rich presence [11] to include other context information apart from mere availability. We state that presence extensions like RPID (RFC4480) [3] from existing PIDF (RFC 3863) [4], though a step forward, still needs further consideration. We also believe that both presence aggregation [1] which is the incorporation of presence from multiple sources (devices) and presence federation [1] which allows it to be shared S. Balandin et al. (Eds.): ruSMART/NEW2AN 2010, LNCS 6294, pp. 50–61, 2010. c Springer-Verlag Berlin Heidelberg 2010
CrysP: Multi-Faceted Activity-Infused Presence
51
across multiple technological domains (IMs, social-networking sites, etc) are also taking presence in the right direction. As new communication tools such as social networking sites [5] (like Facebook, LinkedIn and bookmarking sites like Delicious, Digg, StumbleUpon), blogging tools (micro-, video-, photo-, voice-blogs like Twitter, YouTube, Flickr, Bubbly respectively), citizen journalist and community news sites (Newsvine) are gaining popularity, we see a new form of communicating real-time information (relating to the user or his/her environment) in the form of status updates, tweets, etc. In this paper, we make an attempt to merge presence with activity information as we see it in the new realm of communication tools which are not limited to IM alone and include devices, services, social-networking, microblogging and community news sites. Such social interaction tools provide users with a means to communicate and share real-time information about them like activities undertaken, places visited, and the events related to the user as well as those happening in surrounding environments or which the user comes across from different media. This is a continuous process in time and space and updates are made using a number of devices from stationary (home and office) and mobile locations. This information is shared in real-time with people (family, friends, colleagues, acquaintances, complete unknowns, etc) having different types of relationships with the user. Also, we notice that the content of updates on each social networking site is related to the type of domain of that site and in turn, to the type of relationships the user has with the people in those sites. If a user is a member of the professional site LinkedIn, he/she will most likely make work-related status updates on this site and personal status updates may go to Facebook, for example if the user is writing a report and listening to music in library at the same time, he/she may want to share the information “writing a report in the library” on LinkedIn and “listening to music in library” on Facebook. Thus, to automate this process, we infuse user’s multiple activities to determine user’s multiple presence. We believe a) that presence information is linked closely with the user’s activity information, this is also evident from RFC 4480 on RPID, b) that presence is not merely linked with availability on IMs but transcends through all types of social networking tools mentioned previously, c) presence is now multi-faceted i.e., user can have different presence on different tools, and d) users can also have multiple types of presence information at the same point in time since it is linked closely to the user’s activity information as mentioned in a). Thus, in this paper we present Crystal Presence (CrysP) which is the user’s multiple activity infused presence information, implement the CrysP System (CrysPSys) to determine user’s CrysP from their sensed multiple activity information and then intelligently share CrysP using a number of social networking tools. In section 2, we present the related work, followed by the CrysP concept in section 3. The architecture of CrysPSys is presented in section 4 and section 5 gives the details of the test-bed and prototype implementation. Section 6 and 7 present the discussion and conclusion and future work respectively
52
2
Saguna, A. Zaslavsky, and D. Chakraborty
Related Work
In recent years the growth of social-networking and micro-blogs have gained immense popularity and have paved the way for new means of communicating. Location is becoming an important context parameter, and geo-tagging is on the rise, for example, portals like Plazes [6] are being used for location based applications and activities by users. At the same time, there is considerable amount of research in the field of activity analysis being undertaken. An increasing number of sensors are used to detect user’s physical as well as physiological activity. This research is mainly focussed in healthcare or industry environments, but we discuss some of those projects which look outside those two domains and closer to our research domain. Though, there are some similarities here with existing research but we highlight that our system looks at multiple activities of user and then its fusion with a user’s multiple presence information. In [9], the authors have built a context-aware IM client, called Nomatic IM. The main context information incorporated is location using Place Lab WLAN positioning in the campus which is then tagged by the user manually to a particular activity that they might be doing in that location. In the future the system assumes that the user will be doing the same activity at that location. CenceMe system [12], senses low level user activity information (standing/walking/running and stationary/driving are the activities sensed currently) and shares the same presence information across all social networking sites. Sensors inbuilt like the accelerometer on the Nokia N95, Nokia 5500 Sport mobile phone or the Nike+ and microphones and web cameras on the mobile phone were used. A mobile based CenceMe system where majority of the processing is done using light weight classifiers is also built. Personal Environmental Impact Report (PEIR, a Nokia funded research) [7] is an online tool to study the impact of users commuting using different types of transport and to encourage teenagers to use their bikes or public transport. A large-scale people-centric sensor network in [9] is proposed and implemented using two application scenarios: SkiScape and BikeNet. A mobile sensing platform (MSP) in [8], is a wearable device used to detect a person’s activity. The system is tested in different applications like the UbiFit Garden and UbiGreen. A Micro-Blog is researched and implemented as a prototype for users in [10], which leverages localization techniques (like GPS, Wi-Fi and GSM) to spatially geo-tag user micro-blogs. The aforementioned research does not consider identifying multiple user activities simultaneously and then reason about how to share user’s multiple activity information on different social networking sites.
3
Crystal Presence
In this section, we present the following definitions and terminology related to CrysP. Crystal Presence (CrysP) - activity-infused, multi-faceted presence information of a person shared with the relevant devices, services and people.
CrysP: Multi-Faceted Activity-Infused Presence
53
Audience - can include devices which interact with the user in different locations, services that the user might be interested in using and the user’s contacts on multiple social-networking sites such as his/her family, friends and colleagues, as well as those who read the user’s blogs and follow him/her on micro-blogging sites and contacts on IMs. CrysP can be derived from the user’s availability, moods, thoughts and activities. We currently focus only on activity of the user. Figure 1 shows the concept of CrysP. A user’s CrysP is the multi-faceted presence information which is determined using user and environmental context information and user audience consists of everyone who is interested in this real-time information related to the user. As shown in figure 1, we can see that the user’s CrysP can be of many types at different points in time and space for example, based on personal activity information such as “cooking pasta”, “baking a cake”, “reading a novel”, “planning a vacation”. Similarly, this can be based on professional activity information such as “working on research project”, “writing thesis”, and “reading references”.Thus, we present CrysP in figure 1 which shows that, at one point in time, the user’s CrysP can be a combination of these different types of activities (every facet of the crystal depicts a different type of activity at the same point in time); for example, the user can be “Cooking Pasta” and “Reading References” or “Baking a Cake” and “Watching News” or “Writing Code” and “Listening to Pink Floyd” and “Uploading images to Flickr”. These activities can overlap one another. For example, we can start cooking pasta at the start of time (ti ) t1 =1min, and at t2 = 5min we can be reading a reference paper, then resume to cooking pasta at t3 =10min, at t4 =12min we resume reading the reference paper while the pasta cooks on till t4 =20min. Thus, from t1 =1min to t4 =20min, we were involved with two activities which were not related to each other, but overlapped each other in time slots. So, CrysP for the user will be “Cooking Pasta” and “Reading References” in a spatio-temporal domain. The user and his/her audience interact with each other using a wide array of social-networking tools [5] (web portals like Facebook, Orkut, LinkedIn, IMs like Gtalk, Skype, micro-blogs like Twitter, etc). Table 1 shows a wide-array of tools that are available to the user to share his/her multi-faceted presence information (CrysP) with their audience. Some of these tools can be put into broad categories like, social networks, corporate social networks, social bookmarking sites, interest networks, blogs, micro-blogs, video-blogs, photo-blogs, voice-blogs, email and instant messengers. The user shares certain types of information on each of these networks depending on what type of network it is, what type of relationships he/she shares with people on that network along with what type of information related to the user people in that network are interested to see. Therefore, in this research we aim to automate the process of: a) determining CrysP based on activity analysis and b) share the presence and status updates with the relevant audience. Our proposed CrysP System (CrysPSys) can be seen as having the following advantages: a) it helps to reduce the amount of time people usually spend in making status updates; b) people who are unable to update their status information as often as they would like to, are able to
54
Saguna, A. Zaslavsky, and D. Chakraborty
Fig. 1. Crystal Presence (CrysP)
do so via CrysPSys, c) a user can send different updates to different audience sharing only that information which is of interest to a particular set of people; d) the previous point helps in privacy concerns of the user as well; e) this provides an interesting means of passively communicating what you want to with whom you want to in real-time; f) it also opens a lot of opportunities for providing value-added services to the user based on their activities in real-time; g) it can help in reducing energy consumption in smart homes and offices as shown in scenario A below; h) such updates can also help in increasing the revenue of service providers such as telecom companies, when the updates are made from mobile devices. Table 1. Social Network tools for sharing user’s multi-faceted presence Types of Social Networks
Social-networks
Personal Corporate Interest Blogs MicroBlog (Text, Voice Video, Photo, Mood)
Facebook, Orkut, MySpace LinkedIn, Xing HealthRanker, Newsvine, Plaxo, Blogger, Technocrati, Wordpress Twitter, Tumblr, Bubbly, YouTube, TwitPic, I rate my day, Moodmill, GTalk, Skype, MSN, Jaiku
Instant Messengers
Use case scenarios: Scenario A: John and Rhea are a couple living in one house. On Monday, John tells Rhea that he would be working late at office. Rhea reaches home at 7:00PM eats her dinner and leaves John’s dinner in the
CrysP: Multi-Faceted Activity-Infused Presence
55
oven on a timer for three hours later, at 10:00PM, but John is delayed further by an hour. John’s food gets heated at 10:00PM and then gets cold again. John reaches home at 11:30PM and has to reheat his food. This wastes John’s time as well as electricity. If the oven could be switched on after knowing that John has reached home or when John was driving to his home, it could save the need to reheat/recook the food. Similarly, on Tuesday, Rhea was about to finish work half an hour after John, but she was unable to communicate this to him as she was in a meeting with a client. John was driving home, from work and if he was able to learn this information in real-time, John instead of going home and then returning back to pick her up, could have gone straight to Rhea’s office and picked her up. If Rhea was able to share her activity information in real-time with John, this would help save John the time spent in driving again and help the couple save on their fuel costs. Scenario B: Iris is a student at university, from the time she wakes up in the morning, till she sleeps at night, she performs a wide range of activities. She usually makes updates of random activities thoughts, moods, news/articles which she has read, study or research related documents read or experiments conducted to her social-networking sites, IM presence, micro-blog, and blog. These updates are read by a number of people which include as mentioned earlier, her family members, friends, colleagues, advisors, students, and the general public depending on where the updates are made and which tool she uses to make a particular update. This is a time consuming job and can also be distracting when you have too much work on hand. The problem of not being able to make updates and share the right information with the right audience in real time needs to be addressed. This is where our novel architecture is used to determine and detect the presence information based on activity intelligently and then share it with the audience automatically.
4
CrysP System Architecture
The proposed architecture is shown in Figure 2. Our architecture consists of three components, Sensing Component, CrysP Determination Component, and CrysP Sharing Component. 4.1
Sensing Component
The Sensing Component (SC) is responsible for collecting context information from the different wearable sensors such as accelerometers, gyroscopes, location sensors, video cameras, RFID tags, microphones, physiological sensors, and other sensors in the user’s environment. Context information related to the user includes activity information and physiological information which helps to determine his/her cognitive activities as well as moods or emotions. The context information related to the user’s environment is also collected such as location of the user (spatial context), time, day of week, date of the year (temporal context), proximity to other people, the type of social activity, etc. This context information is collected and passed on to the CrysP Determination Component where activity determination is performed.
56
Saguna, A. Zaslavsky, and D. Chakraborty
4.2
CrysP Determination Component
The CrysP Determination Component (CDC) determines the user’s CrysP i.e., the activity infused presence. As shown in figure 2, firstly, activities are inferred by performing segmentation and classification on the raw context information which is received from the sensors. For activity inference we use decision trees to infer body motion activities, we use RFID object interaction with temporal analysis to infer activities linked to objects. We use the Context Spaces Model [13] to infer activities at a higher level of granularity which is then further shared with any device, services or people who may be interested in this information. The pseudo code for inferring user CrysP set is as follows: Repeat for every activity start match if (activity start match) then place time stamp on activity start; iterate through activity sequence till activity end match is not found; if(activity end match) then place time stamp on activity end; Record inferred activity with time stamp; Add every inferred activity with a set time duration to CrysP activity set; Check for each value of CrysP Set in Activity-Presence Ontology; Return(semantic CrysP Set); After the preliminary activity inference is done, we also determine multiple activities (level 2) based on the temporal analysis. In other words, this would also mean activities which overlap each other, or where one activity is paused to perform another activity, and the previous activity is resumed after the completion of the second activity. We only look at activities which are not related to each other i.e., they are not sub-activities of a higher level activity. The activity presence ontology is used to provide us with key sub-activities which are part of higher level activities along with the spatial and temporal aspects of each sub and higher level activity. For example, an activity like “studying in the library” can be inferred by user sitting and interacting with a word document related to his/her studies on a laptop. There can be other ways of inferring the user studying and these can be populated in the ontology. Also, if the user’s location is “in library” it further adds to construct the activity “studying in library”. If user is also interacting with their lunch box while sitting in the library we can add the activity “user is studying and eating food in the library”. This higher level activity can then be shared others. In CDC, the activity-presence ontology is further used to give semantic readability to the inferred activities which creates the activity fused presence information (CrysP). This semantic meaning provides more readability as well as it removes the otherwise plain update of mere activity update, like “walking”, to
CrysP: Multi-Faceted Activity-Infused Presence
57
Fig. 2. Architecture of CrysP System
“walking to catch a tram”, “walking towards the mall”, etc. The use of such ontology gives the advantage of adding more information in the way we would like to express the same activity. Here location information, distances between semantic locations and direction in which user is moving can also be used. The activity-presence diary provides the user a tool to view the CrysP information at later points in time. A user can also provide feedback to the system to help it learn and modify its behaviour, accordingly. 4.3
CrysP Sharing Component
In the CrysP Sharing Component (CSC), we use semantic reasoning to decide how to disseminate and share the CrysP information with the user’s audience. The semantic reasoning allows for sharing the relevant CrysP information with the relevant user audience. The presence-sharing ontology is built which contains the linkages between the different social-networking tools, services and devices with the user, i.e. the type of relationship the user has with the audience. We argue that the user shares a similar class of relationship with all the contacts in each tool he/she uses in regards to the CrysP information that he/she wants to share using a particular tool. Though, such lines are getting blurred with social networking tools and in the future perhaps individual or group based sharing mechanisms may be required. Since, most social networks do not allow currently to share presence information in such a way and a presence update to a network is visible to all friends within that network we hereby take a homogenous
58
Saguna, A. Zaslavsky, and D. Chakraborty
approach. The CSC also includes within it a number of adapters for sharing the CrysP information with the user audience. These adapters can be added or removed based on which tools the user uses to share his/her CrysP information. Another important point to note is that the semantic rules for sharing CrysP across different tools are user-specific. Everyone can have different levels of privacy; some people are more comfortable in sharing personal information with everyone while others find it uncomfortable to do the same. Thus, the semantic rules should be easily modifiable by user intervention. The following is the pseudo code to check the type of CrysP element and then share it with the correct tool. We have built a preliminary ontology and this will be further enhanced in the future. Load Presence Sharing Ontology; For every CrysP element; Check relationship of user with each social tool; If(matched) Send CrysP element as update to that tool; Else Send CrysP as direct message to individual contact in the form of IM/Email/SMS;
5
Test-Bed and Prototype Implementation
We have built a test-bed comprising of several sensors such as accelerometer (body motion), indoor and outdoor positioning systems (location), temperature sensor, RFID tags and reader (object interaction) and other sophisticated software based context collection threads (user’s activity on devices such as laptop and mobile phone). We place a Mulle v3 sensor (which has on it an accelerometer and a temperature sensor) on the user’s waist to gather information about user body motion. Based on previous research in body motion detection using accelerometer we use chose the waist of the user as the most appropriate position [8]. The Mulle sends data at 15Hz which is collected on our laptop where we use Java based Weka API to process the acceleration data. We use J48 decision tree which is an extension of C4.5 to infer three body motion related low-level activities such as sitting, standing, and walking. We achieved an accuracy of 92.56%1. An RFID reader is attached to the user’s wrist which detects RFID tagged objects such as coffee mug, lunch box, etc. The RFID reader scans for tags once every second. Each tag is labeled in terms of an activity, and the RFID readings are inferred as activities using varying time windows for different activities. This eliminates the problem of reading a tag when the object is not intentionally been touched. Multiple readings of an object over time are inferred as an intentional activity. The user’s activity on the laptop and the mobile phone are inferred using both Java and .Net based plug-ins. A .Net based hook is used to infer the user’s interaction with an application running on the device by logging mouse clicks which log current activity on the laptop such as the current 1
This was achieved by using a single user.
CrysP: Multi-Faceted Activity-Infused Presence
59
Fig. 3. CrysP Portal
application (adobe reader, word document, power point presentation, eclipse, etc). A URL logging tool Slogger is used to log URLs visited by the user in a text file and are retrieved every second to monitor user’s browsing activity such as email, online news, Google search, Google scholar search, library website, airline booking site, Flickr photos, etc. Location information is collected using three methods, GPS (outdoor), Wi-fi positioning (Ekahau WLAN positioning is deployed on university campus office of the user), and RFID based location (at home). We use the inbuilt AGPS on an android development phone (ADP1) to gather the location, speed and temperature. The RFID smart home helps in locating the user at home using multiple fixed tags. We have built a web-based GUI as shown in figure 3 to view the user’s CrysP information along with the sensor/context streams. In order to share CrysP we infer activities of the user with varying time windows. The user’s current location and activity can be viewed in the pop-up on the map. Also, the user’s movement patterns and activities at different locations and time can be viewed by clicking anywhere on the user’s path. We are further building the web-interface to make it more interactive as well as to get user feedback. We are working towards running longer experiments on a variety of users to gather different types of information. We are also building the adapters on the mobile phone to infer activity and send updates to the correct social-network/blog directly from the mobile phone. It is also important to note that these activity updates could be shared with devices and services that the user uses in different locations and at different times
60
Saguna, A. Zaslavsky, and D. Chakraborty
in the day. Currently we share these updates only in the social networking domain. Adapters for sharing CrysP were written for a number of social-networking tools like LinkedIn (using OpenSocial Java API) and Facebook (using Facebook Java API); micro-blog like twitter (using Twitter Java API, twitter4j); IM like Skype (using skype win32.jar API) and GTalk (using smack.jar API). We send CrysP updates to each of these tools depending on the preliminary presence sharing (PS) ontology. The PS ontology is written to classify the various tools and various categories of CrysP information along with the relationships between them and the user. We have only assumed five categories, personal, very personal, work, private work and public at present. We send personal updates to Facebook, very personal updates to GTalk, private work updates to Skype, public work updates to LinkedIn and public updates to Twitter. We use the Jena API to interact with the two ontologies and they are written in OWL 1.0 and RDF. Further details of the ontology are omitted here due to lack of space. Figure 4 shows how the CrysPSys shares an element of the user’s CrysP with the user’s Facebook audience.
6
Discussion
The CrysP System (CrysPSys) is currently tested in a scenario limited to a student’s life. We used a specific set of sensors to infer our activities; there is a need to build the system further so that more different types of sensing technology can be incorporated into the same test-bed. There is also a requirement that labeling of activities and creating the relationships between higher and lower level activities can be easily created with limited user interaction. The higher level activity models should be built using these relationships. In our test-bed we currently infer activities using a laptop but we intend to use the mobile phone for inference and sharing of higher level activities, though this would significantly affect the battery life of the device. The system needs to be further evaluated on multiple sets of users on longer periods of time. Further evaluation on reducing user interaction is also important.
Fig. 4. Dissemination of user’s CrysP to Facebook via CrysP plug-in adapter
7
Conclusion and Future Work
In this paper, we present the concept of Crystal Presence (CrysP) which opens new avenues of communicating the user’s activity information with the user’s
CrysP: Multi-Faceted Activity-Infused Presence
61
audience and thus allows users to passively communicate at all times in the day. Our novel system determines activity-infused multi-faceted presence information at a point in time and each facet of this activity infused presence is shared accordingly with the intended audience. We also present a test-bed and prototype which infers multiple higher level activities of a user and then share this information with devices, services and with other people on social networking tools. In future, we will further develop and extend our architecture to incorporate more complex activities and in different scenarios with more number of users.
References 1. Jabber: The XMPP Cloud: Building a Presence-Enabled Infrastructure for RealTime Communication, White Paper (2008) 2. RFC 2778, A Model for Presence and Instant Messaging, http://www.ietf.org/rfc/rfc2778.txt (access date: January 12, 2009) 3. RFC 4480, RPID: Rich Presence Extensions to the Presence Information Data Format (PIDF), http://www.rfc-editor.org/rfc/rfc4480.txt (access date: January 12, 2009) 4. RFC 3863, Presence Information Data Format (PIDF), http://tools.ietf.org/html/rfc3863 (access date: January 12, 2009) 5. Social Networking Sites, http://en.wikipedia.org/wiki/List_of_social_networking_websites (access date: January 12, 2009) 6. Plazes, http://plazes.com/ (access date: January 12, 2009) 7. Agapie, E., et al.: Seeing our signals: Combining location traces and web-based models for personal discovery. In: HotMobile ’08: Proceedings of the 9th workshop on Mobile computing systems and applications, pp. 6–10. ACM, New York (2008) 8. Choudhury, T., et al.: The mobile sensing platform: An embedded activity recognition system. Pervasive Computing 7(2), 32–41 (2008) 9. Eisenman, S.B., et al.: Metrosense project: People-centric sensing at scale. In: WSW 2006 at Sensys (2006) 10. Gaonkar, S., Li, J., Choudhury, R.R., Cox, L., Schmidt, A.: Micro-blog: sharing and querying content through mobile phones and social participation. In: MobiSys ’08: Proceeding of the 6th international conference on Mobile systems, applications, and services, pp. 174–186. ACM, New York (2008) 11. Jachner, J., Petrack, S., Darmois, E., Ozugur, T.: Rich presence: A new user communications experience. Alcatel Telecommunications Review - 1st Quarter (2005) 12. Miluzzo, E., Lane, N.D., Eisenman, S.B., Campbell, A.T.: CenceMe – injecting sensing presence into social networking applications. In: Kortuem, G., Finney, J., Lea, R., Sundramoorthy, V. (eds.) EuroSSC 2007. LNCS, vol. 4793, pp. 1–28. Springer, Heidelberg (2007) 13. Padovitz, A., Loke, S.W., Zaslavsky, A.: Towards a theory of context spaces. In: Proceedings of the Second IEEE Annual Conference on Pervasive Computing and Communications Workshops (2004)
Multichannel System of Audio-Visual Support of Remote Mobile Participant at E-Meeting Andrey L. Ronzhin, Victor Yu Budkov, and Alexey A. Karpov SPIIRAS, 39, 14th line, St. Petersburg, Russia {ronzhin,budkov,karpov}@iias.spb.su
Abstract. Web-based collaboration using the wireless devices that have multimedia playback capabilities is a viable alternative to traditional face-to-face meetings. E-meetings are popular in businesses because of their cost savings. To provide quick and effective engagement to the meeting activity, the remote user should be able to perceive whole events in the meeting room and have the same possibilities like participants inside. The technological framework of the developed intelligent meeting room implements multichannel audio-visual system for participant activity detection and automatically composes actual multimedia content for remote mobile user. The developed web-based application for remote user interaction with equipment of the intelligent meeting room and organization of E-meetings were tested with Nokia mobile phones. Keywords: E-meeting, smart space, remote collaboration, mobile communication, voice activity detection; multimodal interfaces.
1 Introduction Internet applications for teleconference and distant learning (E-meeting and E-lecture) become more popular in business, research, education and other organizations. Such systems allow us to save cost and provide self-paced education and convenient content access and retrieval. However the main part of the secretary work on documentation and connection of remote participants is performed by a human-operator. Another constraint of E-meeting systems is the capacities of the communication network and the capacities of the audio and video user devices that influence on the user interface features and sufficiency of remote participant possibilities. Key issue of remote communication is high uncertainty, caused by the spatial and temporal distance between co-participants [1]. The physical and psychological barriers that exist in hybrid meetings make difficult for remote partners to attend a selected conversational flow, which the participants share the same room and to initiate a new topic of discussion. Thus the main task of the research is to make remote meetings more engaging by giving remote participants more freedom of control in discussions and decision making processes. Research projects AMI [2], CHIL [3], AMIGO [4], CALO [5] were targeted to study various aspects of arrangement of meetings or teleconferencing in smart environments and development of meeting support technology, multi-modal meeting S. Balandin et al. (Eds.): ruSMART/NEW2AN 2010, LNCS 6294, pp. 62–71, 2010. © Springer-Verlag Berlin Heidelberg 2010
Multichannel System of Audio-Visual Support of Remote Mobile Participant
63
browsers, as well as automatic audio-, and video-based summarization systems. Meeting support includes (semi-) automatic retrieval of information needed for support remote participation in hybrid meetings, in which one or more participants are remote and others stay in a shared room [6]. Development of technologies for automatic selection of the best camera view, switching on the projector or whiteboard output and selection of microphone of the current speaker is capable to improve audio-visual support for a remote mobile participant. The novelty of the proposed web-based application for hybrid meeting consists in automatic selection of web-camera of active speaker by multichannel speech activity algorithm and designing a user interface adaptive to mobile device features of remote participant. Section 2 introduces the speaker diarization task and discusses algorithms for speech activity detection in multichannel audio stream. A technological framework of intelligent meeting room and a model of media stream fusion used for designing actual content of the web-based application are considered in Section 3.
2 Methods for Speech Activity Detection in the Multichannel System Audio recordings of meetings in most cases contain spontaneous speech that complicates their automatic processing because of: (1) "overlapped" speech when several persons talk simultaneously; (2) artefacts of speech (lip smack, tongue clucking) and nonverbal pauses (cough, laughter); (3) short remarks and frequent speaker changes. Besides, quality of recordings is strongly affected by environment, layout of speakers and characteristics of the recording equipment [7]. ICSI and CHIL European projects are devoted to the given problem, and the NIST RT-07S standard on evaluation of performance of the systems for meeting transcription [8] has been developed. At present there are some annotated corpora (for example, ISL, AMI, ICSI, NIST) containing up to tens of hours of speech, which have been recorded during formal meetings and impromptu meetings both in natural and informal conditions. The task of speaker diarization (SD), also known under the name «Who Spoke When» consists in detecting the speaker turns within an audio document and in grouping together all the segments belonging to the same speaker [8]. Most of the state-ofthe-art speaker diarization systems perform several key sub-tasks, which are: speech detection, speaker change detection, gender classification and speaker clustering [9]. In some systems the re-training of speaker’s models is made on-line taking into account last speech segments for improvement of system’s performance [10]. A preliminary segmentation of the signal on frames containing silence or speech allows the system to reduce considerably the level of speech recognition errors and to raise speed of processing. Unfortunately, the methods for voice activity detection (VAD) based on an estimation of the level of energy of the signal or its spectrum, well proved at the single channel speech recognition, give poor results at the processing of multichannel audio records with several speakers [11]. For solution of this problem the methods based on a normalization of energy of the multichannel signal [12], an estimation of channel cross-correlation, or Hidden Markov Models containing not 2 states (speech/silence) as usually in VAD methods, but 2K states,
64
A.L. Ronzhin, V.Y. Budkov, and A.A. Karpov
where K is the number of speakers [13], are used. The peculiarity of the latter approach is preliminary training of the models, therefore at the current stage of the research for speech detection in the multichannel system simpler approaches without any training of the models are used. Implementation of crosscorelation methods is possible with the synchronous multichannel recordings. The methods based on channel normalization, calculation of relative signal energy and its spectrum, considering the phonetic rules for speech activity detection are most effective in the case of distributed meetings and using independent devices for recording and processing of audio signals. The developed multichannel system uses a set of web-cameras with built-in microphones for recording speech of participants that provide automatic selection of the current speaker and transmit his/her image to remote participants. The proposed algorithm RESW (Relative Energy estimation in Sliding Window) determines the current active speaker (number of webcam) ωˆ t at the time t by calculating the relative channel energy in the sliding window, thus the random deviation of energy values in the particular channels are suppressed:
ωˆ t = arg max[log10 { n
1 N −1 n 1 M N −1 j ⋅ (∑ κ Amp ⋅ En (t + i ) − ⋅ ∑∑ κ Amp ⋅ E j (t + i))}] , N i =0 M j =1 i =0
n where κ Amp is an amplification of the channel n , which allows the system to take
into account the different gain levels of microphones, N is the size of the sliding window, M is the number of audio channels, En (t + i ) is the short-term energy of L −1
segment of speech: En (t + i ) = ∑ xL2 ( t +i )+ j . To improve the robustness of speech activj =0
ity detection, in the selected channel ωˆ t we use an additional analysis, in which an indicator W is calculated as the number of segments in a sliding window of length N with the value of energy Eωˆ exceeding the threshold E sil : t N −1 ⎧ 0, Eωˆt (t + i ) ≤ Esil . W = ∑ f (t , i ), where f (t , i ) = ⎨ overwise i =0 ⎩1, The threshold function f (t , i) preliminarily classifies audio segment as silence f (t , i) = 0 or speech f (t , i ) = 1 . Two cases for assessment of the indicator W were proposed. Decision on presence of speech in the current sliding window in the channel is made only if the indicator W : a) is greater than zero (an algorithm RESW1 ), or b) exceeds a certain value Wsil (an algorithm RESW sil ). A maximal pause d max between speech segments was taken into account for suppression of single false speech segments. If the number of silence segments between the current segment and the nearest speech segment from left or right exceeds value d max , then the current segment is classified as silence:
W sd
⎧ ⎡ i +d max ⎤ ⎪ 1, ⎢ ∑ f (t + l , i)⎥ − 1 > 0 . , where φ (t , i ) = ⎨ l =i−dmax = ∑ [ f (t , i) ∧ φ (t , i)] ⎣ ⎦ i =0 ⎪0, overwise ⎩ N −1
Multichannel System of Audio-Visual Support of Remote Mobile Participant
65
In the given algorithm ( RESWsil+dist ) the sliding window consists of speech if the following condition was is: W sd > Wsil . The performance of audio stream segmentation of speakers was estimated by calculation of miss rate ( MS ) and false alarm rate ( FA ). The estimates are summed by at analysis of the multichannel system [13]: M channels M
M
M
M
M
M
k =1
k =1
k =1
k =1
k =1
k =1
(S ) MS = ∑ Tk( MS ) /(∑ Tk( S ) + ∑ Tk( MS ) ) , FA = ∑Tk( FA) /(∑Tk( S ) + ∑Tk( FA) ) , where Tk is
quantity of speech segments in the channel k determined by the system as speech;
Tk( MS ) is quantity of speech segments in the channel k missed by the system;
Tk(FA)
is quantity of non-speech segments in the channel k determined by the system as speech. A trade-off between the miss rate and the false alarm rate is chosen at optimization of parameters of the speaker activity detection algorithm [8]. We use the detection error trade-off curve (DET) as the evaluation tool. The DET plot shows the miss rate MS as a function of the false alarm rate FA . For a detection error curve, we can also compute the equal error rate (EER), which corresponds to the threshold, where MS = FA . The database containing five-channel audio recording was prepared for experimental verification of the speech activity detection algorithms. The signal was digitized with the sampling rate of 16 kHz, then samples are combined in the segments (1600 samples per segment). The size of sliding window was 10 segments. The window was shifted with step of 1 segment. Total duration of the database was 28 minutes. In the experiment, five participants consistently said sentences of various lengths from the prepared text. The printed sheets of paper with the text laid on the table in front of each participant. Thus, in this experiment some artificial situation was modeled: participants did not interrupt each other, and read the sentences consistently; no obstacles (hand, paper or other items) arose between the microphone and participant’s mouth; the presenter's face was directed whole toward the microphone over the period of recording. The phrases boundaries in each channel with accuracy up to one audio segment were manually segmented listening all the records. The obtained labeling was used as a standard one for matching and assessment of the performance of automatic segmentation. The obtained dependencies for the error rates MS and FA at several values of E sil for the algorithms RESW1 , RESWsil and RESWsil +dist are coordinated with results of similar researches. In this experiment the phrase boundaries were determined by the algorithm RESW sil + dist more accurately ( EERRESW = 9.16% ). sil + dist
3 Web-Based Application for Organization of Hybrid E-Meetings The developed intelligent meeting room is equipped by projector, touchscreen TV for smart desk application, several cameras for video monitoring of audience and presentation area, personal web-cameras for analysis of behavior of participants sitting at the conference table, as well as three microphone array for sound processing. A combination of the desktop web-cameras and microphone arrays provides both spatial localization of
66
A.L. Ronzhin, V.Y. Budkov, and A.A. Karpov
sound sources and record of participants’ speech with high quality. Multimedia content compiled from audio and video signals captured by the referred devices are used in the web-based application for organization of hybrid E-meetings. Graphical interface of the web-page, on which remote participants could observe a meeting organized inside the intelligent meeting room, contains several basic forms F = {F1 , F2 ,...FN } , where N F is a number of the forms depending on current meeting F
state and features of browser used in a client device. Content of the forms could be changed during meeting, but it always includes a graphical component from a set G = {G1 , G2 ,...GNG } , where N G is a number of used components (in the current version N G = 10 ):
G1 is a component representing current image on the projector; G2 is a component representing current image on the smart desk; G 3 is a component representing current image captured by the video camera
directed to main speaker; G4 is a component representing current image captured by the video camera directed to audience; G5 is a component representing a assemble of current images captured by personal web-cameras directed on participants sitting at the conference table in the meeting room; G6 is a component representing current image captured by a web-camera assigned with a participant, which currently gives a speech comment; G7 is a component representing an indicator of speech duration; G8 is a component representing a clock with time labels of the current meeting; G9 is a component representing a logo of the current meeting; G10 is a component representing main data about the current meeting. The enumerated components connected with corresponding source, which transmits graphical data (the projector – a presentation slide; the smart desk – window with handwriting sketches; the video and web-cameras – frames with an image; the software services – time indicators, logo and other data about meeting). Receiving new data on a source leads to updating content of corresponding form in the web-page. Two main states in meeting process were selected and corresponding notation of forms was used for the current version of web-page software: (1) registration (preparations before meetings), forms F reg ; (2) presentations (main part of meeting), F meeting . Further the number of the meeting states will be increased taking into account peculiarities of participant behavior and necessity of use of specific technical equipment during the discussion, voting and other formal stages. Another important factor influencing on the web-page content is the display resolution and correspondingly maximal size of browser window used for remote view of the meeting. So two classes of device, which have especially different sizes of screen, and corresponding notation of the forms were selected: (1) personal computer, forms F (PC) , (2) mobile device, forms F (MD) . Table 1 shows basic variants of the webpage layout depending on the current state of meeting and type of the client device.
Multichannel System of Audio-Visual Support of Remote Mobile Participant
67
Table 1. The layout variants of the web-page for E-meeting translation
F PC
F MD
F reg
F meet ing
Symbol “/” designates that several variants of graphical content are possible in the form. For instance, the current image G1 on the projector or the current image G2 on the smart desk will be represented in fourth form F4meeting ( PC ) during presentations on the web-page browsed by a personal computer. A list of parameters of objects (equipment, software, participants), which are located inside the intelligent meeting room and influenced on changing the graphical content of the web-page, is presented in Table 2. Values of parameters of hardware and software are determined by a query of its states via TCP/IP protocol or by means of Object Linking and Embedding Automation. Behavior of participants sitting at the conference table, as well as the main speaker located in the presentation area is analyzed by developed technologies of sound source localization, video tracking of moving objects, seeking and tracking of human face.
68
A.L. Ronzhin, V.Y. Budkov, and A.A. Karpov Table 2. The list of parameters influenced on graphical content of the web-page
Object in the Parameters meeting room Notation Values Description Projector 0 Projector turned-off p1 1 Projector turned-on 0 Presentation is not started p2 1 Presentation is started 0 Current slide of a presentation is shown longer, than τ slide , p3 ( t cur
Smart desk
d1 d2
d3
1
Slide of a presentation was changed (time of the changing t slide is
0 1 0 1 0
saved) Wide touchscreen TV is turned-off Wide touchscreen TV is turned-on Smart desk application is not loaded Smart desk application is loaded Touch input was not used longer than τ desk , ( t cur − t desk > τ desk ).
1 Main speaker (presenter)
Personal webcameras assigned with participants sitting at the conference table
0 1
Personal web-camera
0
s2
1 0 1
s3
0 1
c1
0 1
c3i
c4i
c5i
Touch input was used (beginning time of touchscreen using t desk is saved) A speaker in the presentation area is not observed by the video monitoring system Speaker is found in the presentation area Speaker face is not found Speaker head is directed to (the face tracking system founded presenter face) Speech activity in the presentation area is not detected A presenter gives a speech (the sound source localization system detected an activity in the presentation area) Currently there are no speakers at the conference table Currently a participant at the conference table gives a speech comment Personal web-camera i is turned-off
s1
c2i
− t slide > τ slide , where tcur is current time).
i
is turned-on
0
No participant in front of the web-camera
1
There is a participant in front of the web-camera i (the video monitoring system estimates degree of changing image background recorded before the meeting) No speech activity of the participant sitting in front of the webcamera i
0
i
1
A participant sitting in front of the web-camera i gives a speech comment (the multichannel speech activity detection system determines useful signal in the audio channel of the web-camera i )
0
Face of the participant sitting in front of the web-camera i is not found Face position of the participant sitting in front of the web-camera i is detected (the face tracking system found the participant face)
1
Let’s consider the process of graphical content compilation in the forms. Each graphical form F j on the web-page is described by the following tuple
F j = l j , u j , w j , h j , g j , where l j is upper left corner position of the form at the
Multichannel System of Audio-Visual Support of Remote Mobile Participant
69
abscissas axis, u j is upper left corner position of the form at the ordinates axis, w j is the form width, h j is the form height, g j is a graphical content of the form, which is actual and was chosen from the set G . Sizes of the forms could be changed depending on the current features of browser used in the client device. In the forms meeting ( PC ) , F2meeting ( MD ) the number of graphical component is F2meeting ( PC ) , F4 changed depending on the parameter values mentioned in Table 3. In other forms the graphical component numbers are kept during the whole meeting. Selection of the current graphical component g ∈ G for the referred forms is realized by a logicaltemporal model of compiling the graphical interface web-page. The following set of logical rules is an essence of the model: ⎧G3 , s1 ∧ s 2 ∧ s3 ∧ ¬c1 , ⎪G , ¬s ∧ ¬c ∧ (( p ∧ p ) ∨ (d ∧ d )), , ⎪ 4 2 1 1 2 1 2 g 2meeting ( PC ) = ⎨ G s c , ¬ ∧ , 3 1 ⎪ 6 ⎪⎩G9 , overwise. ⎧G1 , ( p1 ∧ p 2 ∧ (¬d 1 ∨ ¬d 2 )) ∨ ( p1 ∧ p 2 ∧ p 3 ∧ d 1 ∧ d 2 ∧ ¬d 3 ) ∨ ⎪ ( p ∧ p ∧ p ∧ d ∧ d ∧ d ∧ (t slide > t desk )), 1 2 3 1 2 3 ⎪ , ⎪ , (( ) ) ( ¬ ∨ ¬ ∧ ∧ ∨ ∧ G p p d d p p 2 ∧ ¬p 3 ∧ d 1 ∧ d 2 ∧ d 3 ) ∨ ⎪ 1 2 1 2 1 g 4meeting ( PC ) = ⎨ 2 ⎪ ( p1 ∧ p 2 ∧ p 3 ∧ d 1 ∧ d 2 ∧ d 3 ∧ (t slide < t desk )), ⎪G 9 , (¬p1 ∨ ¬p 2 ) ∧ (¬d 1 ∨ ¬d 2 ), ⎪ ⎪⎩G 4 , overwise . ⎧G1 , (( p1 ∧ p 2 ∧ (¬ d 1 ∨ ¬d 2 )) ∨ ( p1 ∧ p 2 ∧ p 3 ∧ d 1 ∧ d 2 ∧ ¬ d 3 ) ∨ ⎪ ( p1 ∧ p 2 ∧ p 3 ∧ d 1 ∧ d 2 ∧ d 3 ∧ (t slide > t desk ))) ∧ ¬ s 2 ∧ ¬c1 , ⎪ ⎪G 2 , (((¬ p1 ∨ ¬ p 2 ) ∧ d 1 ∧ d 2 ) ∨ ( p1 ∧ p 2 ∧ ¬ p 3 ∧ d 1 ∧ d 2 ∧ d 3 ) ∨ . ⎪ ( p1 ∧ p 2 ∧ p 3 ∧ d 1 ∧ d 2 ∧ d 3 ∧ (t slide < t desk ))) ∧ ¬ s 2 ∧ ¬c1 , ⎪ meeting ( MD ) = ⎨ g2 ⎪G 3 , ¬ p 3 ∨ ¬ d 3 ∧ s1 ∧ s 2 ∧ s 3 ∧ ¬c1 , ⎪G 4 , ¬ p 3 ∨ ¬ d 3 ∧ ¬ s 2 ∧ ¬c1 , ⎪ ⎪G 6 , ¬ p 3 ∨ ¬ d 3 ∧ ¬ s 3 ∧ c1 , ⎪current component is saved , overwise . ⎩
The component G5 , which consists of actual images of participants sitting at the conference table, is compiled by an analysis of states of personal web-cameras and presence of participants and faces on frame. Let’s identify an set of images from the web-cameras as W = {W1 ,W2 ,...WN } , where N W is a number of the web-cameras W
mounted on the conference table (in the developed meeting room N W = 10 ). Then the component G5 consists of the images captured by turned-on web-cameras, in which a NW
participant is detected: G5 = ∪ (Wi c2i ∧ c3i = 1) . Taking into account limited sizes of i =1
the forms used for representing the component G5 , the number of displayed participants is reduced by an analysis of his speech activity c4 i and/or presence of his face
70
A.L. Ronzhin, V.Y. Budkov, and A.A. Karpov
in the frame c5i . Particularly, the form F1meeting ( MD ) for mobile device contains up to three participant images, so both parameters are used for selection of more active participants: NW
F1meeting ( MD ) = ∪ (Wi c2i ∧ c3i ∧ c4i ∧ c5i = 1) . i =1
The proposed logical-temporal model of compilation of web-page graphical interface was tested on a personal computer and several models of smartphones donated by the Nokia company. Table 3 presents some examples of web-page content during a meeting. The sound source localization system and multichannel speech activity detection system were used for selection of source of audio stream transmitted to remote participant. Speech of a presenter is recorded by the microphone of the web-camera closest to the presentation area. The built-in microphone of the web-camera assigned with the presently active participant sitting at the conference table is used for recording his/her speech. Table 3. Content examples of the web-based application for meeting Screen of client device Personal computer
Mobile device
Presentations
Registration
Meeting state
The developed web-page layout is oriented on passive remote participants. To enhance his potentials a toolbar allowing a participant located outside the meeting room to give a question to the presenter and to share the current discussion.
Multichannel System of Audio-Visual Support of Remote Mobile Participant
71
4 Conclusion Development of the multichannel system for speech activity analysis, which is used for preparation the multimedia reports of distributed meetings, contributes to saving costs of verbatim record preparation, improving quality of teleconferences, as well as provides statistics of meetings and convenient archive access and retrieval. The developed speaker diarization system uses the algorithm for speech activity detection in the multichannel audio stream that provides appropriate quality of phrase boundaries determination and automatic selection of the web-camera of the active speaker. The proposed logical-temporal model of compilation of web-page graphical interface allows remote participants to perceive whole events in the meeting room via a personal computer or smartphones. Acknowledgments. This work is supported by The Russian Program “Research and Research-Human Resources for Innovating Russia in 2009-2013” (contract П2360), RFBR (projects 08-08-00128-а, 08-07-90002-СТ_а) and Nokia.
References 1. Yankelovich, N., Kaplan, J., Simpson, N., Provino, J.: Porta-person: telepresence for the connected meeting room. In: Proceedings of CHI 2007, pp. 2789–2794 (2007) 2. McCowan, I., Gatica-Perez, D., Bengio, S., Moore, D., Bourlard, H.: Towards computer understanding of human interactions. In: Aarts, E., Collier, R.W., van Loenen, E., de Ruyter, B. (eds.) EUSAI 2003. LNCS, vol. 2875, pp. 235–251. Springer, Heidelberg (2003) 3. Waibel, A., Steusloff, H., Stiefelhagen, R.: CHIL—Computers in the human interaction loop. In: Proc. of 5th IAMIS Workshop (2004) 4. Danielson, T., Panoke-Babatz, U., et al.: The AMIGO project: Advanced Group Communication Model for Computer-based Communication Environment. In: Proc. of CSCW’86 (1986) 5. Kaiser, E.: Can Modeling Redundancy in Multimodal, Multi-Party Tasks Support Dynamic Learning? In: CHI ’05 Workshop: CHI Virtuality 2005 (2005) 6. Op den Akker, R., Hofs, D., Hondorp, H., op den Akker, H., Zwiers, J., Nijholt, A.: Supporting Engagement and Floor Control in Hybrid Meetings. In: Esposito, A., Vích, R. (eds.) COST Action 2102 2008. LNCS (LNAI), vol. 5641, pp. 276–290. Springer, Heidelberg (2009) 7. Fredouille, C., Evans, N.: The influence of speech activity detection and overlap on speaker diarization for meeting room recordings. In: INTERSPEECH 2007, pp. 2953– 2956 (2007) 8. The 2009 (RT-09) Rich Transcription Meeting Recognition Evaluation Plan (2009), http://www.itl.nist.gov/iad/894.01/tests/rt/2009 9. Tranter, S., Reynolds, D.: An Overview of Automatic Speaker Diarization Systems. IEEE Trans. ASLP 14(5), 1557–1565 (2006) 10. Markov, K., Nakamura, S.: Never-Ending Learning System for Online Speaker Diarization. In: IEEE ASRU Workshop, pp. 699–704 (2007) 11. Pfau, T., Ellis, D., Stolcke, D.: Multispeaker Speech Activity Detection for the ICSI Meeting Recorder. In: IEEE ASRU Workshop, pp. 107–110 (2001) 12. Dines, J., Vepa, J., Hain, T.: The segmentation of multi-channel meeting recordings for automatic speech recognition. In: ICSLP 2006, pp. 1213–1216 (2006) 13. Laskowski, K., Jin, Q., Schultz, T.: Crosscorrelation based multispeaker speech activity detection. In: INTERSPEECH 2004, pp. 973–976 (2004)
On-the-Fly Ontology Matching in Smart Spaces: A Multi-model Approach Alexander Smirnov1, Alexey Kashevnik1, Nikolay Shilov1, Sergey Balandin2, Ian Oliver2, and Sergey Boldyrev2 1 SPIIRAS, 39, 14 line, 199178, St. Petersburg, Russia {smir,alexey,nick}@iias.spb.su 2 Nokia Research Center, Itämerenkatu 11-13, 00180, Helsinki, Finland {sergey.balandin,ian.oliver,sergey.boldyrev}@nokia.com
Abstract. Proper functioning of smart spaces demands semantic interoperability between knowledge processors connected to it. As a consequence it is required to develop models that would enable knowledge processors to translate on-the-fly between the internal and smart space ontologies. This paper presents the developed multi-model approach to the above problem, where the major elements of the selected approach are described in detail. Keywords: Smart spaces, ontology matching, semantic similarity, semantic interoperability.
1 Introduction Smart-M3 is an open source software platform [1] that aims to provide "Semantic Web" [2] information sharing infrastructure between software entities and various types of devices. The platform combines ideas of distributed, networked systems and Semantic Web [3]. The major application area for Smart-M3 is the development of smart spaces solutions, where a number of devices can use a shared view of resources and services [4, 5]. Smart spaces can provide better user experience by allowing a user to flexibly bring in new devices and seamlessly access all the information in the multi-device system from any of the devices. The reference model of the Smart-M3 smart spaces is shown in Fig. 1 and it is close to the Linda ideology [6, 7]. The smart space is implemented by one or several Service Information Brokers (SIBs) connected to the common space. SIB is a part of the Smart-M3 platform and it is responsible for information management within the smart space. Knowledge Processors (KPs) represent different applications that use the smart space and implement protocols defined by the Smart-M3 platform for communicating with SIBs. The information exchange is organized through transfer of information units (represented by RDF triples “subject-predicate-object”) from KPs to the smart space and back. The information submitted to the smart space becomes S. Balandin et al. (Eds.): ruSMART/NEW2AN 2010, LNCS 6294, pp. 72–83, 2010. © Springer-Verlag Berlin Heidelberg 2010
On-the-Fly Ontology Matching in Smart Spaces: A Multi-model Approach
Application
KP
73
Smart Space
SIB …
Application
KP
Knowledge Base
Fig. 1. Smart space based on Smart-M3: reference model
available to all KPs participating in the smart space. The KPs can also transfer references to the appropriate files/services into the smart space, since not all information can be presented by RDF triples (e.g., a photo or a PowerPoint presentation). As a result the information is not really transferred but shared between KPs by using smart space as a common ground. However, real life implementation of smart spaces faces a number of problems. Let’s consider a simple case study when a user is having his/her mobile device with KP running on it. The user KP (UKP) is going to make a presentation. The LCD projector represented in the smart space by Projector KP (PKP) is a key enabler of this functionality. But in order for the presentation to be shown, the UKP has to share the information about the presentation location (URI) with the PKP. As a result the following conditions have to be fulfilled: 1. 2.
The UKP has to know that the PKP is a part of the smart space; The UKP has to share the presentation’s URI in such a way that the PKP can understand it.
Even from this simple example we can see that proper functioning of the smart space requires semantic interoperability between knowledge processors connected to it. As a result it is required to develop models that would enable knowledge processors to translate on-the-fly between their internal and smart space’s ontology to a certain extent. The paper presents the developed multi-model approach for the above stated problem. Section 2 introduces the main principle of the suggested approach. Sections 3 to 5 describe the major elements of the approach in more details. Section 6 presents the developed demo use case that illustrates the approach. The main results and findings are summarized in Conclusions.
2 Multi-model Approach for On-the-Fly Matching KPs and SIB Ontology In order to analyze the existing ontology matching techniques an extensive state-ofthe-art review has been done, which covered about 20 systems/approaches/projects related to ontology matching. Among them the following ones are worth to be mentioned: GLUE System 8, 9, Falcon-AO 10, MLMA 11, Hovy 12, SKAT 13, ONION
74
A. Smirnov et al.
14, Promt 15, H-Match 16, CTX-MATCH 17, SMART 18, Cupid 19, COMA 20, Similarity Flooding Algorithm 21, AgreementMaker 22, Pattern Based Approach 23, MinSMatch 24, OntoView 25, Chimaera 19. The below proposed approach allows matching of KPs and SIB ontology for the interoperability purposes and is based on the ontology matching model illustrated by Fig. 2. The approach takes into account that the matching procedure has to be done “on-the-fly” by mobile devices with limited resources and remembering the fact that knowledge processors are responsible for performing certain concrete and welldescribed tasks, which means that the corresponding ontology generally should be small – to – medium size and describe only very limited domains. Ontology is represented as RDF triples, consisting of the following ontology elements: subject, predicate, object. Degree of similarity between two ontology elements is in the range [0, 1]. The approach consists of the following steps: 1. Compare ontology elements taking into account synonyms of both ontologies. The degree of similarity between equal elements is set to 1 (maximum value of the degree of similarity). 2. Compare all elements between two ontologies and fill the matrix M using similarity-based model described in Section 2. Matrix M is of size m to n, where m is the number of elements in the first ontology and n is the number of elements in the second ontology. Each element of this matrix contains the degree of similarity between the string terms of two ontology elements using the fuzzy string comparison method described in Section 3.2. 3. For knowledge processors, which can access Internet, e.g. WordNet or Wiktionary, the model of searching semantic distances was developed and presented in Section 3. 3.1. Compare all elements of two ontologies and fill the matrix M’. Matrix M’ is of size m to n, where m is the number of elements in the first ontology and n is the number of elements in the second ontology. Each element of this matrix represents the degree of similarity between two ontology elements. Matching model
Method class
Comparison of elements of two ontologies using synonyms
Combined
Comparison of elements of two Comparison of elements of two ontologies using similarityontologies using semanticbased method based distances search method
Linguistic
Graph-based distance improvement
Contextual
Fig. 2. Multi-model approach to on-the-fly matching KP and SIB ontology
On-the-Fly Ontology Matching in Smart Spaces: A Multi-model Approach
75
3.2. Update values in matrix M, where each new value of elements of M is the maximum value of (M, M’) 4. Improve distance values in the matrix M using the graph-based distance improvement model described in Section 4. As a result the matrix M contains degrees of similarity between ontology elements of two knowledge processors. This allows determining correspondences between elements by selecting degrees of similarities which are higher than the threshold value. The next sections describe major elements of the proposed approach in details.
3 Similarity-Based Model for Matching Ontology The similarity-based model for the ontology matching is presented in Fig. 3. It contains a stemming procedure to normalize words, improved fuzzy string comparison procedure, and normalization procedure. The normalization procedure makes it possible to reduce the resulting similarity for its easier interpretation (Section 3.3). Improved FSC Stemming
Conventional Fuzzy string comparison (FSC)
Normalization
Fig. 3. Similarity-based model for the ontology matching
3.1 Stemming To improve the matching quality the application of the stemming procedure is proposed. This operation makes it possible to identify ontology elements even if they are written in different forms. The following conversions can be done: “looking” → “look”, “device” → “devic”, “vertical” → “vertic”, and “horizontal” → “horizont”. This procedure is uniquely tuned for each supported language. 3.2 Fuzzy String Comparison The basis of the string comparison algorithm is the well-known conventional algorithm that calculates occurrence of substrings from one string in the other string. The algorithm can be illustrated by the following example from matching the iCal and vCal formats. Table 1 presents the comparison of strings “DTSTART” and “START_D”. Number of substrings of the first string contained in the second string: 16 Total number of substrings: 28 The result: string “DTSTART” is contained in the string “START_D” with similarity of 0.57 or 57%. However, this algorithm does not take into account the length of the second string. As a result it was decided to introduce the following additional modifications:
76
A. Smirnov et al. Table 1. Fuzzy string comparison example
Substrings of the first string
Contained in the second string
Substrings of the first string
Contained in the second string
D
+
TST
-
T
+
STA
+
S
+
TAR
+
T
+
ART
+
A
+
DTST
-
R
+
TSTA
-
T
+
STAR
+
DT
-
TART
+
TS
-
DTSTA
-
ST
+
TSTAR
-
TA
+
START
-
AR
+
DTSTAR
-
RT
+
TSTART
-
DTS
-
DTSTART
-
1. Perform the comparison based on the above algorithm twice: FC1 = FuzzyCompare(Element1, Element2) and FC2 = FuzzyCompare(Element2, Element1). 2. Calculate the result as an aggregation of the above results in accordance with the following formula: Re’=n*FC1+(1-n)*FC2, where n is a weight, n∈[0;1]. n = 0.5 sets the same weight to the both strings, n = 0 searches only Request within Class, and n = 1 searches only Class within Request. It is proposed to set n = 0.5 . 3.3 Normalization of Results Since the similarity metrics are obtained by via different techniques they have to be normalized. This is proposed to be done via the following formula: ⎛1 Re = ⎜ − ⎜2 ⎝
1 1⎞ + ⎟ − 4 a ⎟⎠ a * Re ′+
1 1 ⎛1 ⎜ − ⎜2 ⎝
1 1⎞ + ⎟ 4 a ⎟⎠
,
On-the-Fly Ontology Matching in Smart Spaces: A Multi-model Approach
77
where a is a coefficient, a∈(0;+∞). Values of a close to 0 (e. g., 0.00001) produce almost no normalization, values of a more than 1 significantly decrease the resulting similarity level. The value of a should be set empirically.
4 Model of Searching Semantic Distances of Ontology Elements For measuring the semantic distances between ontology elements we use the machine readable dictionary extracted by direct access to Wiktionary and WordNet. This dictionary includes: 1) a set of words defined in dictionary along with 2) definitions given for each word, 3) a set of synonyms for each word, and 4) a set of associated words for each word. Words associated to a word are considered as hyperlinked words occurring in the Dictionary definition given for this word. The nodes of ontology are linked to nodes representing their synonyms and associated words as this is given in the machine-readable dictionary. The links between the nodes are labeled by the distance of relations specified between the concepts represented by these nodes in the machine-readable dictionary. Weight w of a relation specified between two ontology elements t i and t j is assigned as: ⎧0,5 − t i , t j are synonyms ⎪ w = ⎨0,3 − t i , t j are associated words ⎪∞ − t , t are the same word i j ⎩
The values for the weights were evaluated based on the following principles: 1. Semantic distances between synonyms are assumed to be smaller than semantic distances between associated words; 2. Semantic distance is proposed to be calculated as inversely proportional to weights raised to a power. The power is proportional to the path between the compared words. The longer path the greater the semantic distance for the two different words is expected to be. To meet this expectation with reference to the way of the semantic distance calculation, a weight of the relation between two different words should be in the range (0, 1) and ∞. Taken into account the first principle, we empirically selected the weights: 0,5 - for the relation between the synonyms; and 0,3 - for the relation between the associated words; 3. The semantic distance between the same words is equal to 0. For searching semantic distance between elements of two ontologies, nodes of the first ontology are checked for their similarity to nodes of the second ontology. As a measure of similarity the semantic distance (Dist) is used.
Dist (t i , t j ) =
1 sj
∑ ∏ wk S k = si
78
A. Smirnov et al.
where t i , t j – ontology elements; w – weight of lexical relation existing between t i and t j ; S – a set of paths from t i to t j , where a path s is formed by any number of links that connect t i and t j passing through any number of nodes. The degree of similarity depends inversely on distance.
5 Graph-Based Distance Improvement Model The graph-based improvement model for propagation similarities from one ontology element to another is presented in Fig. 4. The main goal of this model is to propagate the degree of similarity between closely matching ontology elements to ontology elements related to them through RDF triples. Ontology 1
Ontology 2 d(xi, xj)
xi
d(ri, rj)
ri
d(x’i, x’j)
x’i
xj rj x’j
Fig. 4. Matching of two ontology model
Set X=(x1, x2, ..., xn) is the set of subjects and objects in the ontology of two knowledge processors. Set Dx = (d(xi, xj), ...) is a degree of similarity between xi and xj. Set R = (r1, r2, ..., rn) is a set of predicates in the ontology of two knowledge processors. Set Dr = (d(ri, rj), ...) is a set of degrees of similarity between ri and rj. Constant Tr is a threshold value that determines whether two ontology elements mapped to each other or not. The following algorithm allows propagating similarity distance to RDF subjects and objects. d(xi, xj) = maximum(Dx) while (d(xi, xj) > Tr) do for each d(x’i, x’j) as xi rm x’i and xj rl x’j do if d(rm, rl) > Tr then d(x’i, x’j) = endif endfor Exclude d(xi, xj) from Dx d(xi, xj) = maximum(Dx) endwhile
2
d (x i , x j ) ∗ d (x i′ , x ′j )
On-the-Fly Ontology Matching in Smart Spaces: A Multi-model Approach
79
The following algorithm allows propagating similarity distance to RDF predicates. for each d(xi, xj) > Tr do for each d(x’i, x’j) > Tr as xi rm x’i and xj rl x’j do d(rm, rl) = 3 d (x i , x j ) ∗ d (x i′ , x ′j ) ∗ d (r m, r l
)
endfor endfor
6 Smart-Room Case Study This is an extended scenario originally proposed in one of our previous papers [6]. The meeting takes place in an “intelligent room” including LCD projector, whiteboard, and access to Internet-based translation service. Users that are planning to make presentation have the special knowledge processor (called User Knowledge Processor or UKP) on their Nokia MAEMO devices, that implements the required functionality as described below. Upon event of user entering to the smart space meeting room, at least the following information from users’ mobile devices become accessible for other smart space UKPs: -
the user profile information (name, photo, domain of interests, e-mail, and phone number, etc.); presentation information (title, keywords, URI).
Before the meeting starts the agenda has to be automatically built and shown on the whiteboard including the speakers’ names, photos, and presentation titles. The current presentation is highlighted on the screen. All meeting participants can see the detailed agenda on the screens of their personal mobile devices. Users can change their user profile items. When the user changes the information about his/her presentation in the profile, UKP changes the appropriate rules in the smart-space. A GUI interface for visualizing the detailed agenda in the screen of participants’ Nokia MAEMO devices has been implemented as well. At the scheduled time the appropriate presentation starts automatically, i.e. the LCD projector is switched ON and the appropriate presentation is shown. The user can control the slideshow directly from the mobile device. Five minutes before the presentation ending time and when the presentation time is about to be over the whiteboard reminds the speaker about the time restrictions. The architecture of this case study is presented in Fig. 5. It includes the following knowledge processors: KP1 – whiteboard (PC); KP2 – projector (PC); KP3 – translator (PC + Internet service); KP4 – coordinator (PC); KP5… N – UKPs (e.g. Nokia N810). The application of the matching procedure is shown in Fig. 6. Let’s consider the following example. The user having mobile device with a knowledge processor (User KP or UKP) running on it and he/she is going to give a lecture. The corresponding ontology is presented in Fig. 7.
80
A. Smirnov et al. Whiteboard
Projector KP1
KP2
Participants’ UKPs
Translator
SIB
KP5
KP3
KP…
Coordinator KP4
KPN
Fig. 5. The case study architecture
The LCD projector represented in the smart space by Projector KP (PKP) is capable of this function, as it is shown by ontology of the projector KP in Fig. 8. This ontology currently located in the smart space. In order for the presentation to be shown UKP has to share the information about the presentation location (URI) with PKP. Participants’ UKPs
Projector PKP SIB
UKP Ontology
PKP
Ontology Matching
Ontology
Matching
Fig. 6. The extended case study architecture user gives
lecture part of
current slide
is of has
type
is
number
file location
Fig. 7. Ontology of user KP
For this purposes the UKP and PKP ontology have to be merged. The element “lecture” from UKP is merged with the “presentation” from PKP as a result of the semantic-based distances search method. Distance between these elements is 0.3 since
On-the-Fly Ontology Matching in Smart Spaces: A Multi-model Approach
projector
81
has
state
show
presentation part of
slide
is of has
is
number
type URI
synonym
file location
synonym
URL
Fig. 8. Ontology of projector KP
they are synonyms according to the Wiktionary. The element “file location” from UKP is merged with the “URI” from PKP as a result of checking the synonyms rules of the PKP ontology. The element “Current slide” is merged with the element “Slide”. The degree of similarity between these elements estimated via the fuzzy string comparison method is 0.59. As a result of this matching UKP knows that the following rule has to be added to the smart space to start the lecture: (“URI”, “is”, “http://myexample.com/presentation1.ppt”).
7 Conclusion The paper proposes the multi-model approach to on-the-fly ontology matching in smart spaces. The approach has been developed integrating the most efficient techniques applicable to Smart-M3 and based on analysis of the state of art. It takes into account that the matching has to be done “on-the-fly” by mobile devices with limited capacities and uses the fact that knowledge processors are responsible for performing certain concrete and well-described tasks. The experiments showed that the matching procedure based on the proposed algorithm (without using on-line dictionaries) takes less than a second on a Nokia N810 mobile device. Any measurement of similarity comes with a certain probability. In this study the minimal probability when the nodes are considered to be the same is captured by comparison to a certain threshold value. The state-of-the-art study discovered that at the moment there are no methods for automatic (without involvement of experts) setting of the threshold value for the matching procedure. Hence, within scope of this project we had to spend significant efforts to minimize as much as possible the need for the expert involvement in the future. As a result, the developed model assumes involvement of experts only at two stages of the design phase and no expert involvement at the run time. First, the experts are obviously needed at the development of the ontology for KPs. At this stage they also define synonyms for the appropriate domain that will be used during matching.
82
A. Smirnov et al.
Another stage when the experts are required is at the setting of the threshold value. This could be done either during the ontology design or postponed till the designed ontology is matched against some other existing ontology from the appropriate domain. For example, it could be specially developed testbed ontology, or accumulated ontology from the real world, depending on the availability. If the threshold is defined when the KP ontology is designed, its quality would be usually worse than when its definition is postponed because of the lack of the matching results with other existing ontology. However, it is not always possible to postpone since the testbed or real world ontology is not be available in many cases, e.g. in case of developing KPs for new domains. In this study we could not find fully automated solution for defining threshold value, but we are continuing to think in this direction and your ideas on how it can be solved are very welcome.
Acknowledgement The presented work is a result of the joint project between SPIIRAS and Nokia Research Center. Part of the implementation work has been also supported by Open Innovations Framework Program FRUCT – www.fruct.org.
References 1. SourceForge, Smart-M3 Retrieved: 10.4, (2010), http://sourceforge.net/projects/smart-m3 2. Semantic Web site, http://www.semanticweb.org (retrieved 10.4.2010) 3. Smart-M3 Wikipedia page, http://en.wikipedia.org/wiki/Smart-M3 (retrieved 10.4.2010) 4. Oliver, I., Honkola, J.: Personal Semantic Web Through a Space Based Computing Environment. In: Middleware for Semantic Web 08 at ICSC’08, Santa Clara, CA, USA (2008) 5. Oliver, I., Honkola, J., Ziegler, J.: Localized Space Based Semantic Webs. In: WWW/Internet Conference, Freiburg, Germany (2008) 6. Carriero, N., Gelernter, D.: Linda in context. Artificial Intelligence and Language Processing 32(4), 444–458 (1989) 7. Wells, G.: Coordination Languages: Back to the Future with Linda. In: Proceedings of the Second International Workshop on Coordination and Adaptation Techniques for Software Entities (WCAT’05), Glasgow, Scotland, pp. 87–98 (2005) 8. AnHai, D., Jayant, M., Pedro, D., Alon, H.: Ontology Matching: A Machine Learning Approach. In: Handbook on Ontologies in Information Systems, p. 660. Springer, Heidelberg (2004) 9. Doan, A., Madhavan, J., Domingos, P., Halevy, A.: Learning to map between ontologies on the semantic web. In: Proceedings of the 11th international conference on World Wide Web, pp. 662–673 (2002) 10. Hu, W., Jian, N., Qu, Y., Wang, Y.G.: A Graph Matching for Ontologies. In: K-CAP Workshop on Integrating Ontologies, pp.43–50 (2005) 11. Alasoud, A., Haarslev, V., Shiri, N.: An Effective Ontology Matching Technique. In: 17th International Symposium ISMIS 2008, Toronto, Canada, May 20-23, pp. 585–590 (2008) 12. Hovy, E.: Combining and standardizing largescale, practical ontologies for machine translation and other uses. In: The First International Conference on Language Resources and Evaluation (LREC), Granada, Spain, pp. 535–542 (1998)
On-the-Fly Ontology Matching in Smart Spaces: A Multi-model Approach
83
13. Mitra, P., Wiederhold, G., Jannink, J.: Semi-automatic Integration of Knowledge Sources. In: 2nd International Conference on Information Fusion (FUSION 1999), Sunnyvale, CA (July 6-8, 1999) 14. Mitra, P., Kersten, M., Wiederhold, G.: Graph-Oriented Model for Articulation of Ontology Interdependencies. In: Zaniolo, C., Grust, T., Scholl, M.H., Lockemann, P.C. (eds.) EDBT 2000. LNCS, vol. 1777. Springer, Heidelberg (2000) 15. Noy, N., Musen, M.: Anchor-PROMPT: Using Non-Local Context for Semantic Matching. In: Workshop on Ontologies and Information Sharing at the Seventeenth International Joint Conference on Artificial Intelligence (IJCAI-2001), Seattle, USA (2001) 16. Castano, S., Ferrara, A., Montanelli, S.: H-Match: an Algorithm for Dynamically Matching Ontologies in Peer-based Systems. In: Proc. of the 1st VLDB Int. Workshop on Semantic Web and Databases (SWDB 2003), Berlin, Germany (2003) 17. Serafini, L., Bouquet, P., Magnini, B., Zanobini, S.: An algorithm for matching contextualized schemas via SAT. Technical report, DIT University of trento, Italy (2003) 18. Noy, N., Musen, M.: SMART: Automated Support for Ontology Merging and Alignment. In: 12th Workshop on Knowledge Acquisition, Modeling, and Management, Banff, Alberta (1999) 19. McGuinness, D.L., Fikes, R., Rice, J., Wilder, S.: An Environment for Merging and Testing Large Ontologies. In: Proceedings of the Seventh International Conference on Principles of Knowledge Representation and Reasoning (KR’2000). Breckenridge, Colorado, USA (2000), http://www.ksl.stanford.edu/software/chimaera/ (retrieved, 10.4.2010) 20. Aumueller, D., Do, H., Massmann, S., Rahm, E.: Schema and Ontology Matching with COMA++. In: Proceedings of the 2005 ACM SIGMOD International Conference on Management of Data, pp. 906–908 (2005) 21. Melnik, S., Garcia-Molina, H., Rahm, E.: Similarity flooding: a versatile graph matching algorithm and its application to schema matching. In: Proceedings of the 18th International Conference on Data Engineering, USA, pp. 117–128 (2002) 22. Cruz, I., Antonelli, F., Stroe, C.: Efficient Selection of Mappings and Automatic Qualitydriven Combination of Matching Methods. In: The Fourth International Workshop on Ontology Matching, Washington DC, USA (2009) 23. Ritze, D., Meilicke, C., Šváb-Zamazal, O., Stuckenschmidt, H.: A pattern-based ontology matching approach for detecting complex correspondences. In: The Fourth International Workshop on Ontology Matching, Washington DC, USA (2009) 24. Giunchiglia, F., Maltese, V., Autayeu, A.: Computing Minimal Mappings. In: The Fourth International Workshop on Ontology Matching, Washington DC, USA (2009) 25. Klein, M., Kiryakov, W., Ognyanov, D., Fensel, D.: Ontology Versioning and Change Detection on the Web. In: 13th International Conference on Knowledge Engineering and Knowledge Management (EKAW02), Sigiienza, Spain (2002) 26. Smirnov, A., Kashevnik, A., Shilov, N., Oliver, I., Lappetelainen, A., Boldyrev, S.: Anonymous Agent Coordination in Smart Spaces: State-of-the-Art. In: Balandin, S., Moltchanov, D., Koucheryavy, Y. (eds.) ruSMART 2009. LNCS, vol. 5764, pp. 42–51. Springer, Heidelberg (2009)
Access Control in Personal Localized Semantic Information Spaces Alexey Koren and Alexander Buntakov State University of Aerospace Instrumentation, Bolshaya Morskaya 67, Saint-Petersburg, Russia {alexey.koren,alexander.buntakov}@extendedmobile.net
Abstract. Leading by global convergency of devices and software from different domains (mobile, desktop, web) information exchange platforms are also evolve into form of smart spaces, personal localized semantic information clouds. New architectures appear, fulfilling needs of crossplatform interaction, semantic reasoning and many other novel functions. In this paper we discuss access control problem for such kind of platforms. Several achievements are described briefly highlighting features that are most interesting in given context, requirements for access control framework are suggested and finally we introduce model of access control framework for one of the most promising smart spaces architecture. Keywords: Smart spaces, access control, personal semantic web.
1
Introduction
Despite of tremendous growth of World Wide Web in last decades and appearance of huge number of social services that affect volumes of information shared on the Web, most information sources are still private and open to limited number of trusted users. That is in human nature and most probably situation will not change in short time. People are about to create personal spaces and share it only with chosen person. On the other hand there are blogs, online diaries, social networks where exactly the same people publish a lot of information with everybody. But during the sharing people still want to hold ownership and control of all published information. Explaining in terms of Tim Berners-Lee’s idea of Giant Global Graph [1]: actors (both corporate or individuals) want personal subgraph of it to be under their control. One of the most promising architectures for fulfilling such need is localized personal semantic spaces (we will use term ”smart space” in this paper) described in [2, 3]. It takes advantage of many concepts, which are now popular in discussions about semantic technologies: automated reasoning, context-awareness, personalization etc. Such technologies and platforms emerge rapidly and we can say that it is predictable and fits common trend of device and software domains convergency [4]. Basic architecture is simple and it is shown on figure 1. Each smart space obtains a shared place for information exchange called ”whiteboard”. This is a S. Balandin et al. (Eds.): ruSMART/NEW2AN 2010, LNCS 6294, pp. 84–91, 2010. c Springer-Verlag Berlin Heidelberg 2010
Access Control in Personal Localized Semantic Information Spaces
85
canvas for creating semantic subgraph. For representing information open standards such as RDF [5] and OWL [6] are used. Agents, also called ’knowledge providers’ (KPs), connect to SIB during simple protocol allowing ”Insert”, ”Retract”, ”Query”, ”Subscribe” and ”Unsubscribe” actions, where: Insert - insert a set of triples; Retract - remove a set of triples; Query - query information store for a set of triples; Subscribe - set up a subscription for a given query; Unsubscribe - terminate processing of the given subscription. This set of actions could be easily represented as traditional [SELECT, INSERT, REMOVE] or [GET, PUT, DELETE] actions set, which seems more clear and strict for defining access.
Fig. 1. Smart space architecture
Agent could potentially connect to any number of spaces, combining information from them for more complex reasoning or simple aggregation for user. 1.1
Smart Spaces’ Security Requirements
Such way of organizing information makes traditional access control methods not applicable for providing security in pervasive highly dynamic systems filled with semantic information. New model of information interaction imposes new security requirements. We consider following as basic ’must have’ features for smart spaces oriented access control methods: 1) Access control on information (RDF) level. Security administrator should be able to restrict or permit explicitly any atomic piece of information, in terms of RDF (as it is standard de-facto for semantic technologies) - any particular triple or a set of triples. 2) Flexible semantic policies definition. From the other hand ability to define access policies on high semantic level is also important. It is more handy way to manage security permissions when it is no need to work on information pieces
86
A. Koren and A. Buntakov
level but on information types level. For example instead of restricting access to every person’s credit card information for anonymous it is more correct way (both from usability and security points of view) to restrict access to all triples of type [*, ”hasCreditCardNumber” ,*]. On higher level it should be possible to group different triple patterns into sets according to its semantics and work with such semantic categories instead of single patterns. In this example all personal financial information which could consist of several card numbers, bank accounts etc. may be combined in one group and administrator will permit or restrict access to this high-level object. Common way for organizing this is using ontologies. 3) Context-awareness and dynamic permission management. In novel ubiquitous multi-agent systems environmental conditions are changing very rapidly. And the same time state of environment sometimes affects permissions of agents. Most usual example is location - mobile agents located in particular area, lets say hotel main hall, may be able to access hotel’s internal services and the same time agents located outside the building are not permitted. Security system must handle all possible context situation gracefully, associating rights to agents in real-time after context changing. 4) Support for reasoning. Many semantic platform gain advantage of reasoning process. During reasoning new portions of information may appear in the system. After this security system must perform secure reasoning to determine what security attributes should be assigned new objects. It should be done not only to prevent unwanted access to private information but also for measuring trust for newly appeared information: may other users take into account these information in some sensitive reasoning processes or not? Lets look at one more example from financial world. Knowledge providers perform reasoning to recalculate estimations of air transport company after messages about planes’ crashes. If somebody will add untrusted information about several incidents - it should not affect stock prices before information will be confirmed officially. 5) Simple lightweight design. It is worth to note that multi-agents environment such as smart spaces are often consist of small devices restricted on energy, memory and/or processing power like mobile phones or consumer electronics. In order to minimize resource consumption for security subsystem it is very important to reuse as much as possible existing infrastructure and make security solution as resource-efficient as possible.
2
Related Work
Rebecca Montanari, Alessandra Toninelli et al. in a series of works dedicated to security for mobile agents [7, 8] achieved significant results in observing area of semantically-rich context-aware security policies based on rules and onthologies. Authors proposed hybrid approach which combines description logic and logical programming for definition of security policies for ubiquitous dynamically changing multi-agent environments. Benefits of this approach is very well-reasoned and these results affects architecture proposed in this paper pretty much.
Access Control in Personal Localized Semantic Information Spaces
87
One more example of using logical programming of context-aware security policies is Cerberus [9], security service of Gaia project. Cerberus is a ubiquitous security mechanism integrating context-awareness and reasoning over security policies for providing access control in ubiquitous computing environments. Interesting idea is to use confidence values for different types of authentication and several probabilistic methods to combine them to net value. But theoretically interesting idea is difficult for implementation in real world. For every system security administrator should manually define confidence levels for every authentication method and their combinations. For really complicated system it might be painful and lead to multiple vulnerabilities (or will require development of verification process). And in most cases confidence levels will anyway be created by administrator which uses Boolean logical clauses to determine permissions in every case, so it is more convenient to don’t use fuzzy conditions instead of natural logical programming and strict statements for defining security. Covington et al. proposed a mechanism of environmental roles [10] as a method of assigning security permissions to subjects and groups basing on state of system (environment). An environmental role can be activated if set of environmental variables (day of week, time, location etc.) satisfies a policy defined by security administration. This set is stored in a role definition. Such kind of encapsulation seems reasonable in many cases but sometimes it could be not very handy in situation when there are many roles which are pretty similar in general but different in details. In this case administrator should either place piece of logic into role definition (not good semantically and from encapsulation point of view) or create a very big set of environmental roles each of them is a small modification of base one. That is one more argument for separation logic of assigning roles basing on system’s state and roles itself. This way usage of environmental variables together with logically programmed security policies is a better way. There were several attempts to describe security infrastructures for RDF stores. In [11] authors propose a policy based access control method for an RDF stores. Idea is clear and simple and most applicable architecture detail is that all metadata and security policies is stored into RDF store itself. In [12] it is proposed to store security object for every triple or triple pattern. This object will define access rights for given triple. Also two-level conflict resolution policy mechanism is described. Main disadvantage of RDF-based security approaches (in our case) is that there is no possibility to define policies on high semantic level using ontologies and context.
3 3.1
Access Control Model Access Control on Low Level
Main functionality of smart spaces is to provide information and knowledge exchange in unified open format. Thereby main functionality of its access control system is to manage access to these information and knowledge. As vice as information mainly represented in RDF format - security will also work with RDF format and RDF stores at lowest level.
88
A. Koren and A. Buntakov
Common representation of RDF data set is oriented graph. Security check for given user and given may be presented as taking a subgraph in accordance with user’s access rights for every vertex and edge. We can manage access to RDF graph by assigning access right to different triple templates. Vertices are represented as triples like [John Smith, ?, ?] or [?, ?, John Smith], which corresponds to ’subject’ or ’object’ position of Mr. Smith in triple accordingly. Edges are represented as triples like [?, foaf:knows, ?]. Given triple describes simple relation between two subjects in social network using FOAF ontology [13]. Managing access to such kind of triple it is easy to make all contacts of person invisible to any group of users, let say ’anonymous’, ’banned’ or ’not friends’. Also it is possible to define access to any particular triple using same notation: [John Smith, ”salaryIs”, ”2000000 euro cash”]. Triple templates may be grouped together by any thematic, access or other property to simplify policy definition and changing. For managing access in dynamically changing environments it is most reasonable to use role-based access control model (RBAC) [14], instead of mandatory or discretionary as most flexible. We suppose P to be set of possible actions [PUT, GET, REMOVE] for all triple templates and template groups. Roles could be assigned to user in two ways: statically and dynamically. We don’t distinguish common roles and environmental or context-based roles explicitly. Secure reasoner is just assigning common roles to user basing on its context and/or state of environment. User can not only gain roles during environment check, but also lose some of roles statically assigned to him/her earlier. Consider the following example. Mr. Smith could be member of ’sales managers’ groups and thereby have access to some sensitive company information. But if he will trying to access this information from public computer or through insecure connection, then during security reasoning process he will temporary lose ’sales managers’ privileges and gain weaker privileges from role ’sales manager in untrusted environment’ and all security check for his queries will be done against this role, instead of his membership statically assigned before. It will have place until he will return to trusted corporate network or will set up VPN connection. 3.2
High-Level Policies Definition
Common way to define high-level security constraints is using ontologies (most likely in OWL format). Security administrator can describe any clause relevant to his/her domain in semantic way, like ’colocatedWithUser’, ’customerOfOurCompany’ or ’memberOfDependentProject’. Roles should also be described in OWL because ownership of some roles could be part of another role’s or policy’s description. Alessandra Toninelli et al in [8] suggest architecture of policy framework for pervasive computing to exploit logical programming [LP] in addition to description logic [DL] for encoding strict and clear rules operating high-level constraints and entities derived from ontology. In such manner we separate and encapsulate 3 main domains:
Access Control in Personal Localized Semantic Information Spaces
89
1) Description of all possible constraints, entities, actors, environment states etc. given in form of ontology; 2) Set of roles, independent from any environmental or other constraints, defining access right for different types of users and/or automated agents; 3) Set of logical rules, defining which user will grant which permissions in current context. At the same time all of 3 domains could be expressed in RDF form and be put into regular RDF storage without need to upgrade its structure for storing new data formats, that is important for small weak devices. 3.3
Support for Reasoning
Dynamic rules-based policy also should be applied to newly added triples. If triple fits any template defined in access policy it automatically gains according information group membership and access constraints. In case when triple doesn’t fit any template its access constraints may be chosen basing on user’s roles set or it can be assigned to default category. Trust maintenance is organized via adding ’trust level’ attribute to information groups or triple templates. Once system controls access to information group it could guarantee that information in given group may be changed only by authorized entities (users or automated agents) with proper level of trust. If triple appears as a result of reasoning process - KP should take trust level of all information considered during the process and set the lowest level from presented to resulting triples. Problems may appear when KP aggregates information from different smart spaces to perform reasoning. They most probably will have totally different trust policies which have no common semantics. In this case KP could treat resulting information as having trust lowest level from different spaces, because at the time when they don’t have common semantics in general and can not be comparable, KP knows meaning of every trust level and is able to decide is information combined from pieces with given trust levels trusted enough or not. Consider example with 2 news resources, each has authorized information from its journalists, verified by editors, and forum where any guy could leave his/her message. Thus first type of information is trusted (has trust level = ”1”) for both resources and forum threads has no trust in general (trust level = ”0”). Mathematically they are equal but let us say first resource is very old and respectful periodical and second one has doubtful reputation. In this case user will treat information appeared by reasoning on main news lines of both resources as not trusted enough, as he knows reputation of both resources. 3.4
Summary
Summarizing all details described above we see effective access control framework for dynamic, localized space based semantic webs as it shown on figure 2.
90
A. Koren and A. Buntakov
Fig. 2. Smart space architecture with security modules
All system information is stored into regular RDF store. Logical programming rules are not RDF-based itself usually, but they can be serialized to unify storage mechanism. Context is integrated with secure reasoning module which analyzes system’s state and assign or remove roles to users. We don’t analyze authentication and authorization methods in this paper that’s why there are no changes at all at the agents’ side. Trust maintenance need some changes on agents’ side but it also not actually a part of access control architecture.
4
Conclusion and Future Work
This paper describes our vision smart spaces access framework architecture. Multi-agent pervasive environment and semantic personalized webs are extremely new and intensively developing areas. All trends in World Wide Web, mobile devices and consumer electronics faster or slower but going in these directions. As soon as pure information now is the only possible level of interaction, research and development in area of security should also evolve to fulfill novel needs. In this paper we done review of related work in these areas and tried to build solution utilizing some of achievements to apply them to providing access control in smart spaces based environments. Practically, this paper leaves too much interesting questions out of scope. Authentication and authorization in multi-spaces environment is very interesting area for research as well as advanced support of reuse and reasoning over information from different sources, which can be described using different ontologies and have totally different trust levels. Solution given in this paper is far too simplistic and we will work further to make this concept-level model to work in dynamic environment with high level of uncertainty.
Access Control in Personal Localized Semantic Information Spaces
91
Acknowledgement The authors would like to thank FRUCT (Finnish-Russian University Collaboration in Telecommunications) program [15] for support and assistance provided and also would like to acknowledge importance of the financial help provided by Nokia and Nokia Siemens Networks.
References 1. Shadbolt, N., Berners-Lee, T., Hall, W.: The Semantic Web Revisited. IEEE Intelligent Systems 21(3), 96–101 (2006) 2. Oliver, I., Honkola, J., Ziegler, J.: Dynamic, Localised Space Based Semantic Webs. In: WWW/Internet Conference, Freiburg, Germany (2008) 3. Oliver, I., Honkola, J.: Personal Semantic Web Through A Space Based Computing Environment. In: Middleware for Semantic Web 08 at ICSC’08, Santa Clara, CA, USA (2008) 4. Koren, A., Balandin, S.: The Road to Smart Spaces: Convergence of Consumer Electronics with a Change of Content Management and Application Design Principles. In: Proceedings of 6th FRUCT Seminar, Helsinki, Finland (2009) 5. Resource Description Framework (RDF): Concepts and Abstract Syntax. In: W3C Recommendation, http://www.w3.org/TR/rdf-concepts/ (retrieved 03-08-2009) 6. OWL Web Ontology Language Overview, W3C Recommendation, http://www.w3.org/TR/owl-features/ (retrieved 03-08-2009) 7. Montanari, R., Toninelli, A., Bradshaw, J.M.: Context-Based Security Management for Multi-Agent Systems. To be published Proc. of the Second IEEE Symposium on Multi-Agent Security and Survivability. IEEE Press, Philadelphia (2005) 8. Toninelli, A., Kagal, L., Bradshaw, J.M., Montanari, R.: Rule-based and Ontologybased Policies: Toward a Hybrid Approach to Control Agents in Pervasive Environments. In: Proc. of the Semantic Web and Policy Workshop (SWPW), in conj. with ISWC 2005, Galway, Ireland (2005) 9. Al-Muhtadi, J., Ranganathan, A., Campbell, R., Mickunas, M.D.: Cerberus: A Context-Aware Security Scheme for Smart Spaces. In: Proceedings of 1st IEEE Annual Conference on Pervasive Computing and Communications (PerCom 2003), Fort Worth, TX, pp. 489–496 (2003) 10. Covington, M.J., Long, W., Srinivasan, S., Dey, A.K., Ahamad, M., Abowd, G.D.: Securing Context-Aware Applications Using Environment Roles. In: SACMAT’01, May 3-4, Chantilly, Virginia, USA (2001) 11. Dietzold, S., Auer, S.: Access Control on RDF Triple Stores from a Semantik Wiki Perspective 12. Jain, A., Farkas, C.: Secure Resource Description Framework: an Access Control Model. In: SACMAT’06, Lake Tahoe, California, USA (2006) 13. FOAF Vocabulary Specification, http://xmlns.com/foaf/spec/ (retrieved, 10-042010) 14. Sandhu, R., et al.: Role based access control models. IEEE Computer 29(2) (February 1996) 15. Finnish-Russian University Collaboration in Telecommunications (FRUCT) program official webpage, http://www.fruct.org (retrieved, 10-04-2010)
Integration of an Answer Set Engine to Smart-M3 Vesa Luukkala and Jukka Honkola Nokia Research Center, Itämerenkatu 11, Helsinki, Finland
[email protected],
[email protected]
Abstract. We present our work in integrating an answer-set engine smodels to Smart-M3 interoperability platform. This gives user a possiblity of programming and trialing logic program reasoning in Smart-M3 environment. The smodels programming model guarantees that evaluation of rules is terminating. We add reactive features on top of the answer-set programming model and discuss its implications.
1
Introduction
Nowadays, a lot of devices with considerable computing power and connectivity mechanisms are present in the everyday environment. The potential of these devices has also increased the interest in getting them to interoperate. Currently, interoperability standards mostly apply for single domains, such as UPnP and DLNA for home entertainment, or are controlled by a single company, such as the Apple ecosystem. The domain specific interoperability standards are challenging for nomadic devices such as smart phones, which ideally should be able to interoperate with any device it can connect to at any given time. A nomadic device will currently have to implement multiple standards to be able to participate in the different domains. Furthermore, the existing standards often target specific use cases instead of attempting to specify a more general interoperability framework. One initiative to target a generic interoperability framework is the semantic web [1]. The goal of the semantic web is to provide machine readable information about the World Wide Web in order to allow automating many tasks that the web is currently used for manually. Eventually this would result in one giant global graph describing the resources of the web in RDF [10] according to ontologies defined in dedicated languages such as in OWL [9]. However, the web is not the best platform for sharing the rapidly changing, dynamic local information about the immediate environment of a device. Smart-M3 is an interoperability platform that allows devices to easily share and access local semantic information, while also allowing the more global semantic information to be available. The information is represented by using same mechanisms as in semantic web, thus allowing easy exchange of global and local information. The RDF representation also allows extremely easy linking of S. Balandin et al. (Eds.): ruSMART/NEW2AN 2010, LNCS 6294, pp. 92–101, 2010. c Springer-Verlag Berlin Heidelberg 2010
Integration of an Answer Set Engine to Smart-M3
93
data also between different ontologies, thus making cross-domain interoperability straightforward. While the used semantic web technologies have been designed to allow reasoning over the information, Smart-M3 currently supports only limited reasoning. Queries in Smart-M3 understand subclass relations, and the query engine also understands the owl:sameAs property. The implemented reasoning mechanisms in Smart-M3 are currently domain-independent, that is, they can take into account properties defined in RDFS or OWL, but not information defined in domain ontologies. However, in many applications creating domain-specific reasoners would be useful. We present an approach where we use logic programming for defining the domain specific reasoning rules. As each RDF triplet can be interpreted as a fact of arity two, there is a straightforward link between RDF content and rulebases. Therefore, various logic programming approaches can be used in a natural manner to deduce new information by means of user constructed rules executed over the fact base. The logic programming approach is useful in creating domain specific instructions for searching and abstracting over the RDF content. Also searching for configurations and policies are natural to express using logic programming means. Furthermore it is possible to generate consitency rules based on ontologies. In short, a rule mechanism can be used to provide additional functional semantics to accompany ontologies. Rules can also be thought of as views in the underlying RDF-base. In this way the logic programming mechanism provides a flexible alternative to query languages like SPARQL [12] or Wilbur Query Language [5]. This approach is not new, for instance Wielemaker et. al. [13] have proposed Prolog to be fundament for application development for semantic web for similar reasons. The smodels [6] is a system for answer set programming. It consists of lparse front end which transforms rule language to input suitable for smodels, which is a solver based on stable model semantics of logic programming. Here we refer to the combination of lparse front end and smodels solver simply as smodels. smodels allows constructing programs using extended logic programming concepts, atoms and inference rules, but by language construction the programs are always decidable. ssls 1 is a command-line tool giving access to Smart-M3 operations with some additional convenience features as well as a possiblity of interfacing external tools to operate on the results of the operations. We use ssls to integrate smodels with Smart-M3 as a node of the system. We do not expect this work to provide a complete programming environment for applications (even though it in principle could be one), but rather as a component in Smart-M3 technology stack. In section 2 we introduce the Smart-M3 interoperability platform, in section 3 we present the integration between smodels and ssls, in section 4 we give an example of use and finally in section 5 we present our conclusions and possibilities for future work. 1
Available at http://sourceforge.net/projects/ssls
94
2
V. Luukkala and J. Honkola
Smart-M3
Smart-M3 is a space-based interoperability platform consisting of two main components: semantic information broker (SIB) and knowledge processor (KP). A smart space is defined as a named search extent of information, where the information is stored in one or more SIBs. The architecture of the system is presented in Figure 1. In the simplest case, one SIB will store all information in a smart space, but there is a possibility of connecting multiple SIBs to make up a smart space. The SIBs making up a smart space will be connected with a protocol that provides distributed deductive closure [2]. Thus, any KP sees the same information content in the smart space regardless of the SIB it connects to2 . The information in the smart space is stored as an RDF graph, usually according to some defined ontology. The use of any specific ontology is not mandated, however. Neither is the consistency of the stored information guaranteed in any way. The KPs may modify and query the information using the insert, remove, update, query and subscribe operations by using the Smart Space Access Protocol (SSAP). Join Leave
: Join a KP to a named space : Leave a named space. After leaving, no more operations may be performed until a join operation Insert : Atomically insert a graph in the space Remove : Atomically remove a graph from the space Update : Atomically update a graph in the SIB. Update is a combination of remove followed by insert, executed atomically Query : Query for information in the space using any supported query language Subscribe : Set up a persistent query in the space; changes to the query results are reported to the subscriber Unsubscribe : Cancel an existing subscription The communication between KPs and SIB may be implemented by using any SOA style service network such as the Network on Terminal architecture [7], or by using a suitable transport protocol such as XMPP or TCP/IP. This approach allows using already deployed communication mechanisms and protects previous investments. Furthermore, it also means that KPs can connect to smart spaces by using the mechanism most suited to them. 2
Distributed deductive closure is not implemented in the current Smart-M3 implementation.
Integration of an Answer Set Engine to Smart-M3
95
Agent Agent
Agent
Agent
M3 Space SIB SIB SIB
Fig. 1. A diagram of the general system architecture: Agents, M3 spaces and SIBs
2.1
Notion of Application
The notion of application in a smart space differs from the concept of traditional application. Instead of a monolithic application running on a single screen, the smart space applications are better seen as scenarios that can be executed to meet the goals of the user. The scenario emerges from the observable actions, taken by the participating knowledge processors, based on the information in the smart space, and also from the use of available services. The scenarios may also be transient: the scenario will change as the participating KPs join and leave the smart space, and also as services become available or unavailable. One of the targets of Smart-M3 has been to make combining the scenarios easy. This is achieved by the loose coupling between the KPs, as they only communicate by modifying and querying the information in the smart space. Thus, any effect that appearance or disappearance of KPs may have on the rest of the environment is limited to the information seen in the smart space. KPs may, of course, communicate with each other by any available means. but this is outside the scope of Smart-M3. A KP understands its own, non-exclusive set of information. This information may be thought of as the ontology of a KP. It is rarely defined formally, and is mostly an implicit concept which can be useful in some situations. Overlap of the sets of information understood by KPs is an essential precondition for achieving interoperability, as otherwise the KPs will never see each others actions. Smart-M3 has been used as an interoperability platform for a number of case studies. The case studies illustrate the applicability of the concept to a variety of areas. A case study of cross domain interoperability is described in [4]. The scenario involved a wellness domain represented by SportsTracker application, home entertainment domain represented by music streaming using a UPnP service, gaming domain represented by SuperTux game, and telecom domain represented by a phone call observer application. All domains share information using a M3 smart space, resulting in improved user experience compared to the case where the components operate independently. For example, when a call is received, the
96
V. Luukkala and J. Honkola
game and music player see the information that a call is ongoing in the smart space and can take appropriate action, in this case pause the music and game. When the call ends, music continues automatically. Furthermore, the played music changes according to the state of the game, for example, when the player loses lives, the music changes appropriately. Finally, the game may award extra lives if the player has been exercising lately. Other case studies include a building automation case [3], a smart meeting room case [11], a home sensor network case [8], and a health care case developed by University of Bologna. In the building automation case, information from the installed sensors and systems is shared for others by using Smart-M3, allowing also devices that do not implement the oBIX or other building automation protocols to access the information. In the smart meeting room case, Smart-M3 is used to coordinate access to the resources of a meeting room, such as a projector for presentations. The home sensor network case is remarkable in that all the hardware used is commercially available. The demo shows input from several sensors on multiple devices, such as Apple iPhone, Nokia N810 and laptop. The health care case uses information from temperature and humidity sensor and patient’s heartrate monitor to determine if the external conditions may be hazardous for the patient. Furthermore, a demonstration showing a combination of the home sensor network and healthcare demos where the information from the sensors used in home sensor network demo was used in the healthcare system, illustrating the ease of creating mash-up applications, has been created. All of these scenarios have been implemented by using various kinds of wrappers on top of SSAP protocol and ontological concepts on procedural languages, like C, C# and Python. All of these scenarios contain KPs whose essential behaviour can easily be expressed as a set of rules and a part of the effort involves re-implementing a specific ruleset and its support mechanisms in the host language.
3
Integrating smodels Using ssls
For this work the purpose of ssls is to integrate the smodels system as a node local rule execution environment. This allows us to easily trial various functionalities, which may later on be integrated to SIB itself. We assume that at least some part of the SIB contents is fetched to the node for the duration of the rule execution. The basic operation of rule execution is that given a command, ssls fetches the contents of the SIB and generates a rulebase from them. The rules themselves have been defined locally in a file or within ssls and they are executed against the rulebase using smodels. The result will be a set of triples to be inserted in the SIB and set of triples to be deleted from the SIB, which are then committed to
Integration of an Answer Set Engine to Smart-M3
97
the SIB. There are mechanisms for limiting the amount of fetched information as well as adding a degree of reactivity, which are described later on in this section. The conversion of the SIB contents is straghtforward: the SIB is queried for all triples and they are converted from subject, predicate, object triplet to fact predicate(subject, object). In order to satisfy the lparse syntax checker all URIs are quoted, which poses some problems for distinguishing between literals and URIs. This is solved by doublequoting the URIs with < and >. For example, a triplet :abcd-2ew4, rdf:type, wp1:Person would produce a fact "
"("<:abcd-2ew4>", "<wp1:Person>"). The rules are written in a text file, which is given to ssls as a parameter. It is also possible to define rules within the ssls interpreter, but only a subset of lparse language features are supported. The result of the smodels execution over the fact base and the rules is parsed for predefined facts which trigger adding content to insert and delete buffers, which are then committed to SIB. The facts for adding triples to insert and delete buffers are correspondingly i and d, both of arity three. Note that both i and d are ssls commands for adding triples to the buffers, which are then committed by the SSAP command update. After all generated facts have been parsed the update command is issued which will cause atomically the deletions to occur, followed by the insertions. In addition to the above i and d there are specific facts which cause ssls to print their arguments. During the rule execution several facts beside the ones which are interpreted (described above) may have been deduced. Unless these facts have been “marshalled” by means of i and d to the SIB, they will disappear after the rule execution. In this way the node with the rule engine is stateless. Of course it may be that the SIB is modified inbetween two rule executions and thus the results will be different for the same rules. However, it may be useful to have a hidden state, which is not published to the SIB. This may be due to security issues or performance reasons. This can be achieved by adding facts to the file with rules, but there is also a mechanims which allows to create a new persistent fact by means of rules themselves. If the fact ssls_fact with arity two is deduced, ssls calls its own rule command to add a new fact to its internal rulebase with the given arguments. The downside of introducing hidden state is that the state is local and the distribution of rules which use ssls_fact may be harder. Above we mentioned that the whole SIB is transferred to the node as the factbase for rule execution. Since the transferral is done by means of the SSAP query command, it is possible to limit the amount of information transferred. ssls contains a view mechanims which allows constructing a query with a set of triple patterns. So far we do not have any mechanism to check whether a view enables all rules, so this is an ad-hoc mechnism of reducing the amount traffic and requires an understanding of the particular rules to be safe. Finally, we may automatically trigger execution of rules by using the SSAP subscription mechanism. The mechanism allows setting up a persistent query and whenever there are changes in SIB pertaining to the query, the node is informed
98
V. Luukkala and J. Honkola
about them. ssls allows naming subscriptions and whenever a subscription with its name prefixed by smls fires, the rule execution is initiated. This provides for a simple reactive mechanism to execute the rules. Note that so far each single execution of the rule-engine is decidable, but the above mechanism together with the notion of state (either in SIB or locally in the node) makes it impossible to determine the behaviour of such a chaining of single rule-engine executions. On the other hand this enables more generic programming, but the required mechanisms and sensibility of this approach needs further research.
4
A Rule Example
We have chosen smodels over other similar alternatives (like Prolog) for couple of reasons: firstly, the execution is decidable, removing a whole class of errors. Secondly, the execution order of the rule body is not relevant. This enables future distribution strategies for rule execution. Thirdly, the system is well established with several implementations. Finally, the smodels language contains several features, which are very useful in constructing rules, namely: possiblity of choosing a defined number of facts, a negation which works as set negation and weighing facts. const threshold = 10. % Rule 1 3 { power_save_on(X),i(X,"<mob:pmode>","low"),d(X,"<mob:pmode>","high") } 3 :""(X,"<mob:Device>"), "<mob:battery_level>"(X,Y), not connection_active(X), lt(Y,threshold). % Rule 2 connection_active(X) :1 {"<mob:gsm>"(X,"on"), "<mob:wlan>"(X,"on")}. % Rule 3 3 { power_save_off(X),i(X,"<mob:pmode>","high"),d(X,"<mob:pmode>","low") } 3 :""(X,"<mob:Device>"), "<mob:battery_level>"(X,Y), ge(Y,threshold). Fig. 2. smodels rules for a simple power manager
We give a simplistic example highlighting the two first features. In this case we have a mobile device, which monitors its battery level and if the level falls
Integration of an Answer Set Engine to Smart-M3
99
under a preset value, puts the device to a power saving mode unless a connection is active. We define connection as either GSM being active or a WLAN connection being active. Figure 2 shows three rules to achieve this. Rule 1 derives three facts power_save_on, i and d for a device instance X, if the battery level is less than threshold and the fact connection_active does not exist in the rulebase. The head has three facts surrounded by minimum and maximum bounds, here both are three. This means that if the body matches, smodels should choose precisely three facts from the three possible choices. Here this is used as shorthand to avoid writing the same body for three different heads, but with different bounds we could choose subset(s) of the facts. Rule 2 states that we can derive connection_active if at least one of the listed possibilities exists. Rule 3 acts as a complement of Rule 1, turning power saving off when needed. Note, that power_save_on and power_save_off facts are not used anywhere in here. They are redundant as we expect that the information produced by i and d will trigger the appropriate behaviour by some other node in the system. However, if there will be other rules, these two facts can be used as components in those. In a way these facts act as an interface that this set of rules is offering for other rules.
5
Conclusions and Future Work
We have presented an integration of an answer-set programming system smodels to Smart-M3. The mechanism we have presented is simple and not specific to smodels, any other logic-programming system could be integrated by similar means. smodels does not as such contain any reactive features but Smart-M3 applications are by their nature reactive. This is why we have introduced a simple reactive mechanism in our system, which allows us the possiblity of implementing those kinds of programs using logic programming mechanisms. This comes at a price as the behaviour of the system when chaining several rule executions may not be easily predictable. One of our future research items is to understand and solve the challenges arising from this. As Smart-M3 does not currently have advanced reasoning capabilities, we use this approach to trial possible solutions to be included in the Smart-M3 system. One interesting possibility is to embed the rule engine in the SIB, and define an ontology for expressing the rulesets. This would allow executing rules defined by KPs in the SIB. For KPs with limited computing power this could be used as a mechanism to offload some of the computation from a small device to a more powerful device. Furthermore, as mentioned in section 2.1, in most scenarios there seems to be a need to utilize reasoning which is well suited to be expressed succinctly using rules. This reasoning could be performed conveniently in the SIB with the embedded rules engine. However, the exact benefits, e.g. less network traffic, and tradeoffs, e.g. performance of SIB, need to be further researched.
100
V. Luukkala and J. Honkola
The previous issues yield more particular technical research topics and work items pertaining to our rule based approach, such as: – how to derive a suitable query expression from the rules to reduce the amount of traffic needed – how to parallelize the execution of rules – implement publishing and reading in rules from SIB, enabling reflective creation of rules – how to derive rules from an ontology to enforce consistency of SIB contents – implement an ssls primitive to start subscriptions from the rules We also aim to reimplement parts of the existing scenarios using the rule based approach.
Acknowledgement This work was supported by SOFIA (Smart Objects For Intelligent Applications) project and funded by Tekes (the Finnish Funding Agency for Technology and Innovation) and the European Commission, and DIEM (Devices and Interoperability Ecosystem) funded by Tekes.
References 1. Berners-Lee, T., Hendler, J., Lassila, O.: The semantic web. Scientific American (May 2001) 2. Boldyrev, S., Oliver, I., Honkola, J.: A mechanism for managing and distributing information and queries in a smart space environment. In: Proceedings of MDMD 2009 (May 2009) 3. Främling, K., Oliver, I., Honkola, J., Nyman, J.: Smart spaces for ubiquitously smart buildings. In: Proceedings of UBICOMM 2009 (October 2009) 4. Honkola, J., Laine, H., Brown, R., Oliver, I.: Cross-domain interoperability: A case study. In: Balandin, S., Moltchanov, D., Koucheryavy, Y. (eds.) ruSMART 2009. LNCS, vol. 5764, pp. 22–31. Springer, Heidelberg (2009) 5. Lassila, O.: Programming Semantic Web Applications: A Synthesis of Knowledge Representation and Semi-Structured Data. PhD thesis, Helsinki University of Technology (November 2007) 6. Niemelä, I., Simons, P.: Smodels – an implementation of the stable model and wellfounded semantics for normal logic programs. In: Fuhrbach, U., Dix, J., Nerode, A. (eds.) LPNMR 1997. LNCS, vol. 1265, pp. 420–429. Springer, Heidelberg (1997) 7. Network on terminal architecture (November 2008), http://www.notaworld.org 8. Openm3 demonstration release, http://sourceforge.net/projects/smart-m3/files/OpenM3.tar.gz/download 9. Web ontology language, http://www.w3.org/2004/OWL/ 10. Resource description framework, http://www.w3.org/RDF/ 11. Smirnov, A., Kashevnik, A., Shilov, N., Oliver, I., Balandin, S., Boldyrev, S.: Anonymous agent coordination in smart spaces: State-of-the-art. In: Balandin, S., Moltchanov, D., Koucheryavy, Y. (eds.) ruSMART 2009. LNCS, vol. 5764, pp. 42–51. Springer, Heidelberg (2009)
Integration of an Answer Set Engine to Smart-M3
101
12. W3c recommendation: Sparql query language for rdf, http://www.w3.org/TR/2008/REC-rdf-sparql-query-20080115 13. Wielemaker, J., Hildebrand, M., van Ossenbruggen, J.: Using Prolog as the fundament for applications on the semantic web. In: Heymans, S., Polleres, A., Ruckhaus, E., Pearse, D., Gupta, G. (eds.) Proceedings of the 2nd Workshop on Applicatiions of Logic Programming and to the web, Semantic Web and Semantic Web Services, Porto, Portugal. CEUR Workshop Proceedings, vol. 287, pp. 84–98. CEUR-WS.org (2007)
A Video Monitoring Model with a Distributed Camera System for the Smart Space Andrey L. Ronzhin, Maria Prischepa, and Alexey Karpov SPIIRAS, 39, 14th line, St. Petersburg, Russia {ronzhinal,prischepa,karpov}@iias.spb.su
Abstract. The developed model of video monitoring based on distributed camera system is intended for automation of registration of meeting participants. The problem of detecting position of participants simultaneously attended in the room and their faces is solved by joint use of wide-angle, omni-directional and PTZ cameras located on walls and ceiling of the room, as well as implementation of face detection technology for photographing the zones of possible occurrence of participants faces. The quality of photos was estimated by accuracy of pointing the PTZ camera and the diagonal value of a face in the captured image. The deviation of face in the frame relatively the center averaged 8% and the square of face was no less than 7% of the image. Also it was observed that the mean time of registration increases, when the number of participants increases and the level of illumination decreases. Further development of the model is aimed at the automation of capturing the main speaker, discussion, vote and other key phases of events. Keywords: Multi-camera tracking, pointing PTZ camera, face detection, meeting support technology, smart space.
1 Introduction Due to the rapid development of broadcast networks, advances in processing and transmitting multimedia information, creation of Internet applications for videoconferencing and distant education, the so-called systems of E-meeting and E-lecture become more popular in business, academic institutes and universities at organization of distributed events. Such systems allow us to save on transportation costs, to choose an individual way of learning and to provide convenient facilities for searching and accessing information. Technological advances in recording, analysis, searching and transmission of