Remote Instrumentation for eScience and Related Aspects
Franco Davoli • Marcin Lawenda Norbert Meyer • Roberto Pugliese Jan We˛ glarz • Sandro Zappatore Editors
Remote Instrumentation for eScience and Related Aspects
123
Editors Franco Davoli DIST-University of Genoa Via Opera Pia 13 16145 Genova Italy
[email protected] Norbert Meyer Pozna´n Supercomputing and Networking Center Noskowskiego 10 61-704 Pozna´n Poland
[email protected] Jan We˛ glarz Pozna´n Supercomputing and Networking Center Noskowskiego 10 61-704 Pozna´n Poland
[email protected]
Marcin Lawenda Pozna´n Supercomputing and Networking Center Noskowskiego 10 61-704 Pozna´n Poland
[email protected] Roberto Pugliese Sincrotrone Trieste S.c.p.A. Department of Information Technology S.S. 14, km 163.5, Area Science Park 34012 Basovizza, Trieste Italy
[email protected] Sandro Zappatore DIST-University of Genoa Via Opera Pia 13 16145 Genova Italy
[email protected]
ISBN 978-1-4614-0507-8 e-ISBN 978-1-4614-0508-5 DOI 10.1007/978-1-4614-0508-5 Springer New York Dordrecht Heidelberg London Library of Congress Control Number: 2011938142 © Springer Science+Business Media, LLC 2012 All rights reserved. This work may not be translated or copied in whole or in part without the written permission of the publisher (Springer Science+Business Media, LLC, 233 Spring Street, New York, NY 10013, USA), except for brief excerpts in connection with reviews or scholarly analysis. Use in connection with any form of information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed is forbidden. The use in this publication of trade names, trademarks, service marks, and similar terms, even if they are not identified as such, is not to be taken as an expression of opinion as to whether or not they are subject to proprietary rights. Printed on acid-free paper Springer is part of Springer Science+Business Media (www.springer.com)
Preface
e-Science is a complex set of disciplines requiring computationally intensive distributed operations, high-speed networking, and collaborative working tools. As such, it is most often (and correctly) associated with grid- and cloud-computing infrastructures and middleware. However, an essential component is sometimes overlooked in this picture, which consists of the scientific instrumentation providing either the very source of data, or special purpose analytical (as in chemical or biological laboratories) and processing tools (as in electronic Very Large Baseline Interferometry, or eVLBI, and in the use of various measurement devices). Making scientific instruments an essential manageable resource over distributed computing infrastructures, such as the grid, has been the focus of a specific research area referred to as Remote Instrumentation, over recent years. European research has been quite active in it, through projects like GRIDCC (Grid Enabled Remote Instrumentation with Distributed Control and Computation), RINGrid (Remote Instrumentation in Grid environment) and DORII (Deployment of Remote Instrumentation Infrastructure), among others. Since 2005, these projects have been accompanied by a Workshop series named INGRID (“Instrumenting” the Grid). This book stems from the fifth Workshop in this series, which was held in May 2010 in Pozna´n, Poland. The contributions touch the main theme of remote instrumentation, along with related technologies that enable the implementation of truly distributed and coordinated laboratories. The content deals with remote instrumentation and sensors’ infrastructure, virtual laboratories and observatories, e-Science applications over the e-Infrastructure, architectural middleware elements, advanced grid and cloud computing, scientific visualization, workflows, virtual machines, instrument simulation, security, network performance monitoring, data mining in virtual laboratories, distance learning tools, software platforms, among other topics. The book is organized in three parts. Part I, Sensors’ Infrastructure, contains six contributions. R. Borghes, R. Pugliese, G. Kourousias, A. Curri, M. Prica, D. Favretto and A. Del Linz examine the use of the functionalities of the Instrument Element (IE), the basic abstraction that allows exposing instruments (of any kind) as manageable resources, to perform computing tasks in a Synchrotron Radiation v
vi
Preface
Facility beamline dedicated to medical imaging. F. Davoli, L. Berruti, S. Vignola and S. Zappatore evaluate the performance of the IE’s embedded publish/subscribe mechanism in acquiring measurement data and delivering them to a multiplicity of users. M. Dias de Assunc¸a˜o, J.-P. Gelas, L. Lef`evre and A.-C. Orgerie present a large-scale energy-sensing infrastructure with software components that allow users to precisely measure and understand the energy usage of their system. P. Gamba and M. Lanati describe the steps and results toward the customization of a seismic early warning system for its porting to grid technology. The contribution by I. Coterillo, M. Campo, J. Marco De Lucas, J. A. Monteoliva, A. Monteoliva, A. Monn`a, M. Prica and A. Del Linz deals with a mobile floating autonomous platform supporting an extensive set of sensors to monitor a water reservoir and its integration in the existing grid infrastructure. M. Adamski, G. Frankowski, M. Jerzak, D. Stokłosa and M. Rzepka examine the security issues of virtual laboratories, also with reference to the use case of a sensor infrastructure. The second part, Software Platforms, includes four contributions dedicated to various supporting software tools and applications, particularly in the field of distance learning. A. Cheptsov and B. Koller present tools for performance analysis of parallel applications. The other four contributions are related to distance learning platforms centered on the grid and on remote instrumentation and virtual laboratories. D. Stokłosa, D. Kaliszan, T. Rajtar, N. Meyer, F. Koczorowski, M. Procyk, C. Mazurek and M. Stroi´nski present the Kiwi Remote Instrumentation Platform and its usage for phenology observations. A. Grosso, D. Anghinolfi, C. Vecchiola and A. Boccalatte describe ExpertGrid, a software infrastructure for the development of decision support tools to train crisis managers. L. Caviglione, M. Coccoli, and E. Punta discuss the issues related with the adoption of virtual laboratories for education in schools and universities, and for the training of professionals, also by using examples in networked control systems. The third part is dedicated to the grid infrastructure, the basis on which e-Science applications build upon. A. Merlo, A. Corana, V. Gianuzzi and A. Clematis address the issue of Quality of Service (QoS), and present a high-level simulator to evaluate QoS-specific tools. D. Adami, C. Callegari, S. Giordano and M. Pagano propose a distributed resource allocation algorithm that is aware not only of grid resources, but also of the network status and of the latencies to reach them, thereby allowing to optimize the choice on the basis of multiple criteria. A. Monari, A. Scemama and M. Caffarel discuss a grid implementation of a massively parallel Quantum Monte Carlo (QMC) code on the EGEE grid architecture, and analyze ´ its performance. B. Strug, I. Ryszka, E. Grabska and G. Slusarczyk introduce a layered graph approach for the generation of a grid structure and its parameters. T. Hopp, M. Hardt, N. Ruiter, M. Zapf, G. Borges, I. Campos and J. Marco present an interactive approach that allows Application Program Interface (API) style communication with grid resources and execution of parallel applications using the Message Passing Interface (MPI) paradigm. M. Sutter, V. Hartmann, M. G¨otter, J. van Wezel, A. Trunov, T. Jejkal and R. Stotzka examine access technologies useable for the development of a Large Scale Data Facility to meet the requirements of various scientific projects and describe a first implementation of a uniform user
Preface
vii
interface. D. Kr´ol, R. Słota, B. Kryza, D. Nikolow, W. Funika and J. Kitowski introduce a novel approach to data management within the Grid environment based on user-defined storage policies, and discuss its implementation in the PL-GRID infrastructure. M. Brescia, S. Cavuoti, R. D’Abrusco, O. Laurino and G. Longo present a distributed web-based data mining infrastructure specialized on Massive Data Sets exploration with Soft Computing methods. Finally, the contribution by D. Adami, A. Chepstov, F. Davoli, M. Lanati, I. Liabotis, S. Vignola, S. Zappatore and A. Zafeiropoulos describes the network monitoring platform deployed in support of grid applications within the DORII project, and presents the measurement results of two selected applications. The editors wish to thank all authors, organizers and participants of the INGRID 2010 Workshop, and particularly the two keynote speakers, Prof. Geoffrey Fox from Indiana University, Bloomington, IN, USA, and Dr. Monika Ka˛ cik from the ´ European Commission, Information Society and Media DG, Unit F3 “GEANT & e-Infrastructure”. Franco Davoli Marcin Lawenda Norbert Meyer Roberto Pugliese Jan We˛ glarz Sandro Zappatore
Contents
Part I
Sensors’ Infrastructure
Grid Computations without the Computing Element: Interfacing Control Systems for On-Line Computations .. . . . . . . . . . . . . . . . . . . Roberto Borghes, Roberto Pugliese, George Kourousias, Alessio Curri, Milan Prica, and Andrea Del Linz Performance Evaluation of the DORII Instrument Element Data Transfer Capabilities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . Luca Berruti, Franco Davoli, Stefano Vignola, and Sandro Zappatore The Green Grid’5000: Instrumenting and Using a Grid with Energy Sensors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . Marcos Dias de Assunc¸a˜ o, Jean-Patrick Gelas, Laurent Lef`evre, and Anne-C´ecile Orgerie
3
15
25
Porting a Seismic Network to the Grid . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . Paolo Gamba and Matteo Lanati
43
Integrating a Multisensor Mobile System in the Grid Infrastructure.. . . . . Ignacio Coterillo, Maria Campo, Jes´us Marco de Lucas, Jose Augusto Monteoliva, Agust´ın Monteoliva, A. Monn´a, M. Prica, and A. Del Linz
59
Defence in Depth Strategy: A Use Case Scenario of Securing a Virtual Laboratory.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . Marcin Adamski, Gerard Frankowski, Marcin Jerzak, Dominik Stokłosa, and Michał Rzepka Part II
75
Software Platforms
Performance Analysis Framework for Parallel Application Support on the Remote Instrumentation Grid. . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 105 Alexey Cheptsov and Bastian Koller ix
x
Contents
New Technologies in Environmental Science: Phenology Observations with the Kiwi Remote Instrumentation Platform . . . . . . . . . . . . 117 Dominik Stokłosa, Damian Kaliszan, Tomasz Rajtar, Norbert Meyer, Filip Koczorowski, Marcin Procyk, Cezary Mazurek, and Maciej Stroi´nski An Agent Service Grid for Supporting Open and Distance Learning.. . . . . 129 Alberto Grosso, Davide Anghinolfi, Antonio Boccalatte, and Christian Vecchiola Education and Training in Grid-Enabled Laboratories and Complex Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 145 Luca Caviglione, Mauro Coccoli, and Elisabetta Punta Part III
Grid Infrastructure
SoRTSim: A High-Level Simulator for the Evaluation of QoS Models on Grid . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 161 Alessio Merlo, Angelo Corana, Vittoria Gianuzzi, and Andrea Clematis MRA3D: A New Algorithm for Resource Allocation in a Network-Aware Grid . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 177 Davide Adami, Christian Callegari, Stefano Giordano, and Michele Pagano Large-Scale Quantum Monte Carlo Electronic Structure Calculations on the EGEE Grid .. . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 195 Antonio Monari, Anthony Scemama, and Michel Caffarel Generating a Virtual Computational Grid by Graph Transformations . . . 209 Barbara Strug, Iwona Ryszka, Ewa Grabska, ´ and Gra˙zyna Slusarczyk Interactive Grid Access with MPI Support Using GridSolve on gLite-Infrastructures .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 227 Torsten Hopp, Marcus Hardt, Nicole Ruiter, Michael Zapf, Gonc¸alo Borges, Isabel Campos, and Jes´us Marco File Systems and Access Technologies for the Large Scale Data Facility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 239 M. Sutter, V. Hartmann, M. G¨otter, J. van Wezel, A. Trunov, T. Jejkal, and R. Stotzka Policy Driven Data Management in PL-Grid Virtual Organizations . . . . . . 257 Dariusz Kr´ol, Renata Słota, Bartosz Kryza, Darin Nikolow, Włodzimierz Funika, and Jacek Kitowski
Contents
xi
DAME: A Distributed Data Mining and Exploration Framework Within the Virtual Observatory . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 267 Massimo Brescia, Stefano Cavuoti, Raffaele D’Abrusco, Omar Laurino, and Giuseppe Longo Network Performance Monitoring for Remote Instrumentation Services: The DORII Platform Test Case . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 285 Davide Adami, Alexey Chepstov, Franco Davoli, Matteo Lanati, Ioannis Liabotis, Stefano Vignola, Sandro Zappatore, and Anastasios Zafeiropoulos Index . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 305
Part I
Sensors’ Infrastructure
Grid Computations without the Computing Element: Interfacing Control Systems for On-Line Computations Roberto Borghes, Roberto Pugliese, George Kourousias, Alessio Curri, Milan Prica, and Andrea Del Linz
Abstract Traditionally the computations on the Grid take place on the Computing Element. In the same line, the Instrument Element is meant to Grid-enable instrumentation. In this chapter we introduce a non-classical use of the Instrument Element where it serves as a virtual instrument for performing a computational task. Specifically it has been used as the interface to a Control System that executes a series of High Throughput Computing tasks in an On-Line manner. This had to be done in order to meet the special requirements of an application in the Synchrotron Radiation Facility Elettra. The instrument control in such institutions is often done through Distributed Control Systems. Such a system is TANGO and the Synchrotron Radiation Facility (SRF) Elettra among other synchrotrons is heavily based on it. The application was for a beamline working in medical imaging (SYRMEP) and aimed to be an improvement of an established Computed Tomography workflow. The task was the generation, in parallel, of sinograms of a specific data format based on the acquired X-ray absorption data. The target was the availability of the complete sinogram data set in a Storage Element by the time of the completion of the CT scan. The Grid related latencies, like job submission and queuing, would have been an issue given the near-real-time requirements. Moreover the inclusion of a set of TANGO devices was necessary and a generic gLite WN would not have been as suitable as a dedicated system. Besides the avoidance of certain Grid parts, the Grid Security infrastructure was required to be fully This works has been partially supported by the DORII EU project (European Commission within the 7th Framework Programme (FP7/2007-2013) under grant agreement no. RI-213110) www.dorii.eu The authors would like to thank Diego Dreossi, a senior SR X-ray CT and Medical Imaging specialist working for the SYRMEP@Elettra beamline, for his valuable feedback and cooperation during all the phases of the project. R. Borghes • R. Pugliese () • G. Kourousias • A. Curri • M. Prica • A.D. Linz Scientific Computing Team, IT Group, Sincrotrone Trieste S.C.p.A., Strada Statale 14 - km 163,5 in AREA Science Park, 34149 Basovizza, Trieste, Italy e-mail: author’
[email protected] F. Davoli et al. (eds.), Remote Instrumentation for eScience and Related Aspects, DOI 10.1007/978-1-4614-0508-5 1, © Springer Science+Business Media, LLC 2012
3
4
R. Borghes et al.
utilised in the final solution. The design followed a bottom-up approach: (a) design and preparation of a dedicated system based on virtualisation, (b) development of a parallel sinogram generator, (c) deployment of suitable TANGO devices for controlling the data acquisition, the generator, and the On-Line progress, (d) a TANGO-to-IE bridge to export the devices as IM, (e) utilising a Grid Web Portal (VCR) in order to serve as the end-user GUI for the application. In this contribution (I) we introduce a novel concept where computation may take place outside the CE, (II) we design an architecture where a Distributed Control System is piloted by an Instrument Manager through the Grid, and (III) we discuss a working implementation of the system.
1 Introduction The domain of the application is that of scientific computing for a large multidisciplinary Physics research establishment, a Synchrotron Radiation Facility [3]. The proposed architecture has been realised to satisfy certain computational needs in the third generation Synchrotron Radiation Facility Elettra.1 Specifically the deployment was for a beamline/laboratory that specialises in advanced Computed Tomography and Medical Imaging. The SciComp team of Elettra has successfully utilised the Grid Computing paradigm [9] in various occasions [1, 19]. The power of the Grid has been proved useful but it has various limitations [11] that will be discussed later in this chapter. In order to overcome certain Grid-related problems we utilise a new architecture that is only-partially Grid based. In general, the computation on the Grid is done on Worker Nodes (WN) of a Computing Element (CE). There are various “logistics” in the process that the middleware, in our case gLite [4], takes care of. It has been observed that there are quite high latencies during the job submission phase. These latencies for certain applications are not a problem but for others they are. The proposed solution carries the computation in an atypical Grid manner while maintaining other Grid features like security and I/O to storage. Elettra, like many other Synchrotrons, bases its control on a Distributed Control System (DCS) [2]. This control core manages more than 10,000 devices, including sensors, detectors, PSUs, and motors. The DCS of choice is TANGO [10]. The proposed solution assumes that a set of TANGO devices are part of the system. In order to blend TANGO with the Grid technologies we used the Instrument Element (IE) [5, 14], a middleware extension that allows instrument integration on the Grid. The IE has been utilised in a non-standard way, so it does not control an actual instrument but a software-controlling TANGO device. This has been possible due to the extensible nature of its architecture. 1
URL:http://www.elettra.trieste.it, note that the establishment also includes a fourth generation light source based on a free-electron laser, FERMI@Elettra.
Grid Computations without the Computing Element
5
In order to satisfy the specific needs of the application a suitable set of Virtualisation [21] technologies was deployed. The actual imaging algorithm and application were designed to take advantage of this setup, yielding satisfactory performance. The user interaction with the application was of good standards since the final solution is based on a modern Grid-portal, the VCR [13].
2 The Application 2.1 Beamlines and Online Processing A beamline (BL) in a Synchrotron is a kind of specialised laboratory. Many of the experiments that take place in such environments have specialised and often high computing needs. Such needs may include the design of novel algorithms, software, hardware, control, and user interaction. The SRF Elettra has 24 beamlines providing more than 105,000 h of user time to visiting and in-house scientists. A mode of operation that is in demand is that of “on-line” processing (Fig. 1), which takes place during the acquisition phase of the experiment. Due to long duration of certain experiments, like >4 h scans, this mode of operation can provide an early feedback that allows the specialist to preliminarily evaluate the correctness of the experiment’s setup.
2.2 Imaging for a Tomography Workflow The requested application aimed to serve as an improvement to an established Computed Tomography (CT) [8] workflow for a Medical Imaging beamline. An established workflow implies that the beamline is already operating and functional: there are various stages of processing (including few Grid-based [6]), other sequential other parallel, and data formats and instrumentation control is already in-place.
Fig. 1 ONLINE takes place during the experiment and ends with it, while OFFLINE starts when the dataset is complete. Where the situations permit, ONLINE is a preferred mode since it saves time
6
R. Borghes et al.
Fig. 2 Non-normalised X-ray absorption and the sinogram line that is contributing. PSGen is processing all the lines in parallel
This increased the challenge since the improvements had to be a transparent part that would integrate well and would yield advantages. An example technical challenge was the on-the-fly conversion of Lempel-Ziv-Welch [22] compressed 12-bit datasets of integers to a RAW 16-bit format. The conversion may have not been an optimal engineering solution2 but had been strictly requested so that the I/O formats would be exactly the same as prior to the introduction of our solution. This resulted in an undisturbed workflow where the data processing procedures, before and after the new system, remained the same. The mode of operation was identified from the early System Analysis stages that would be Online. Before our solution the specific process in the beamline was operating in an Offline mode. In Fig. 1, we see that this can provide a complete output dataset by the end of the experiment. Eventually, the online system that this paper is describing saves 2 h per dataset; therefore, more data can be acquired for a specific experiment. The application had to collect X-ray Absorptions data during a CT scan and process and produce the corresponding Sinogram dataset (Fig. 2). The resulted application was PSGen, a Parallel Sinogram Generator. This through the architecture described in the following sections has resulted in a rapid sub-second Absorption data processing including parallel I/O to multiple hundreds of Sinogram files, with a total I/O of approximately 5 GB per dataset; all in near-real-time.
3 The Distributed Control System TANGO is an open source object oriented control system developed mainly for controlling Accelerators and experiments. It can be used for almost any kind of hardware but deployment on a standard gLite WN may be difficult. The system is actively developed by a consortium of synchrotron radiation institutes: Alba, Desy, Elettra, ESRF, and Soleil. 2
For large scientific datasets modern formats like HDF5 should be considered.
Grid Computations without the Computing Element
7
Fig. 3 There are many ways to control and monitor the TANGO devices. But even the popular GUI based tool JIVE would not be user-friendly enough for the end-user, the beamline scientist
TANGO uses the omniORB implementation of CORBA [7] as its network protocol in order to provide network access to hardware. Hardware can range from single bits of digital input/output up to sophisticated detector systems or entire plant control systems. Hardware access is programmed in a process called a Device Server. The device server implements device classes which implement the hardware access. At runtime the device server creates devices which represent logical instances of hardware. The object model in TANGO supports methods, attributes and properties. The client-server communication model can be synchronous, asynchronous or event driven. Clients and Device Servers can be written in CCC, Java or Python. The choice for the devices of PSGen was CCC in order to achieve the optimal performance. For the online-processing application, two TANGO device servers have been developed: the Watchdog and the Worker. The main task of the Watchdog is signalling the presence of new data to the Worker device. The Worker, after receiving the signal, performs the data processing step. The system is highly configurable and provides at all stages of the processing feedback and status information. The devices have such an open and extensible architecture that can be deployed in other online-processing application with minimal effort. Example settings for such a device include the Watchdog frequency, dataset size indicator, and self-stopping parameters. In Fig. 3, a generic interaction is achieved by using JIVE, a graphical tool for viewing and modifying the TANGO db and testing devices3 .
3
The GUI tool JIVE for TANGO we are referring to, should not be confused with the homonym EU FP7 project.
8
R. Borghes et al.
4 Where Computation Takes Place The TANGO system that has been described in the previous section has requirements that the typical gLite Worker Node (WN) may not have or may be impractically difficult to be satisfied. These requirements usually are OS related like user privileges and kernel tuning, and network related like having inbound connectivity. In addition to the TANGO requirements, PSGen itself requires a hardware and software setup that is not present in a general gLite CE deployment. These include custom and cutting-edge versions of libraries and Dynamic Memory Provisioning. An additional design decision was to provide the application with a direct, fast and low latency access to the storage which at all times can be accessed securely through the GridFTP protocol [20]. The system we designed for deploying the application coupled with the TANGO devices, is hosted in the Elettra Cloud Platform. This infrastructure is based on a Virtualisation solution, XEN Hypervisor. It provides a flexible way to add, remove, upgrade, and change resources like CPUs, memory, and Network connectivity. In the future, the application may see an increase to its computational requirements (i.e. as CCDs get larger) so scalability and upgradability is necessary [16]. The system, that we named “Dynamite”, has been configured in a paravirtualised mode and the mixed PSGen-specific performance hit has been measured to be less than 4.2%. The current setup of Dynamite is a 64-bit OS (CentOS 5.4) with many upgraded libraries. The 64-bit OS permits access to the 14GB of available memory, allowing massive RAM caching and large array in-memory operations. The available CPU cores that can be dedicated are 10 but the setup can easily scale and be upgraded. The current performance satisfies the processing needs of PSGen resulting to the calculation of an approximate 15:6 106 16-bit Integers per second. An additional consideration of the system’s performance is that of the I/O. The system does not use local storage at any stage. The input and output datasets are stored in a high performance Storage Area Network (SAN). The connection between Dynamite and the front-end server of the SAN is 4Gb/s. The available storage for the specific beamline/lab is 20TB. The overall performance, including calculation and I/O, has been fast and stable enough for near-real-time computation (Fig. 4).
Fig. 4 In theory, each PSGen cycle for a given dataset should be the same. The actual numbers confirm the stability and constant performance of Dynamite, showing an insignificant variation. The initial cycles reflect cached I/O
Grid Computations without the Computing Element
9
5 The IE not for Instrument Control The Instrument Element (IE) is an advanced Grid middleware extension that enables the inclusion of Instrumentation to the Grid. The exact details of it are beyond the scope of this contribution but the reader may refer to the suitable literature for better understanding [14]. In general, it is a set of technologies that permit the easy inclusion of instruments like sensors, motors and cameras to a gLite infrastructure. The main objective is the remote operation [15, 18] and control of the instrument through a secure channel by respecting the Grid security protocols. The Instrument Element is an open source middleware implemented in Java as an Axis SOAP web service that runs in the Tomcat web server. To interface the actual instruments and sensors it uses the so-called Instrument Managers (IM). An IM is a suitably structured Java client code that communicates with the physical instrument or better, its control system. Instrument managers have to run inside the IE framework which provides a number of common services. In particular, the authentication and authorisation mechanism based on user credentials and VOMS roles accordingly to the GSI/gLite schema and a common interface to all instruments attached (Fig. 5). The architecture of the IE is extensible and the scope of it may be stretched beyond its original objectives. Specifically, the IE can be seen as an Integration Element by exploiting the abstraction of the concept instrument [17, 19]. When the concept is generalized it may include software application too; and in our case we assume as an instrument the underlying TANGO device server that manages PSGen. Thus, the full control of the PSGen application and its accompanying TANGO
Fig. 5 The Instrument Element architecture. It is an extensible and expandable model so it can be used for purposes beyond its original scope, like steering the TANGO devices that control PSGen
10
R. Borghes et al.
Fig. 6 The IE for TANGO Watchdog of PSGen can be controlled from a web portal through an automatically generated graphical panel
devices can be interfaced and provided as an IE via an Instrument Manager. At this point, the IE controls a Computing entity, the PSGen application, without accessing a traditional CE. This irregular use of the IE demonstrates a case where it may even compute or, more precisely, steer a computation. Alternative Web Services based technologies could be used as well, but other than the overhead of the new development, the main disadvantage would be the disconnection of the application from the rest of Grid infrastructure that it profits from (like the SE) (Fig. 6).
6 The User Interaction The Scientific Computing team of the IT Group of the Synchrotron Elettra, for the past years has been involved in R&D for technologies that aim to realise single sing-on and Web based access to various resources. One of the latest products is that of the VCR. The VCR was initially designed to serve as a portal for easy access to Grid resources. During a project, DORII [12], the VCR was further developed to include Workflow Management, Remote Rendering (GVid), collaboration features and other. The latest version of the VCR had to be suitably updated in order to accommodate the novel hybrid approach for applications that was described in the previous
Grid Computations without the Computing Element
11
Fig. 7 The end-user, the beamline scientist, interacts with the application through a web browser in a simple manner
chapters. An updated Application Manager allowed an easy integration of the TANGO subsystem and rapid application deployment. PSGen, at this point a VCR Application, can be accessed by the users that have access to portal, taking into account most of the fine Grid-related security control, like Certificates, VOs, and the VCR-side TAGS4 . The PSGen user can start the application at the beginning of the experiment and receive feedback at any stage; even after logoff and re-login. The input and output datasets are stored in Storage Elements (SE) that can be browsed from the web portal. In addition to the above, the user may start other Applications that make full use of the Grid in standard manner. These may include online and offline, single, parametric, MPI, and interactive and any combination permitted from the gLite, the Common Library, and the rest of the technologies developed during the DORII project. Finally, the scientists may use features like the logbook for note-taking purposes while using PSGen. Additional work is in progress in order to allow richer GUI elements that will permit richer interaction with the remote application (Fig. 7).
7 Conclusions The Grid can be useful in many scenarios but there are cases where its limitations are obvious. One such limitation is constituted by the various latencies. They usually occur during the job submission phase. There is a large family of applications where these latencies are not a problem but there are other cases that can be limited from these. The presented work was for an application with near-real-time requirements for on-line processing in a Tomography lab of a Synchrotron. The presented architecture and the related R&D demonstrated a hybrid approach where storage
4
TAGS in the VCR domain can be seen as flags that allow a flexible association of a specific user or user-group with a set of applications.
12
R. Borghes et al.
(I/O on the SE), interaction (VCR portal) and security is done on the Grid while the computation is done in a non-CE based infrastructure controlled by an IE. Such a hybrid approach is novel in the field and in the future may be extended towards other directions.
References 1. R. Pugliese, M. Prica, G. Kourousias, A. Del Linz, A. Curri, Integrating Instruments in the Grid for On-line and Off-line Processing in a Synchrotron Radiation Facility, COMPUTATIONAL METHODS IN SCIENCE AND TECHNOLOGY 15 (2009), no. 1, 21–30. 2. A. Butkovski˘ı, Distributed control systems, Elsevier Publishing Company, 1969. 3. E. Koch, G. Marr, T.Sasaki, H Winick, Handbook on synchrotron radiation, (1983). 4. E. Laure, S. Fisher, A. Frohner, C. Grandi, P. Kunszt, A. Krenek, O. Mulmo,F. Pacini, F. Prelz, J. White, and others, Programming the grid with glite, Computational Methods in Science and Technology 12 (2006), no. 1, 33–45. 5. E. Frizziero, M. Gulmini, F. Lelli, G. Maron, A. Oh, S. Orlando, A. Petrucci,S. Squizzato, S. Traldi, Instrument Element: a new Grid component that enables the control of remote instrumentation, doi:10.1109/CCGRID.2006.146), 2006. 6. F. Brun, G. Kourousias, D. Dreossi, L. Mancini, An improved method for ring artifacts removing in reconstructed tomographic images, IFMBE World Congress on Medical Physics and Biomedical Engineering, vol. 24, Springer, 2009, pp. 926–929. 7. M. Henning, The rise and fall of CORBA, Queue 4 (2006), no. 5, 34. 8. G.T. Herman, Fundamentals of computerized tomography: Image reconstruction from projections, Springer Verlag, 2009. 9. I. Foster, C. Kesselman, The grid: blueprint for a new computing infrastructure, Morgan Kaufmann, 2004. 10. J.M. Chaize, A. G¨otz, W.D. Klotz, J. Meyer, M. Perez, E. Taurel, Tango-an Object oriented control system based on CORBA, Proceedings of the International Conference on Accelerator and Large Experimental Physics Control Systems (ICALEPS99), 2000. 11. Lovro Ilijaˇsi´c, Computational Grids as Complex Networks, Master’s thesis, Scuola di Dottorato in Scienza e Alta Tecnologia, 2010. ´ David, G. Barcel´o,I. Coterillo, F. Davoli, P. Gamba, R. Keller, 12. M. Pł´ociennik, D. Adami, A. D. Kranzlm¨uller, I. Labotis, DORII–Deployment of Remote Instrumentation Infrastructure, Relation 10 (2009), no. 1.109, 800. 13. M. Prica, R. Pugliese, A. Del Linz, G. Kourousias, A. Curri, D. Favretto, F. Bonaccorso, An advanced web portal for accessing grid resources with virtual collaboration features, 5th EGEE User Forum (Uppsala, Sweden), EGEE User Forum, April 2010. 14. M. Prica, R. Pugliese, A. Del Linz, A. Curri, Adapting the instrument element to support a remote instrumentation infrastructure, Remote Instrumentation and Virtual Laboratories: Service Architecture and Networking (2010), 11. 15. M. Prica, R. Pugliese, C. Scafuri, L.D. Cano, F. Asnicar, A. Curri, Remote operations of an accelerator using the grid, Grid Enabled Remote Instrumentation (2009), 527–536. 16. R. Pugliese, G. Kourousias, A. Curri, A quantitative method for the projective approximation of computational requirements, Tech. Report 090500:01, Sincrotrone Elettra, 2009. 17. R. Pugliese, G. Kourousias, M. Prica, A. Del Linz, A. Curri, An infrastructure for the integration of geoscience instruments and sensors on the grid, Geophysical Research Abstracts, vol. 11, EGU, 2009. 18. R. Pugliese, M. Prica, Remote operations of an accelerator using the grid, International Conference on Accelerator and Large Experimental Physics Control Systems (ICALEPCS), WOPA02, 2007, pp. 303–306.
Grid Computations without the Computing Element
13
19. R. Pugliese, M. Prica, G. Kourousias, A. Del Linz, A. Curri, The grid as a software application provider in a synchrotron radiation facility, Remote Instrumentation Services on the eInfrastructure, vol. X, Springer, 2009. 20. I.J. Taylor, From P2P to web services and grids: peers in a client/server world, Springer-Verlag New York Inc, 2005. 21. T.L. Borden, J.P. Hennessy, J.W. Rymarczyk, Multiple operating systems on one processor complex, IBM Systems Journal 28 (1989), no. 1, 104–123. 22. T.A. Welch, A technique for high performance data compression, IEEE Computer 17 (1984), no. 6.
Performance Evaluation of the DORII Instrument Element Data Transfer Capabilities Luca Berruti, Franco Davoli, Stefano Vignola, and Sandro Zappatore
Abstract The possibility of accessing and managing remote laboratories by exploiting the facilities offered by the Internet represents a challenging issue. To this purpose, many research projects have been funded and different hardware/software solutions have been proposed. A possible approach to the problem may consist of adopting a grid-based platform to expose and share the resources present in a group of laboratories in a uniform way. Recent projects have proposed a complex and complete architecture that enables Internet users to remotely control instrumentation and perform experiments. According to such architecture, the instrument element (IE) represents the basic abstraction that allows physical instrumentation to be exposed as a web service and to become a manageable resource in a grid environment. This chapter aims at evaluating the performance of the transfer of data generated by the instruments from the IE to a multiplicity of users. The evaluation is performed under different traffic load values, generated by the acquisition of measurements, with different numbers of clients connected to the system, and by means of two different data transfer methodologies supported by the IE. The results related to a large number of tests are presented and discussed.
1 Introduction Remotely controlling a laboratory and the related instruments and devices, sending commands, and acquiring measurements are not new activities in the instrumentation and measurement scenario – they have been performed in a whole range of different applications. However, the goals of recent remote instrumentation service
L. Berruti • F. Davoli () • S. Vignola • S. Zappatore DIST-University of Genoa/CNIT, University of Genoa Research Unit, Genoa, Italy e-mail:
[email protected];
[email protected];
[email protected];
[email protected] F. Davoli et al. (eds.), Remote Instrumentation for eScience and Related Aspects, DOI 10.1007/978-1-4614-0508-5 2, © Springer Science+Business Media, LLC 2012
15
16
L. Berruti et al.
paradigms, such as those proposed by GRIDCC [1], RINGrid [2], and DORII [3], three recent projects funded by the European Community, are more ambitious. Indeed, they aim at: • Providing a set of standard capabilities to perform whatever functionality may be required • Constructing suitable abstractions of the remote instrumentation, in order to make it visible as a manageable resource • Presenting the user standard interfaces, which allow browsing the “distributed laboratory space,” choose different pieces of equipment, configure their interconnection, orchestrate experiment executions, and collect, process and analyze the results – all by providing also built-in security services In order to accomplish such tasks to a full extent, laboratories, devices, and instruments should be exposed as a set of web services, whose interfaces may be compliant with some grid architecture, much in the same way as computing and storage devices are. In this way, they would take advantage of: (a) isolation from and relative independence of the underlying networking infrastructure providing connectivity, (b) tools for resource allocation and management, (c) standard user interfaces, and (d) nontrivial quality of service (QoS) control. All the previously mentioned concerns have been the subject of recent interest, strictly connected to the issue of remote instrumentation services (RIS). The widespread diffusion of such services can foster the use of sophisticated and costly scientific equipment, and of eScience applications. Furthermore, it should be highlighted that this paradigm does not only apply to large-scale laboratories and devices, but it can be fruitfully employed even with smaller and relatively widespread measurement instrumentation, as often adopted in engineering applications. The interest in this field is witnessed by a number of papers (see, e.g., [4–13]) available in the literature and dealing with these topics. A possible approach for controlling and managing remote laboratories and instrumentation may consist of employing the grid-based architecture initially developed with the GRIDCC project. According to this architecture, the main role is played by a software component called the instrument element (IE). It represents the basic abstraction that allows physical instrumentation to be exposed as a web service and to become a manageable resource in a grid environment. Through the presence of one or more instrument managers (IMs), the IE actually interfaces the instrumentation, both hiding the details of the drivers from users, and possibly embedding popular proprietary solutions (e.g., LabView), to provide a unified standard interface. After initial implementations, the IE has been redesigned within the DORII project to simplify and better organize its structure and enhance its performance. A similar enhancement has been undertaken for another basic component of remote instrumentation services, namely, the VCR (virtual control room), a web portal that provides user access to the distributed environment. The present chapter aims at evaluating the performance of the transfer of data generated by the instruments from the IE to a multiplicity of users, under two
Performance Evaluation of the DORII Instrument Element Data Transfer Capabilities
17
different data transfer methodologies supported by the IE. To this purpose, we emulate the generation of measurement data from an instrument as an array of variables, under different load values and with different numbers of subscribing clients. Java message service (JMS) is the publish/subscribe mechanism adopted, and the generation of data is effected by using JMeter, a Java application designed to measure performance. There are basically two ways in which the combination of these architectural elements can be used to interact with the physical instruments. These are: (a) configuring and setting parameters and (b) collecting measurement data from the instruments. With respect to the second point, possible choices are polling the instrument from the client, by traversing the whole web service protocol stack (which might hinder performance) or using a publish/subscribe approach, whereby clients which subscribed for a certain topic are notified asynchronously by means of a messaging system. This chapter is structured as follows. Section 2 illustrates the overall DORII architecture, pointing out the role played by its principal components. Section 3 describes the experimental setup employed to carry out the measurement campaigns devoted to evaluate the performance of the data collection from the IE. The results obtained are presented and discussed in Sect. 4. Finally, in the last section conclusions are drawn.
2 Overall DORII Architecture The overall DORII architecture is diagrammatically depicted in Fig. 1. The architecture stems from the grid-based platform adopted within the GRIDCC project and represents the natural evolution of the latter. Though the GRIDCC platform may include components (called “elements”) of different kinds, in the context of distributed remotely controlled laboratories, the main rules are played by the instrument elements (IEs) and the virtual control room (VCR). The latter constitutes the actual interface exploited by the user to carry out any sort of actions and/or measurements that the laboratory, handled by the IE, is able to perform. The IE consists of a web service module embedded in a grid middleware, specifically gLite. The IE actually controls all the devices/instruments present in the laboratory through a set of instrument managers (IMs). The IE runs in the Apache Tomcat container as an Axis web service and it includes two ancillary sub-components (the access control manager – ACM – and the information and monitor service – IMS), plus a set of IMs that actually handle the instrumentation. The grid framework also provides another data exchange protocol (GridFTP), which allows IMs to send their outputs (in a batch fashion) to other grid elements (e.g., storage elements, SEs). The main task of an IM is to drive and handle a specific class of devices or instruments: obviously, each class of devices requires a specific IM implementation. Under this perspective, an oscilloscope, independent of its manufacturer, will require an “oscilloscope IM,” as well as a spectrum analyzer
18
L. Berruti et al.
JMS - Data/Info Publishing/Alarms/Events
Web GUI
Instrument Element
Control Manager
IMS Proxy
GridFTP
Access Control Manager
Web Service – Instrumentation control
Information and Monitor Service
Instrument Manager Instrument Abstraction Layer Dynamic Link Library Instruments Access Device Driver
Storage Element
Compute Element
GridFtp
Web Service
Device
Web Service
Virtual Control Room
Web Service
Grid
Fig. 1 Overall DORII architecture
will need a “spectrum analyzer IM.” In order to handle the instrumentation of a laboratory, the IM exploits a set of application program interfaces (viz., APIs) implemented as a software library, called instrument abstraction layer (IAL). The adoption of the IAL permits us to develop an IM without considering the specific technology used to interface the real device/instrument; in other words, an IM programmer does not need to take care of the hardware specifications at the physical layer, the low level communication protocol, and so on. As regards the command and data exchange, the requesters (viz., users at the VCR end) can send SOAP requests and get SOAP responses from the IE over a HTTP connection. Upon receiving a message (i.e., a request), the IE dispatches it toward the IM referenced in the message. According to this mechanism, a possible way a user can read the current value of a certain quantity (namely, the “attribute” of a “variable”) acquired by a device/instrument consists of triggering a read operation (specifically, a getAttribute method) by means of a user request: hence, a near realtime monitoring may involve a continuous polling of the IE and, in turn, of the appropriate IM. Employing the getAttribute method is surely more suitable for asynchronous (nonpersistent) reading of an attribute, since a continuous polling may produce an overburden of the system hosting the IE/IM and an overall wastage of resources. If a continuous (though still asynchronous) monitoring is needed, the IE may take advantage of the use of a publish/subscribe mechanism, as the one provided
Performance Evaluation of the DORII Instrument Element Data Transfer Capabilities
19
by a JMS server (sometimes also called a JMS broker) in order to fast deliver data messages from the instrument managers to the users or to the VCR. According to the JMS terminology [14], the messaging system provided by a JMS server may use a managed object, called a “Topic,” to identify a specific message flow. A message producer acquires a reference to a JMS topic on the server, and sends messages to that Topic. When a message arrives, the JMS provider is responsible for notifying the presence of a new message to all consumers who have subscribed to that Topic. According to this scheme, if an attribute is declared “subscribable,” the IM internally polls the variable associated to the attribute and, only upon changes of the variable, the IM publishes the new value by means of the services offered by a JMS server “coupled” with the IE.
3 Experimental Setup The experimental setup adopted to evaluate the performance of the data collection from the IE is sketched in Fig. 2. The PC denoted as “Grid-Node” hosts the IE (ie-release2.1), its related IMs, and the JMS broker (Sun JMS Message Queue 4.3). The possible actions a user can perform, both directly and through a VCR, are mimicked by a JMeter (jakarta-jmeter-2.3.4) application [15] hosted in the PC denoted as “Client Station.” More precisely, the grid-node is a Fujitsu-Siemens workstation “Celsius 460,” running a Windows-XP sp3 operating system, and the Client Station is a Dell XPS 1,330 laptop running a Debian GNU/Linux 5.0 operating system. The two PCs are directly linked via an Ethernet interface at 100 Mbit/s. It should be highlighted that in order to effectively estimate the performance of data gathering, no real device is connected to the IMs, thus avoiding possible
Client Station
IM IE
JMeter suite tool
IM IM
100 Mbit/s Ethernet
JMS broker
Fig. 2 Experimental setup used during the performance measuring tests
Mock devices
Grid-Node
20
L. Berruti et al.
bottlenecks ascribable to the protocols and/or technologies employed to connect the IMs to the devices. Therefore, any IM sends commands and receives responses and data to/from a “mock” device that, upon receiving a data request from the IM, simply assembles a vector of doubles (8 bytes in a 32 bit x86 Intel architecture) of predefined length. A specific application of the IE/IM abstractions to physical instrumentation can be found in [12, 13]. JMeter is a pure Java application originally written to test functionalities of static and dynamic resources, such as Java objects, data base queries, Servlets, and successively expanded to evaluate the performance of a wide class of objects, including SOAP/web services. Nowadays, JMeter represents a quasistandard, wellknown tool for performance evaluation able to run in a variety of environments, to provide fast operation and precise timings, and to allow great personalization.
4 Performance Evaluation The performance of the data collection from the IE has been evaluated in two different operating contexts. According to the former operative mode (hereafter referred as “Polling Mode”), the JMeter mimics a certain number of users that continuously pool the IE, which transmits back a vector of doubles. The term “continuously” means “at the maximum speed supported by the system”; viz., upon receiving a response from the IE, the client promptly sends a new request to the IE. Obviously, the maximal frequency of user requests will depend on several factors, such as: the number of clients which are issuing the requests, the length of each answer, the computation power of the computer hosting the IE, and so on. The purpose is to estimate the average response time (viz., the time spent by a user to receive a vector of data from the IE) versus the number of users and the length of the vector of doubles, and to evaluate the overall traffic offered to the network. In the latter operative context (hereafter referred as “JMS Subscribing Mode”), the JMS broker is exploited to fast deliver the data from the IE to the users. Thus, the users have to subscribe themselves to a topic corresponding to the attribute to be monitored. Successively, any user can get the attribute directly from the topic whenever a new attribute is published by the IE. In this case our measurement campaign is aimed at evaluating the overall traffic present in the network, by varying both the number of the platform users and the frequency at which the attribute changes. It is worth noting that the frequencies of attributes’ renewal have been chosen as the inverse of the average response time estimated when the JMS broker was not active. To better detail this point, let us suppose that under certain specific conditions and with the JMS disabled (i.e., with the system operating in polling mode), the average response time estimated by JMeter is Tr and Lp is the average traffic load measured over the network. Then, the goal of our measurement campaign is to estimate the network traffic load Lj and compare it with Lp , when the JMS broker is enabled and 1=Tr is the renewal frequency of the attribute handled by the IM.
Performance Evaluation of the DORII Instrument Element Data Transfer Capabilities Mean 2500
Lower 99%
Upper 99%
Mean 1250
Lower 99%
21
Upper 99%
16000 Response Time [ms]
14000 12000 10000 8000 6000 4000 2000 0 0
5
10
15
20
25
30
35
40
45
Number of Clients
Fig. 3 Response time versus the number of clients when the “Polling Mode” is used
Figure 3 reports the behavior of response time versus the number of clients which poll the IE in order to get a data vector. The upper line (marked with triangles) refers to the case where the IE response consists of a vector of 2,500 doubles (i.e., a vector of 20,000 bytes); the lower line (marked with squares) shows the response time when the IE returns a vector of 1,250 doubles. The two dotted lines on either side of each solid line represent the borders of the confidence interval at 99%. It is worth noting that: (a) the response time increases almost linearly with the number of clients and (b) the time spent by a requester to obtain a response from the IE could be unacceptable for near real-time applications in most cases, even if only a limited number of users are accessing the laboratory controlled by the IE. Figure 4 shows the response rate (viz., the inverse of the response time, Tr / versus the number of clients (Nc / when the response of the IE consists of a vector of 2,500 and 1,250 doubles. The solid and dotted curves represent the best polynomial interpolating functions of the measured data. The interpolating functions are almost hyperboles, thus highlighting that the product Rr Nc is equal to a constant Kl , which, in turn, weakly depends on the length of the data vector returned by the IE. Observing the data in Table 1 can shed light on the factors affecting the overall IE performance. Indeed, Table 1 reports the traffic load (kbytes/s) offered to the transmission network by varying the number of clients (a) connected to the IE when the “Polling Mode” is in use or (b) picking data from the JMS broker when the “JMS Subscribing Mode” is enabled. It should be noted that, when the “Polling Mode” is active, the traffic load does not practically depend on the number of clients accessing the IE: actually, the load is almost constant and equal to about 1 MB/s, i.e., 8 Mb/s. Hence, the IE performance
22
L. Berruti et al. 1.600
Response Rate [Hz]
1.400 1.200 1.000 0.800
f(x) = 8.4 x^-1.11
0.600 0.400 f(x) = 3.65 x^-1.08
0.200 0.000 0
5
10
15
20
25
30
35
40
45
Number of Clients
Fig. 4 Response rate versus the number of clients in the “Polling Mode” Table 1 Average network traffic load (kbytes/s) under different operative conditions
Number of clients Polling 1250 JMS 1250 Polling 2500 JMS 2500
5 1; 214 199 1; 137 165
10 1; 063 161 1; 027 154
20 1; 003 150 1; 000 151
40 971 145 966 149
is not surely limited by the network capacity (100 Mb/s during our tests), but mainly by the IE’s specific implementation, the computational capabilities of the grid node hosting the IE, and partially by the JMeter. Furthermore, Table 1 presents the traffic load when the JMS broker is exploited by the IE and the IM (of the IE) internally polls the device at a sample frequency equal to the average response rate achievable when the “Polling Mode” is adopted. To better clarify this point, in the “Polling Mode” when five stations are active and the IE produces a response vector of 2,500 doubles (assumed to be the payload of a packet at the network layer), the average response rate amounts to 0.658 vectors/s (corresponding to an average response time of 1,520 ms, see Fig. 3) and the total traffic offered to the network is about 1,137 kB/s (see Table 1). On the contrary, in the “JMS Subscribing Mode,” when five stations contemporarily read data vectors (of 2,500 doubles) published, via the JMS broker, by the IM every 1.52 s, the total traffic load amounts to 165 kB/s. Analyzing Table 1 highlights how the adoption of the “JMS Subscribing Mode” permits us to scale down the traffic load by a factor of about six; alternatively, keeping the traffic load fixed, the IM can sample the device/instrument about six times faster than the case when the “Polling Mode” is adopted. The better performance achievable by the “JMS Subscribing Mode” can be ascribed to two main factors. The first and the most relevant one depends on the different data
Performance Evaluation of the DORII Instrument Element Data Transfer Capabilities
23
structures employed by the IE in polling mode and the JMS broker to dispatch data to the clients. As a matter of fact, the IE polling mode exploits the data structures provided by XML/SOAP, which are about 6.7 and 5.7 times more “space”-consuming than the data structures adopted by the JMS broker in the cases of 1,250 and 2,500 doubles, respectively. The second factor, which becomes more evident when a higher quantity of data is managed (viz., in the case of 2,500 doubles), likely depends upon the more efficient implementation of the JMS broker with respect to that of the IE in polling mode. In the case of 2,500 doubles, the data in Table 1 show a ratio of more than 6.48 between network traffic loads produced by the “Polling Mode” and the “JMS Subscribing Mode,” while the ratio between the memory space used by the data structures is about 5.7. In any case, in many situations the benefits resulting from the adoption of a JMS broker may significantly improve the realism perceived by users while handling remote instrumentation.
5 Conclusion In this chapter, we have evaluated the data gathering performance of the IE, the main software component of the GRIDCC/DORII architecture devoted to remotely control the resources present in real laboratories. During all the tests, the JMeter tool suite has been employed to estimate the time spent by a user to get a measure under the two different operational modes provided by the IE. The measurement campaign revealed that the “Polling Mode” may be often inadequate to get long data arrays from instrumentation, even if only a limited number of users access the laboratory. This mode seems to be more suitable for obtaining atomic, asynchronous data. On the contrary, in case the user requires a continuous monitoring of a data array (e.g., an oscilloscope trace), the “JMS Subscribing Mode” is surely effective, as it allows reducing the total traffic load while increasing the reactivity of the entire platform that manages the remote laboratory. Specifically, the performance tests highlighted that the “JMS Subscribing Mode” permits us to scale down the traffic load by a factor of about six and, conversely, keeping the traffic offered to the network fixed, the user can require the IE to interrogate the device/instrument about six times faster than in the case when the “Polling Mode” is in use. Future work will be aimed at evaluating the possible use of a JMS broker to gather data from a multitude of small sensor nodes that can be seen as a sort of distributed probe of a complex instrument, whose computation capabilities are inside the IE. Acknowledgments This work was supported by the European Commission, Information Society, ´ and Media DG, Unit F3 ‘GEANT & e-Infrastructure’, in the framework of the DORII FP7 Project (contract no. 213110).
24
L. Berruti et al.
References 1. 2. 3. 4.
GRIDCC project website, http://www.gridcc.org. RINGrid project website, http://www.ringrid.eu. DORII project website, http://www.dorii.eu. V.J. Harward et al., “The iLab shared architecture: A Web Services infrastructure to build communities of Internet accessible laboratories”, Proc. IEEE, vol. 96, no. 6, pp. 931–950, June 2008. 5. F. Davoli, N. Meyer, R. Pugliese, S. Zappatore, Eds., Grid-Enabled Remote Instrumentation, Springer, New York, NY, 2008; ISBN 978-0-387-09662-9. 6. F. Lelli, E. Frizziero, M. Gulmini, G. Maron, S. Orlando, A. Petrucci, S. Squizzato, “The many faces of the integration of instruments and the grid”, International Journal of Web and Grid Services, vol. 3, no. 3, 2007, pp. 239–266. 7. F. Davoli, S. Palazzo, S. Zappatore, Eds., Distributed Cooperative Laboratories: Networking, Instrumentation, and Measurements, Springer, New York, NY, 2006; ISBN 0-387-29811-8. 8. I. Foster, “Service-oriented science”, Science Mag., vol. 308, no. 5723, May 2005, pp. 814–817. 9. D.F. McMullen, R. Bramley, K. Chiu, H. Davis, T. Devadithya, J.C. Huffman, K. Huffman, T. Reichherzer, “The Common Instrument Middleware Architecture”, in F. Davoli, N. Meyer, R. Pugliese, S. Zappatore, Eds., Grid Enabled Remote Instrumentation, Springer, New York, NY, 2009, pp. 393–407; ISBN: 978-0-387-09662-9. 10. The RINGrid project team, “Remote Instrumentation Whitepaper”, available at http://www. ringrid.eu. 11. L. Berruti, F. Davoli, G. Massei, A. Scarpiello, S. Zappatore, “Remote laboratory experiments in a Virtual Immersive Learning environment”, Advances in Multimedia, vol. 2008 (2008), Article ID 426981, 11 pages, doi:10.1155/2008/426981. 12. L. Berruti, F. Davoli, M. Perrando, S. Vignola, S. Zappatore, “Engineering applications on the eInfrastructure: The case of telecommunication measurement instrumentation”, Computational Methods in Science and Technology, Special Issue on “Instrumentation for e-Science”, vol. 15, no. 1, pp. 41–48, 2009. 13. L. Caviglione, L. Berruti, F. Davoli, M. Polizzi, S. Vignola, S. Zappatore, “On the integration of telecommunication measurement devices within the framework of an instrumentation grid”, in F. Davoli, N. Meyer, R. Pugliese, S. Zappatore, Eds., Grid Enabled Remote Instrumentation, Springer, New York, NY, 2009, pp. 282–300; ISBN: 978-0-387-09662-9. 14. K. Haase, “JMS Tutorial”, available at http://download.oracle.com/docs/cd/E17477 01/javaee/ 1.3/jms/tutorial/1 3 1-fcs/doc/jms tutorialTOC.html. 15. “Apache JMeter”, The Apache Jakarta Project, available at http://jakarta.apache.org/jmeter/.
The Green Grid’5000: Instrumenting and Using a Grid with Energy Sensors Marcos Dias de Assunc¸a˜ o, Jean-Patrick Gelas, Laurent Lef`evre, and Anne-C´ecile Orgerie
Abstract Targeting mostly application performance, distributed systems have constantly increased in size and computing power of their resources. The power supply requirements of these systems have increased in a similar fashion, which has raised concerns about the energy they consume. This paper presents the Green Grid’5000 approach1 , a large-scale energy-sensing infrastructure with software components that allow users to precisely measure and understand the energy usage of their system. It also discusses a set of use-cases describing how an energy instrumented platform can be utilised by various categories of users, including administrators, Grid component designers, and distributed application end-users.
1 Introduction Large-scale distributed systems have become essential to assist scientists and practitioners in addressing a wide range of challenges, from DNA sequence analysis to economic forecasting. Driven mostly by application performance, these systems have constantly increased in size and computing power of their resources. The power supply requirements of these systems have increased in a similar fashion, which has raised concerns about the electrical power they consume and fostered research on improving their efficiency.
1
Some experiments of this paper were performed on the Grid’5000 platform, an initiative from the French Ministry of Research through the ACI GRID incentive action, INRIA, CNRS, and RENATER and other contributing partners (http://www.grid5000.fr).
M.D. de Assunc¸a˜ o () • J.-P. Gelas • L. Lef`evre • A.-C. Orgerie ´ INRIA, LIP Laboratory (UMR CNRS, ENS, INRIA, UCB), Ecole Normale Sup´erieure of Lyon, University of Lyon – 46 all´ee d’Italie 69364, Lyon Cedex 07 – France e-mail: marcos.dias.de.assuncao; jean-patrick.gelas; laurent.lefevre;
[email protected] F. Davoli et al. (eds.), Remote Instrumentation for eScience and Related Aspects, DOI 10.1007/978-1-4614-0508-5 3, © Springer Science+Business Media, LLC 2012
25
M.D. de Assunc¸a˜ o et al.
26
A range of techniques can be utilised for making computing infrastructures more energy efficient, including better cooling technologies, temperature-aware scheduling [8], Dynamic Voltage and Frequency Scaling (DVFS) [11], and resource virtualisation [3]. To benefit from these techniques and devise mechanisms for curbing the energy consumed by large-scale distributed systems [5], it is important to instrument the hardware infrastructure. The instrumentation can provide application and system designers with the information they need to understand the energy consumed by their applications and design more energy efficient mechanisms and policies for resource management. However, monitoring large-scale distributed systems such as Grids remains a challenge. Although various solutions have been proposed [2, 6, 12, 13] for monitoring the usage of resources (e.g. CPU, storage and network) to improve the performance of applications, only a few systems monitor the energy usage of infrastructures [9, 10]. This chapter presents the Green Grid’5000 approach, a large-scale energysensing infrastructure with software components that allow users to precisely measure and understand the energy usage of their system. When architecting the energy-sensing infrastructure and deciding on the required data management techniques, the following goals were considered: • • • • •
Increase users’ awareness on the energy consumed by their applications Improve the design of user applications Obtain data to advance Grid middleware Help managers in implementing power management policies Create an energy-data repository for further studies
The rest of this chapter is organised as follows. Section 2 presents the deployed energy sensing hardware infrastructure, whereas Sect. 3 describes the design of software components. Section 4 presents some concrete use-cases benefiting from the Green Grid’5000 solutions. Conclusion and future work are presented in Sect. 5.
2 Instrumenting a Distributed Infrastructure with Energy Sensors The Grid’5000 platform [1], depicted in Fig. 1, is an experimental testbed for research in distributed computing which offers 5000 processors geographically distributed across 9 sites in France (i.e., Bordeaux, Grenoble, Lille, Lyon, Nancy, Nice, Orsay, Rennes, and Toulouse), all linked by dedicated high-speed networks. This platform can be defined as a highly reconfigurable, controllable and monitorable experimental Grid. Its utilisation is specific: each user can reserve a number of nodes on which she deploys a system image; hence, the node is entirely dedicated to the user during her reservation. The Resource Management System (RMS) used by Grid’5000 is called OAR [4]. OAR is an open-source batch scheduler that provides a simple and flexible interface for reserving and using cluster
The Green Grid’5000: Instrumenting and Using a Grid with Energy Sensors
27
Fig. 1 The Grid’5000 infrastructure in France
resources.2 Users can submit best-effort jobs, which are started by OAR when resources are available, and advance reservations where users specify the desired start time of their requests. Figure 2 illustrates one of the interfaces provided by the Grid’5000 API, showing the status of the platform. The API also gives users access to information about each node, such as CPU, memory, network, disk and swap usages. However, to obtain this information a daemon must run on each reserved node. As running this daemon is optional because it can interfere with user experiments, the information may not be available for all reservation requests. Although Grid’5000 is different from a production infrastructure, the concerns surrounding its energy consumption are similar to those of production environments. Therefore, solutions proposed to tackle challenges in Grid’5000 can fit both experimental and production infrastructures. The energy-sensing infrastructure3 has been deployed on three Grid’5000 sites (i.e. Lyon, Grenoble, and Toulouse) monitoring in total 160 computing and networking resources. The infrastructure comprises a large set of wattmeters manufactured by OMEGAWATT.4 Each wattmeter is connected to a data collector via a serial link. A dedicated server stores the gathered data and generates live graphs of energy usage. In the Lyon site, where the whole infrastructure is monitored, the wattmeters periodically measure the energy consumed by 135 nodes and a router. Four Omegawatt boxes where required: one 48-port box, two 42-port boxes, and one 8-port box. Each box works as a multi-plug measurement device. For example, in the case of the 8-port box, 8 physical nodes can be plugged into it and the energy data is transmitted to a data collector (i.e. a remote server) via a serial link
2
http://oar.imag.fr/index.html. The infrastructure is financially supported by the INRIA ARC GREEN-NET initiative and the ALADDIN/Grid’5000 consortium. 4 http://www.omegawatt.fr/gb/index.php. 3
28
M.D. de Assunc¸a˜ o et al.
Fig. 2 Example of information provided by Grid’5000 API
(i.e. RS232). Hence, in this site, the energy data-collector server is connected to four boxes through four different serial links. As the energy data collector does not have four serial ports, it uses USB-RS232 converters. Another problem faced with serial links, was that the node racks are not next to each other, but in front of each other. Thus, two boxes are connected to the energy data collector via 6-meter long USB cables; cables must go under the floor. Although USB cables have a theoretical limit of 5 m, above which distance they do not operate well, we have not observed failures during the 10 months that this infrastructure is in operation.
The Green Grid’5000: Instrumenting and Using a Grid with Energy Sensors
29
On the data collector, a daemon for each box is responsible for collecting the energy data. At each second, this daemon sends a data request in a hexadecimal code with a CRC to the box to ask for the energy consumption value of all the resources monitored by that box. The response is a hexadecimal code with a CRC in the form of “packets”, each packet containing the instantaneous power consumption of six nodes. The software responsible for communicating with the wattmeters for obtaining the data was not provided with the boxes. We had to design it by ourselves for collecting at every second the measurements of 135 nodes including CRC checking, hexadecimal decoding, and storage of these values. The data gathering process should not exceed one second. As the goal of instrumenting the Grid infrastructure was to provide energy consumption information to different groups of users and system administrators, several data management techniques had to be applied. For example, the energy data is stored in three different ways: • Raw data: that includes a timestamp indicating when a measurement was performed (there is one log file per resource). • Last values: one file (in memory file systems) per resource and the value is overwritten at each second. • Round-Robin Databases5 : one file per monitored node. The data collection and storage process for the Lyon site is illustrated in Fig. 3. The usage of each type of storage for the different categories of users is detailed in the following sections. As each category of user has different requirements, the frequency of measurement is important and must be calibrated depending on what one wants to observe. For administrators that want to observe global energy trends, a few measurements per hour or per day can suffice. Users who want to understand usage and applications impact on energy consumption, demand several measurements per minute. For middleware designers who want to understand peak-usage phenomena (during a system deployment phase for example), multiple measurements per minute are mandatory. Moreover, in this case the data must be stored to allow one to evaluate trends and patterns that can aid the improvement of middleware tools. In the Green Grid’5000 context, one live measurement (in Watts) is performed each second for each monitored resource with a precision of 0:125 W. These measurements are displayed lively and stored in specific repositories. One year of energy logs for a 150-node platform corresponds to approximately 70 GB of data. These energy logs can be correlated with the reservation logs (provided by OAR) and the resource usage logs provided by the Grid’5000 API in order to have a more complete view on the energy usage.
5
RRDtool is the OpenSource industry standard, high performance data logging and graphing system for time series data (http://oss.oetiker.ch/rrdtool/).
30
M.D. de Assunc¸a˜ o et al.
Fig. 3 Energy data collector deployed at the Lyon site
3 Displaying Information on Energy Consumption Measuring energy consumption and storing the data are only part of the process when the goal is to reduce the amount of electricity utilised by an infrastructure. Ideally, the obtained information on energy consumption should drive optimisations in the resource management systems; inform users of the cost of their applications in terms of electricity spent; and possibly offer means to raise users’ awareness and adapt their behaviour so they can make more energy-aware choices when executing their applications. In addition, the information should be presented to users in a way they can measure the impact of their choices in terms of energy consumption when they run their applications differently from the manner they are used. Towards achieving these goals, we have developed a set of tools for visualising the data collected from instrumenting the Grid platform. This section describes a graphical interface, termed as ShowWatts, for displaying the measurements in near real-time, and a list of Web pages and graphs for viewing historical data on energy consumption. The first interface allows users to view the instant energy consumption of the whole platform and check the immediate impact of executing an application on a set of resources; whereas the Web pages and graphs enable the analysis of energy consumption over larger periods and the identification of patterns and application behaviours.
The Green Grid’5000: Instrumenting and Using a Grid with Energy Sensors
31
Fig. 4 Screenshot of ShowWatts graphical interface
3.1 ShowWatts: Live Display ShowWatts comprises a Java Swing application for displaying the energy consumption of the instrumented platform in nearly real-time (see Fig. 4), and a module responsible for aggregating the energy consumption information and transferring it to the node where it is displayed. It works on the client-server model: a daemon is launched on the server side and another is launched by the client; the client daemon uses the “last values” described in Sect. 2. These two processes communicate through an SSH tunnel. The server side part is optimised to minimise the data volume sent: only the values that differ from the previous transfer are sent. From a system administrator’s point of view, this interface can be used to monitor the whole platform or the consumption of specific resources. The administrator can select the resources in which she is interested and view their share of the overall energy consumption. In addition, it is possible to view other statistics such as a rough amount of money spent with electricity. Additionally, the application is customisable and can be changed to display information on other platforms. This is achieved by implementing the module that aggregates the consumption information and changing two XML files; one that describes general configuration and display preferences and another that specifies the monitored resources. For demonstrative purposes, we have implemented a reservation scheduler in python, which shows the impact in energy consumption when switching machines off and on, and aggregating reservation requests.
32
M.D. de Assunc¸a˜ o et al.
Fig. 5 Web interface for energy usage visualisation
3.2 Portable Web-Based Visualisation As mentioner earlier, in addition to displaying the instant energy consumption, visualisation tools must provide means whereby users and system administrators can analyse past consumption. By visualising the energy consumed by individual resources or groups of resources over larger periods, administrators and users can better know the behaviour of their platform, design more energy-efficient allocation policies, and investigate possible system optimisations. We have developed a set of scripts that automatically generate Web pages and graphs for displaying the energy consumption of individual resources and groups of resources. Figure 5 shows an example of Web page containing a list of graphs with measurements taken from a Grid’5000 site. These graphs are updated every minute so that the user has an up-to-date view of the platform in terms of energy consumption.
3.3 Collecting and Providing Energy Logs Measuring the power consumption of a Grid is the first step to understand the energy usage of the platform, while the following step is to collect and store this data and give users access to it. As discussed in Sect. 2, to collect the energy measurements, we have developed software to interact with the wattmeters and to store the received information in different formats: RRD databases and text files.
The Green Grid’5000: Instrumenting and Using a Grid with Energy Sensors
33
RRD databases offer the advantage of having a fixed size. They use average values for generating graphs of long periods. They are used to have an overview of the energy platform. It provides different views of the node’s energy consumption (i.e. per hour, per day, per week, per month, and per year). Each view presents maximum, minimum, and average values. The current value per node (in Watts) is also displayed and updated every second (by an ajax script) as well as the global value for the whole site. These tools are particularly suitable for administrators. They provide a quick overview of the whole platform with an in-depth access to the most recent values for each node. An example of usage for this kind of energy log is given in Sect. 4.4. These aggregated views, however, are not precise enough to understand the link between running an application on a node and the energy it consumes. Hence, text files are used to store each measurement (i.e. for each node at each second with a timestamp). These text files are archived and compressed each week in order to reduce the disk space required; this allows us to keep all the logs. While collecting and storing energy logs is important, the main goal is to provide these logs to users, so that they can see the impact of their applications on energy consumption of the platform. This may increase users’ awareness about the energy consumed by large-scale infrastructures. Therefore, the monitoring Website includes log-on-demand features. As shown in Fig. 6, the Web portal entails a Web page where a user queries the energy consumption of servers by specifying the period, the time increment (i.e. value in seconds between two measurements) and the nodes. These tools are specially designed for the users and the middleware developers who want to precisely profile their applications in terms of energy usage and to correlate the energy data with resource (e.g. CPU, memory, disk, network, etc.) utilisation to view the impact of their programming decisions on energy consumption. Examples of such applications and utilisations of these tools are given in Sects. 4.1, 4.2 and 4.3. The architecture responsible for collecting, storing and exposing the logs is summarised in Fig. 7. The logs and graphs are available for all Grid’5000 users and administrators. Figure 8 presents the typical result of a log request. First, the global information of the request is given, such as the global consumption of the requested nodes during the given time frame. Moreover, for each node, the energy graph is provided to show the energy profile of the user’s request. For each request, we provide a directory (with a permanent address) that contains: • A log file for each requested node with all the collected measurements during the time period requested (with the requested increment between two measurements). • An SVG graph for each requested node plotting its energy profile. • A log file with the global consumption of each requested node during the requested time period and the total consumption of the requested nodes. • A .tar.gz file containing all the files above.
Fig. 6 Web Interface for energy-log requests
34 M.D. de Assunc¸a˜ o et al.
The Green Grid’5000: Instrumenting and Using a Grid with Energy Sensors
35
Fig. 7 Overall architecture to collect, store and expose the energy data
Fig. 8 Web page displaying the result of an energy-log request
4 Green Grid’5000 Usage Examples This section presents some examples on how the energy consumption data can be utilised by different types of users.
36
M.D. de Assunc¸a˜ o et al.
Fig. 9 Energy consumption during typical node applications
4.1 Energy Profiling of Applications The energy monitoring architecture is often utilised by Grid’5000 users to study the energy profile of their applications. Figure 9 illustrates the behaviour of the typical life-cycle of a user application on a Grid’5000 node (Sun Fire V20z: 2 CPUs of 2.4GHz). The different phases of the application correspond to: • The boot of the node that lasts for approximately 120 s and is characterised by power consumption peaks. • Then, an idle period during which the node still has a high energy consumption even if it does nothing (around 240 W). • An hdparm6 experiment which represents intensive disk-accesses (243 W on average). • An iperf 7 experiment which makes intensive use of the network (using UDP, it consumes 263 W on average but also uses CPU). • A cpuburn8 which fully loads a CPU (267 W on average). • A stress9 which fully uses all the CPUs and also performs I/O operations (277 W on average). • A halt during which a small power consumption peak can be observed. • The node is off and still consumes around 10 W.
6 hdparm is a command line utility for Linux to set and view the IDE hard disks hardware parameters. 7 Iperf is a commonly-used network tool that can create TCP and UDP data streams to measure TCP and UDP bandwidth performance. 8 cpuburn is a tool designed to heavily load CPU chips in order to test their reliability. 9 stress is a tool to impose load on and stress test systems.
The Green Grid’5000: Instrumenting and Using a Grid with Energy Sensors
37
Fig. 10 Basic MPI application scenario
The logs on demand tool has been used to create Fig. 9. As one can observe, the tool provides detailed information and a summary; the experiment has lasted 474 s, and during that period the node consumed 111,296.34 Joules. Energy profiling of user applications can greatly increase their awareness as well as help programmers to improve the energy efficiency of their applications.
4.2 Improve Distributed Applications As end-users are able to profile the energy usage of their applications, it is possible to design more efficient distributed applications by applying some “green” programming concepts. As an illustrative scenario, a programmer designs an MPI client-server application as depicted in Fig. 10, where 3 processes run concurrently: • One server is assigning tasks to two clients. • Client 1 is computing small tasks (S Tasks). • Client 2 is computing XL Tasks (1 XL task D 2 S tasks). At the first attempt, the programmer designs a basic application where the server assigns tasks to clients in a synchronous way. This application is neither optimal in terms of performance nor energy efficient. As tasks are heterogeneous, some periods of inactivity are observed on client 1 and energy is wasted (Fig. 11). The client spends long periods of inactivity with an idle consumption of 179 W. By using the Green Grid’5000 infrastructure, the end-user is able to detect these idle periods and re-design her application. The programmer then decides to use asynchronous MPI communication. As soon as the client has finished to compute its tasks, it sends the results back to the server, which is then able to send new tasks to the client. We can observe that this more aggressive approach is better in terms of energy and performance as it reduces
38
M.D. de Assunc¸a˜ o et al.
Fig. 11 Power consumption of client 1 during the non-energy efficient MPI application
Fig. 12 Power consumption of client 1 during the energy-efficient MPI application
inactivity periods (Fig. 12). In one minute, the client can compute more tasks than in the previous example. By profiling applications, designers can explore ways to improve their energy efficiency.
4.3 Improving the Energy Efficiency of Grid Middleware As discussed beforehand, recent computer hardware and operating systems have offered a range of techniques to manage the power drawn by servers. The challenge posed to middleware designers is how to benefit from these techniques for
The Green Grid’5000: Instrumenting and Using a Grid with Energy Sensors
39
Fig. 13 Energy consumed by a server during part of an interactive reservation
making distributed systems more energy efficient; hence, reducing both their energy footprint and the costs associated with managing a Grid infrastructure. Tackling this challenge requires an understanding of how users utilise the infrastructure and how their applications behave in terms of resource usage and energy consumption. Previous study on the energy consumption of Grid’5000 has shown that substantial energy savings can be achieved if users accept to change the start or finish times of their reservations, so that they can be aggregated, creating free windows during which unused servers can be switched off [9]. Additional savings can be made when analysing the energy consumption data and crossing it with information from other system components such as the RMS or scheduler. We utilise the “interactive” advance reservations of Grid’5000 as an example to illustrate how Grid middleware can be improved when information on energy consumption of servers and resource utilisation logs are made available to middleware developers. Under an interactive reservation, a user reserves a group of servers on which she deploys an environment that contains the whole operating system customised to her application. Figure 13 shows the energy consumed by a server of the Lyon site during part of a reservation that starts at 10 a.m. on a working day. The noisy measurements obtained after 1,900 s are typical of an environmentdeployment phase during which the disk image is copied to the node that is later rebooted with the new operating system. This figure shows that during the period that precedes the deployment, the server was not utilised, hence wasting resources and energy. If mechanisms for identifying and predicting these behaviours are incorporated into middleware design, the unused servers can be switched off; hence, minimising resource wastage and consequently improving the energy efficiency of the infrastructure.
40
M.D. de Assunc¸a˜ o et al.
Fig. 14 Energy consumption and utilisation of nodes over 6 months
4.4 Administrator’s Use Case Collecting and analysing energy consumption information can aid system administrators: to evaluate the power management techniques offered by the hardware, observe the impact of different policies to curbing the energy consumption of the infrastructure, and guide capacity planning decisions. However, there is only so much that energy consumption data alone can provide. As illustrated in the example above, it is important to cross the data on energy consumption with information from other system components such as cluster schedulers – e.g. information on node failures, job arrivals, job scheduling – and information on resource utilisation – e.g. information on CPU, memory, storage, and network usage. Figure 14 presents the energy in kWh per day consumed by the Grid’5000 site in Lyon. The blue line shows the resource utilisation according to the site scheduler (i.e. OAR) [4]; the utilisation indicates the percentage of reserved nodes, and hence does not imply that CPUs, storage or network resources were used by reservations at the same rate. Although this is a simple example, it illustrates the type of information required by managers to evaluate the efficiency of the infrastructure and identify correlations between resource usage and energy consumption. Managing and analysing this data can provide system administrators with hints on how to provision and manage resources more efficiently.
5 Conclusion and Future Work This chapter presents the steps in instrumenting and monitoring the energy consumption of an experimental Grid. As observed, having a configurable sensing infrastructure is one of the basic components mandatory for designing energy
The Green Grid’5000: Instrumenting and Using a Grid with Energy Sensors
41
efficient software frameworks. The Green Grid’5000 is the first available large-scale platform whose energy is monitored per plug every second. Current research on energy efficiency in large-scale distributed systems is made possible by the instrumented platform [7, 9]. As of writing of this contribution, the energy sensing infrastructure has been in operation for over 10 months. Several techniques have been applied to obtain, store and manage the data resulting from monitoring the energy consumed by the infrastructure. The different manners in which the data has been stored and reported aim to serve the needs of several categories of users, including application developers, middleware designers, system administrators and infrastructure managers. In addition to energy consumption data, obtaining and analysing information from multiple sources – e.g. schedulers and resource utilisation – is essential to understand the usage of the infrastructure and applications’ behaviours. This analysis can help system designers and managers to identify how to devise and implement techniques for reducing the cost of energy consumption. It also allows them to evaluate the return of investment of deploying the sensing infrastructure itself. At present, instrumenting a platform is expensive and its cost can surpass the savings achieved by striving to improve the energy efficiency of the overall infrastructure. As future work, we intend to deploy energy sensors in several Grid’5000 sites and monitor the consumption of both computing and network equipment. In addition, we are in the process of creating a repository where the data on energy consumption will be made publicly available to the research community.
References 1. Cappello et al, F.: Grid’5000: A large scale, reconfigurable, controlable and monitorable grid platform. In: 6th IEEE/ACM International Workshop on Grid Computing, Grid’2005. Seattle, Washington, USA (2005) 2. Andreozzi, S., De Bortoli, N., Fantinel, S., Ghiselli, A., Rubini, G.L., Tortone, G., Vistoli, M.C.: Gridice: a monitoring service for grid systems. Future Generation Computer Systems 21(4), 559–571 (2005). DOI http://dx.doi.org/10.1016/j.future.2004.10.005 3. Barham, P., Dragovic, B., Fraser, K., Hand, S., Harris, T., Ho, A., Neugebauer, R., Pratt, I., Warfield, A.: Xen and the art of virtualization. In: 19th ACM Symposium on Operating Systems Principles (SOSP ’03), pp. 164–177. ACM Press, New York, USA (2003). DOI http:// doi.acm.org/10.1145/945445.945462 4. Capit, N., Costa, G.D., Georgiou, Y., Huard, G., n, C.M., Mouni´e, G., Neyron, P., Richard, O.: A batch scheduler with high level components. In: Cluster computing and Grid 2005 (CCGrid05) (2005). URL (http://oar.imag.fr/papers/oar ccgrid05.pdf) 5. Da Costa, G., Gelas, J.P., Georgiou, Y., Lef`evre, L., Orgerie, A.C., Pierson, J.M., Richard, O., Sharma, K.: The green-net framework: Energy efficiency in large scale distributed systems. In: HPPAC 2009 : High Performance Power Aware Computing Workshop in conjunction with IPDPS 2009. Roma, Italy (2009) 6. Gu, J., Luo, J.: Reliability analysis approach of grid monitoring architecture. Annual Conference ChinaGrid 0, 3–9 (2009). DOI http://doi.ieeecomputersociety.org/10.1109/ChinaGrid. 2009.28
42
M.D. de Assunc¸a˜ o et al.
7. Lef`evre, L., Orgerie, A.C.: Designing and evaluating an energy efficient cloud. The Journal of SuperComputing 51(3), 352–373 (2010) 8. Moore, J., Chase, J., Ranganathan, P., Sharma, R.: Making scheduling “cool”: Temperatureaware workload placement in data centers. In: USENIX Annual Technical Conference (ATEC 2005), pp. 5–5. USENIX Association, Berkeley, CA, USA (2005) 9. Orgerie, A.C., Lef`evre, L., Gelas, J.P.: Save watts in your grid: Green strategies for energyaware framework in large scale distributed systems. In: 14th IEEE International Conference on Parallel and Distributed Systems (ICPADS). Melbourne, Australia (2008) 10. Singh, T., Vara, P.K.: Smart metering the clouds. IEEE International Workshops on Enabling Technologies 0, 66–71 (2009). DOI http://doi.ieeecomputersociety.org/10.1109/WETICE. 2009.49 11. Snowdon, D.C., Ruocco, S., Heiser, G.: Power Management and Dynamic Voltage Scaling: Myths and Facts. In: Proceedings of the 2005 Workshop on Power Aware Real-time Computing (2005) 12. Truong, H.L., Fahringer, T.: Scalea-g: A unified monitoring and performance analysis system for the grid. Sci. Program. 12(4), 225–237 (2004) 13. Zanikolas, S., Sakellariou, R.: A taxonomy of grid monitoring systems. Future Generation Computer Systems 21(1), 163–188 (2005). DOI http://dx.doi.org/10.1016/j.future.2004.07.002
Porting a Seismic Network to the Grid Paolo Gamba and Matteo Lanati
Abstract The chapter describes the experience and lessons learnt during customization of a seismic early warning system for the grid technology. Our goal is to shorten the workflow of an experiment, so that final users have direct access to data sources, i.e. seismic sensors, without intermediaries and without leaving the environment employed for the analysis. We strongly rely on remote instrumentation capabilities of the grid, a feature that makes this platform very attractive for scientific communities aiming at blending computational procedures and data access in a single tool. The expected outcome should be a distributed virtual laboratory working in a secure way regardless of the distance or the number of participants. We started to set up the application and the infrastructure as a part of the DORII (Deployment of Remote Instrumentation Infrastructure) project. In the following sections we will try to explain the steps that led us to integration, the experience perceived by the testers, the results obtained so far and future perspectives.
1 Introduction The aim of the application described in this article is to retrieve data from a seismic sensor network and process them to obtain useful information about earthquakes. At the current status, it is reasonably hard that the grid can effectively support an early warning system: the time available to detect seismic waves and broadcast the alarm message is too tight. This time window varies on the basis of the distance
P. Gamba University of Pavia, Department of Electronics, via Ferrata, 1 - 27100 Pavia, Italy e-mail:
[email protected] M. Lanati () Eucentre, via Ferrata 1 - 27100 Pavia, Italy e-mail:
[email protected] F. Davoli et al. (eds.), Remote Instrumentation for eScience and Related Aspects, DOI 10.1007/978-1-4614-0508-5 4, © Springer Science+Business Media, LLC 2012
43
44
P. Gamba and M. Lanati
between the epicenter and the area being monitored, however a good estimation goes from 30 to 60 s. It is common practice to put sensors near relevant buildings or infrastructures (railways, nuclear, or industrial plants) and perform locally all the computations needed to detect P seismic waves, forecasting the earthquake arrival. All the necessary grid-related procedures, i.e. identification or resource matching, can dramatically delay this phase. On the contrary, the post-event phase opens more interesting scenarios, especially for fast evaluation and assessment. The grid offers enough free space to host a wide catalog of events, always available to be processed. Computations on large quantities of data is a must for this platform and can be exploited, for example, to produce risk analysis maps. We also have to remember the added value of recent developments such as remote instrumentation control, bringing directly into the grid the source of information. New ways of collaboration and resource sharing are possible, in particular the set up of virtual and distributed laboratories. The idea of using computational grids to interpret historical data sets is quite straightforward, see [1], but nowadays a deeper integration is fundamental. Webbased grid services can play a relevant role in state-of-the-art efforts to mitigate or forecast damages brought about natural disasters. They are good candidates for broad collaboration from the geographical and inter-disciplinary point of view due to their intrinsic remote and distributed nature. For example, QuakeSim [2] makes available a web portal capable of merging sensor data sources (satellite images and GPS readings from ground stations, both real time or recorded) with multiscale models to study fault system’s dynamics, focusing in particular on the California case. The potential impact of these activities could help in sharing information in an easier way, improving cooperation experience and reducing response time during critical situations. The intermediate layer adapting the devices to the graphical interface proposed to the final users generalizes the instrument model. The proprietary protocol solutions adopted by the manufacturers are hidden by implementing independent dialogue boxes, commands, or configuration parameters. The chapter is organized as follows: after a brief review of the grid services supporting our work, some space is reserved to the description of the seismic sensor network monitored and its access protocol. Section 4 explains the theoretical basis of the analysis carried out and all the previous concepts are mixed in Sect. 5, where all the details about integration are specified. Finally, a discussion about users evaluation and future perspective is proposed.
2 VCR and IE: New Services for the Grid User Interface (UI), a set of command line tools to access the grid, from identity to resource management, still represents a quick way to enter this world and keeps some advantages. It can hardly be replaced during the debug phase or for fast application prototypation; moreover, it is not resource greedy. However, it is not ideal for interactive tasks and it can appear unfriendly to people approaching the
Porting a Seismic Network to the Grid
45
grid without aiming to become developers. This is the target audience the DORII project1 is dedicated to: scientists belonging to experimental science, earthquake, and earth observation communities demanding for computation/storage resources in strong connection to instrumentation availability. Since they are not specialists, the demand for an easy-to-use and easy-to-learn environment is taken into great consideration. All these problems have been deeply investigated in previous projects such as GRIDCC2 and RINGrid3 , for collaborative environments involving remote instrumentation, or int.eu.grid4 for interaction aspects. DORII relies on these experiences to provide an integrated multiplatform environment to deal with real devices and to perform data analysis. This goes beyond simply putting legacy results all together, the goal is to create an e-Infrastructure that can serve different communities, but sharing the same fundamental software components. The infrastructure’s front-end is the Virtual Control Room (VCR) [3], a web portal making available all the basic grid commands adopting a point-and-click philosophy. First of all, the potential users need only a web browser to start their work, then certificate management is much easier and clearer. It has to be handled only at the first log in, then credentials are issued automatically. A dedicated applet lists all the resources supplied by the project partners: the most interesting ones are the Instrument Elements (IEs) and their Instrument Managers (IMs) [4]. The IE is the middleware element that exposes some fundamental web services, virtualizing instruments functionalities: configuration, monitoring and management. The key components of an IE are its IMs, they act as protocol adapters and describe each device as a finite state machine. The IM is quite autonomous, its Event Processor gets input from the user, typically a command, or from the sensor (errors and monitored parameters), then the Brain Module decides whether to change the state or interact with the instrument through the Resource Proxy. The behaviour of the IM is set in an XML file which defines the available commands, the monitored attributes and state transitions. It is clear that the IM is a general and high abstraction level framework, needing customization. The most difficult part in the developer’s work is to interface the IM and the physical node, but this is a general problem, not a grid specific one. Usually, manufacturers provide a development kit with libraries or a reference guide explaining the protocol adopted.
3 The Instrumentation Put Into the Grid The instruments involved in the application are seismic sensors, that is accelerometers inserted into a shock-proof package measuring the ground velocity along the three Cartesian directions. Each sensor is usually associated to a seismic 1
DORII project web site: http://www.dorii.eu. GRIDCC project web site: http://www.gridcc.org/. 3 RINGrid project web site http://www.ringrid.eu/. 4 int.eu.grid project web site: http://www.interactive–grid.eu/. 2
46
P. Gamba and M. Lanati
Fig. 1 Sensor network deployment and communications
recorder, providing some basic services such as continuous power supply, network connection, geo-localization, and remote administration. The sensor network is owned and managed by Dip.Te.Ris (Dipartimento per lo Studio del Territorio e delle sue Risorse) at the University of Genoa and it is spread throughout the Liguria region, in the north-west of Italy. The coverage area is not too wide in order to ensure good monitoring quality. Moreover, it is more valuable to have the global overview of the phenomena given by the whole set, so all the measurements are gathered by a single server. It is run by a proprietary software solution provided by Nanometrics Inc., responsible also for the communication with the sensors. A network node is connected to the central point by means of a wired or wireless (satellite, GPRS modem) link and data are exchanged over UDP, as depicted in Fig. 1. Inbound data flow [8], running from the nodes to the server, is split in packets identified by a unique sequence number and a timestamp, then encapsulated in UDP datagrams. A packet starts reporting the oldest sequence number available as a 4-byte integer, while the remaining space is organized in bundles, a 17-byte long independent collection of data (timestamp, status message or measurements). The first bundle in a packet acts as header, carrying some common information such as packet type, time stamp, sequence number or instrument identifier, coding the serial number and model, plus other specific fields (i.e., in a data packet header there is the sampling frequency and the value of the first measure). Among the different bundle types that can follow the header, the most commonly found is the one carrying compressed data, so it is worth to look in detail at its structure. It contains from 4 to 16 samples, stored using a differential algorithm, referring to the initial value placed in the data packet header. The compressed data bundle is subdivided in four data sets of 4 bytes each, while the remaining space, 1 byte, encodes how data are packed: byte, word or long difference. So, the single data set can be interpreted as four single byte differences, two word differences or one long difference. A user is not supposed to deal with bytes, but packets related to the same source, for example a ground velocity component or the state of health messages of a station,
Porting a Seismic Network to the Grid
47
are bunched together in a channel. The list of these unique streams of information is then broadcasted by the server. The number of bundles in a packet is variable, an odd value ranging from 1 to 255, to maximize network efficiency depending on the chosen carrier. However, once packet dimension is fixed during the deployment, it cannot be changed, ensuring easy implementation of the protocol even in devices with poor resources. Finally, reliability is guaranteed because wrong or missing packets are retransmitted. The central recording site stores and reorders what the sensors have transmitted to offer data access services over TCP (right side of Fig. 1) relying on two protocols: Private Data Stream and Data Access Protocol. The Data Access Protocol defines the communication between a client and the Nanometrics DataServer component, making available historical recordings in original format or as uncompressed stream. Data types provided include triggers, events, state of health messages, time-series, or transparent serial data. A trigger is generated when the energy associated to a signal (channel) changes, probably due to an earthquake, while event packets describe the seismic phenomena in term of duration and amplitude. State of health messages contain details about the current situation of a sensor, logs or error conditions. Transparent serial data employ only the compressed format and packets are sent when filled with a number of bytes specified by the user, unless a time out occurs. In this case, the space left is padded and there are no mistakes because the header bundle always specifies the number of valid bytes in the payload. On the contrary, seismic data retrieved in time-series mode can be expressed in the original format or as uncompressed 32-bit integer values. In the first case, the DataServer is forwarding the packets received from the sensor, while in the second, data is arranged in fixed-length blocks, usually corresponding to a one second time window. A typical connection goes through these steps: • Open a socket to the DataServer on port 28002 • Read the connection time (4-byte integer) • Send a connection request, including the connection time together with username and password, if necessary • Wait for a Ready message • Send a request message to retrieve data, state of health, events, triggers • Receive answers from the server (this phase ends with a Ready message) • Send a new request or terminate the subscription • Close the socket The server notifies the client that it has successfully carried out a request sending a Ready message, so, from now on, it is available for a new query. If the client does not wait for this reply, demanding immediately different information, the current request is cancelled (however acknowledged by the Ready message) and the new one is taken into account. Moreover, if the client does not generate traffic for more than 20 s the connection is closed by the server. The client can reset the timer issuing a RequestPending packet and handling the subsequent Ready message.
48
P. Gamba and M. Lanati
Private Data Stream protocol supervises online data access in near real time. The component involved is the NaqsServer, or better, the Stream Manager subsystem, which plays a role similar to those seen for the DataServer, but with a strict time schedule. Once again, events, trigger, state of health messages, timer-series and transparent serial data are supported. Compressed data channels (i.e., state of health, time-series, and transparent serial) can be subscribed as raw or buffered streams. In a raw stream, the packets are sent to the client as they are received from the sensor node, so without guarantees on correct order, but with possible duplications or gaps in the flow. If a buffered stream is requested, the most recent packets available at the server for that channel before subscription time, usually 4 or 5, are forwarded to the client. In this sense, collected flow starts in the past. When requesting a channel, it is also possible to define the short-term-completion time, that is the interval the server waits to fill gaps in the measurements with retransmitted packets. This parameter, if not disabled, can vary from 0 to 300 s. In front of a tolerable delay, the Stream Manager provides with reasonable probability a continuous set of values, which is very important for the subsequent analysis, avoiding interpolation. Finally, since the user is not allowed to access sampling frequency set on the instrument, the Timeserver can apply a decimation filter to down-sample the data stream. In order to set up a connection according to the Private Data Stream protocol a client should: • • • • • •
Open a socket to the NaqsServer on port 28000 Send a connection request Receive the channel list Subscribe one or more channels Receive and decode measurements, adding or removing subscriptions When done, send a terminate subscription and close the socket
The Stream Manager can handle multiple requests, so it is possible to subscribe to more channels at the same time, dynamically adding/removing them or changing parameters such as short-term-completion on the fly. The client should also prevent the server to close the connection after 30 s of inactivity sending a keep alive message.
4 The Proposed Application The application discussed in this article is split in two parts: interaction with instruments and data analysis. The instrument oriented task consists in polling the central server to obtain the measurements, in real time or off-line. The two operational modes are equivalent for the results, but the former is closer to an early warning system, while the latter is useful in case of historical analysis, i.e., to evaluate how parameters influence an algorithm.
Porting a Seismic Network to the Grid
49
The computational part of the application starts from the ground velocity provided by the sensors to extract the time history and calculate the Fourier amplitude spectrum together with the acceleration response spectrum. The first one is simply the plot of displacement, ground velocity and acceleration versus time. Wellknown algorithms for numerical integration [5] or differentiation [6] give the result, respectively, for displacement and acceleration. The absolute value of the Fourier amplitude spectrum shows the acceleration in the frequency domain, obtained by a Fast Fourier Transform. Finally, the acceleration response spectrum is the most technical item discussed here and broadly adopted in the seismic community, so some more explanations [7] are needed. It is defined as the mathematical locus of acceleration response peaks (in absolute values) for a set of single degree of freedom oscillators characterized by different natural periods and fixed damping constants. If one of these systems is subject to ground acceleration imposed by an earthquake, the equation describing its behaviour is mRu C c uP C ku D mRug
(1)
where uR , uP and u are, respectively, acceleration, velocity and displacement; m defines the mass; k the spring constant; c the damping coefficient; and uR g is the ground acceleration (in our case derived from measured ground velocity). Equation (1) can be also written as uR C 2!n uP C !n2 u D Rug
(2)
where the damping ratio D
c c D p cc 2 km
(3)
is given by the damping constant over the critical damping constant or expressed as a function of c, k, and m. At the same time, the natural frequency 2 !n D D Tn
r
k m
(4)
depends on the mass and the spring constant. Even if derived from a simple model, the spectral acceleration derived from the response spectrum is important because, multiplied by the mass, represents the peak base shear of the structure. Figure 2 shows the interaction between the grid blocks, all the resources available in the e-infrastructure has been involved. The two basic steps (acquisition and analysis) are recognizable, all the actions are routed through the VCR. VCR and IE are located in Eucentre, while Storage Elements (SEs) and Computing Elements (CEs) are selected from the DORII pool assigned to the generic catch-all VO. The National Research Networks take in charge the traffic generated, even for the communication between the IE andthe Nanometrics server (not shown here). Given
50
P. Gamba and M. Lanati
Fig. 2 Grid blocks involved in the application deployment
the central role of the VCR, new institutions that would like to use the application have only to join the VO and apply for an account on the web portal. The secure and coordinate instrument access is guaranteed through the VOMS architecture.
5 Integration Into the Grid In order to exploit grid potentialities in the seismic application described in Sect. 4, it is fundamental to give access to sensors from the grid’s user interface chosen. This is one of the most relevant points in the DORII project and the final goal is to propose to the users an integrated environment where to take measurements and perform computations. Despite the Nanometrics solution is proprietary, a detailed reference guide [8] describes every aspect of the protocol, so it is quite straightforward to implement a client software tailored for all needs. We wrote a Java client library covering most of the features described in Sect. 3: access to historical and real time data as raw or buffered time-series stream in original compressed format. Java language was chosen to maximize the integration with the DORII framework, however the library was developed as a generic tool and can be used also outside the grid. Captured data are saved in two formats: ASCII plain text and binary. Text files are organized in two columns, relative time from the beginning of the acquisition for that channel and the integer value produced by the A/D converter on the station deployed (quantized millivolts). On the basis of the sensor, a conversion factor gives the ground velocity in meters per second. Binary files are written according to the SAC (Seismic Analysis Code) format5, widely adopted in the earthquake community. 5
SAC Data File Format, http://seismolab.gso.uri.edu/savage/sac/sac format.html.
Porting a Seismic Network to the Grid
51
Fig. 3 Finite state machine for the sensor network
Seismic sensor network, or better, access to the Nanometrics server, is hidden behind two IMs, one for NaqsServer and one for DataServer. Both of them are abstracted as finite state machines, sharing the structure depicted in Fig. 3. From the implementation point of view, IMs are simply a client software based on the Java library running in an application server (the IE), having fixed some parameters. For example, we choose to request only unbuffered streams disabling the short-term-completion time, in order to keep the delay under control. Going into details, the states and related commands, highlighting the differences in the implementation, are Off: The connection to the Nanometrics server is not established yet, the command Turn on opens the socket and triggers the transition to the next state. On: The client sets up the communication by means of a proper request and handles the answer. The NaqsServer provides the channel list, while the same information should be demanded to the DatServer. In this case, a request for a precise list is issued, in order to get the time-window availability of stored data for each channel. Now the IM waits for the user to receive a command, meanwhile keep alive packets are sent regularly (15 or 25 s for Naqs or Dat, respectively) to prevent connection closing. There are two commands available: Shutdown, closing the connection and pushing back the state machine to the Off condition, or Acquire. The second one allows a user to subscribe a channel, for NaqsServer implementation, or to request seismic measurements in a specified time window, for Data Access Protocol. Acquisition: Once the acquisition phase is started, the IM switches to this state. For near real time subscriptions, it is possible to add or remove channels from the current polling list employing the Subscribe and Unsubscribe commands. The offline data access IM version enables other directives: Cancel, clearing
52
P. Gamba and M. Lanati
Fig. 4 IM for NaqsServer during acquisition phase
the current process on the server, and NewAcquisition, to retrieve a new time windowed bunch of data. The action comes successfully to an end if all previous requests have been satisfied, that is no other Ready messages are expected. Otherwise, the execution fails and the IM remains in this state. All the measurements gathered are stored locally on the IE while a set of VCR attributes give a feedback showing the last sample read (see Fig. 4). Stopped: Issuing the Stop command in the previous state, the connection to the server is closed, the files holding the downloaded data are finalized and this transition takes place. The Store command uploads local files to a SE specified on a configuration file, while the user enters only the existing remote folder on the selected resource. The IM can now be turned off or set to the On state, opening a new connection.
Porting a Seismic Network to the Grid
53
Fig. 5 First form of the VCR application. The first argument is filled in by the user, the second one (SE address) is set by default but editable
Error: This is the target in case an error occurs. It is not shown in Fig. 3 for a clarity matter. The analysis of seismic data is completely carried out on the grid, starting from the storage on a SE, as we saw. The code extracting the information described in Sect. 4 is written in C and should run for every file collected. A convenient solution is to use parametric jobs, where file name is the item changing each time. The job is very simple, it consists in downloading the file from the SE then running the executable statically compiled and included in the Input Sandbox. In order to access CEs from the VCR it is necessary to write a JDL (Job Description Language) file, so the potential users are supposed to go into details of grid technology. It is better to address to VCR applications, that is a form-oriented interface launching jython scripts and/or jobs. Our application needed only two steps. In the first one, shown if Fig. 5, the user is prompted for the remote folder on the SE where the IM saved sensors’ output. The script customizes a template and yields a JDL file, used to submit a job in the next step. The user has only to wait for the computation to complete, then the output is placed in the home folder on the VCR. This approach has been chosen to avoid long type writing and to hide some details that scientists feel unnecessary. Some tests undertaken during the deployment phase of the DORII project showed that researchers are more focused on the scientific implications of the grid rather than its technical aspects and they are constantly looking for comfortable interfaces. Another tool going in this direction and under development in DORII is the Workflow Management System (WfMS) [9], a graphical environment to interact with all the grid nodes. Application porting to WfMS is an ongoing activity.
6 Discussion and Future Work First of all, a brief overview of the output produced during the computational phase is proposed. Figure 6 plots the ground velocity, that is the raw data generated by the sensor, also saved in SAC format. This snapshot was captured in real time from
54
P. Gamba and M. Lanati
Ground speed [m/s]
−3.4
x 10−6
−3.6 −3.8 −4 −4.2 −4.4
0
5
10
15
20
25
time [s]
Fig. 6 Ground velocity in the vertical direction
2
b
x 10−5
6
x 10−6
5 1
Acceleration
Ground acceleration [m/s2]
a
0
4 3 2
−1 1 −2
0
5
10
15
time [s]
Time domain
20
25
0
0
20
40
60
80
100
Frequency [Hz]
Frequency domain
Fig. 7 Ground acceleration plots
RORO. HHZ channel and represents the vertical component (HHZ) measured by the station whose identification code is RORO. The absolute values are very small because there is not an ongoing significant seismic activity in the area monitored. Subsequent processing needs the acceleration, so velocity is differentiated and the results are shown in Fig. 7a, b, for time and frequency domain, respectively. The absolute spectrum is a pure number since the ratio with g, the gravity acceleration, was calculated. Finally, the response spectrum is worked out, see Fig. 8. The natural period Tn on the horizontal axis is increased with a step equal to 0.025 s (this means that the mass m and the spring constant k are changed accordingly to (4)), while the damping ratio is fixed to 5%. It is worth noticing that in real cases the response spectrum should take into account a natural period extending to 50 s and the analysis should be repeated for different damping ratios, up to 20%.
Porting a Seismic Network to the Grid
Acceleration [m/s2]
8
55
x 10−5
6
4
2
0
0
0.5
1
1.5 Natural period [s]
2
2.5
3
Fig. 8 Response spectrum
The application has been proposed to some scientists belonging to the earthquake community in order to have an operational point of view on the grid approach. Some difficulties arose at the beginning, mainly related to the new platform. The testers were used to work with different tools, commonly installed on their personal laptop, so authentication is limited to log on procedure. The change of perspective triggered by the distributed environment needs a more accurate authorization management. This is done automatically, but it may take time, which is uncomfortable for the user. However, after the first impact, the benefits were immediately perceived. A unique web portal, accessible from every device equipped with a browser supporting Java, allows to keep under control every aspect of the experiment. Moreover, parametric jobs ensure an intrinsic parallelization, which is very important for a statistical study such as risk evaluation. The description of the work exposed so far is the first step towards other goals. We are planning to continuously monitor some stations and store the accelerograms on a SE, creating a huge data collection. This is useful not only to test the reliability of the platform, it also extends the time window in which data are available (current limitations are due to the fact that the Nanometrics server is a single machine). Users are not restricted only to access historical series, but they can also speed up and broaden response spectrum analysis. Another problem where we think the framework we developed can be applied is seismic waves detection from earthquake recordings, an important issue to locate and analyze the event [10]. In particular, in the next months we will focus on P-waves picking employing the Akaike Information Criterion (AIC) [11]. P-wave onset is associated to the sudden variation in the amplitude and the increase in the energy carried at high frequencies. According to [12], the seismogram can be modeled as a sum of auto-regressive processes, whose coefficients and order
56
P. Gamba and M. Lanati
change before and after the arrival. AIC estimates how real data fit with the model and the onset is identified by a minimum of the function. It is obvious that it is always possible to find a minimum, so, to avoid errors it is fundamental to set a proper time window. This can be accomplished by a filter, a Hilbert transform or a neural network. Some code implementing AIC with Hilbert transform filtering is ready, so we are going to port it into the grid together with a collection of seismograms recorded during some earthquakes. Then, we will launch a series of jobs to study the impact of time window on the quality of auto-picking with respect to the same procedure executed by a human operator. Acknowledgements The work described is carried out under the DORII project, supported by the European Commission with contract number 213110. The authors would like to thank Mirko Corigliano for the analysis software ported into the grid and for the related explanations. The cooperation of Dip.Te.Ris at University of Genoa was fundamental to access the sensor network. In particular, the contribution received from Daniele Spallarossa and Gabriele Ferretti was very useful to fix the application. Finally, the work of Davide Silvestri in implementing the Instrument Managers and part of the Java library was highly appreciated.
References 1. Y. Guo, J. G. Liu, M. Ghanem, K. Mish, V. Curcin, C. Haselwimmer, D. Sotiriou, K.K Muraleetharan, L. Taylor, “Bridging the Macro and Micro: A Computing Intensive Earthquake Study Using Discovery Net”, Proc. of the 2005 ACM/IEEE SC 05 Conference (SC 05), Seattle, WA, USA, Nov. 2005. 2. A. Donnellan, J. Parker, C. Norton, G. Lyzenga, M. Glasscoe, , M. Glasscoe, G. Fox, M. Pierce, J. Rundle, D. McLeod, L. Grant, W. Brooks, T. Tullis “QuakeSim: Enabling Model Interactions in Solid Earth Science Sensor Webs”, Proc. of 2007 IEE Aerospace Conference, Big Sky, MT, USA, Mar. 2007. 3. R. Ranon, L. De Marco, A. Senerchia, S. Gabrielli, L. Chittaro, R. Pugliese, L. Del Cano, F. Asnicar, M. Prica, “A web-based tool for collaborative access to scientific instruments in cyberinfrastructures” in F. Davoli, N. Meyer, R. Pugliese, S. Zappatore, Eds., Grid Enabled Remote Instrumentation, Springer, New York, NY, 2008, pp. 237-251. 4. E. Frizziero, M. Gulmini, F. Lelli, G. Maron, A. Oh, S. Orlando, A. Petrucci, S. Squizzato, S. Traldi, “Instrument Element: a new Grid component that enables the control of remote instrumentation” Proc. 6th IEEE Internat. Symp. on Cluster Computing and the Grid Workshops (CCGRIDW 06), Singapore, May 2006. 5. N. M. Newmark, “A Method of Computation for Structural Dynamics”, ASCE Journal of the Engineering Mechanics Division, Vol. 85, No. 3, 1959, pp. 67-94. 6. M. Abramowitz, I. A. Stegun, “Handbook of Mathematical Functions with Formulas, Graphs, and Mathematical Tables”, Dover, New York, NY, 1964, Section 25.3.4 . 7. A. K. Chopra, “Dynamics of Structures (Theory and Applications to Earthquake), 3rd Edition”, Prentice Hall, Upper Saddle River, NJ, 2007, pp. 208-217. 8. Nanometrics Data Formats - Reference Guide. 9. M. Oko´n, D. Kaliszan, M. Lawenda, D. Stokłosa, T. Rajtar, N. Meyer, M. Stroi´nski, “Virtual Laboratory as a remote and interactive access to the scientific instrumentation embedded in Grid environment”, Proc. 2nd IEEE Internat. Conf. on e-Science and Grid Computing (e-Science 06), Amsterdam, The Netherlands, Dec. 2006.
Porting a Seismic Network to the Grid
57
10. H. Zhang, “Application of Multilayer Perceptron (MLP) Neural Network in Identification and Picking P-wave arrival”, ECE539 Project Report, Department of Geology and Geophysics, University of Wisconsin-Madison, 2001. 11. H. Akaike, “Markovian representation of stochastic processes and its application to the analysis of autoregressive moving average process”, Annals of the Institute of Statistical Mathematics, Vol. 26, 1974, pp. 363-387. 12. R. Sleeman, T. V. Eck, “Robust automatic P-phase picking: an on-line implementation in the analysis of broadband seismogram recordings”, Physics of the Earth and Planetary Interiors, Vol. 113, 1999, pp. 265-275.
Integrating a Multisensor Mobile System in the Grid Infrastructure Ignacio Coterillo, Maria Campo, Jesus ´ Marco de Lucas, Jose Augusto Monteoliva, Agust´ın Monteoliva, A. Monn´a, M. Prica, and A. Del Linz
Abstract This document presents a mobile floating autonomous platform supporting an extensive set of sensors, with depth profiling capabilities and ease of deployment at any place within a water reservoir limits in a short time; and its integration in the existing grid infrastructure. The platform equipment is remotely operable and provides real-time access to the measured data. With the integration in the existing Grid infrastructure, by means of the infrastructure and middleware provided by the DORII project, a remote operator can not only control the platform system but also use the produced data as soon as it is recorded in desired simulations and analysis. The instrumentation is installed in a mobile floating autonomous platform containing a power provider system (consisting of a pair of solar panels and a wind turbine, along with a series of deep-cycle batteries), a set of surface sensors, and a set of water quality sensors integrated in a compact underwater cage, connected with a steel cable to a winch system situated at the platform surface, providing water column profiling capabilities. All the underwater sensors (plus one of the surface sensors) are locally controlled by a datalogger device which has capabilities for storing up to 14 days worth of sampling data, constituting a first layer of data security backup. An onboard computer serves as datalogger for the rest of the sensors and as a second layer data backup. Communication with the land station is managed via a WiFi point to point
This work is made in the frame of the DORII project under grant agreement RI-211693 (7th Framework Program) I. Coterillo • M. Campo • J.M. de Lucas () Instituto de F´ısica de Cantabria (IFCA, CSIC-UC), Santander, Spain e-mail:
[email protected];
[email protected];
[email protected] J.A. Monteoliva • A. Monteoliva • A. Monn´a Ecohydros S.L, Santander, Spain e-mail:
[email protected];
[email protected];
[email protected] M. Prica • A.D. Linz ELETTRA, Sincrotrone Trieste S.C.p.A, Trieste, Italy e-mail:
[email protected];
[email protected] F. Davoli et al. (eds.), Remote Instrumentation for eScience and Related Aspects, DOI 10.1007/978-1-4614-0508-5 5, © Springer Science+Business Media, LLC 2012
59
60
I. Coterillo et al.
link, using standard consumer directional antennas on both sides of the link. With this configuration distances of over 20 km are possible. In the land station, a server hosts both the primary database where the platform computer stores all sensor data (replicated at a second location, for additional data redundancy) and the Instrument Element (IE) middleware that integrates the whole monitoring system in the Grid infrastructure and provides remote access to the data and controlling capabilities through the Virtual Control Room (VCR). From within the VCR an operator is able not only to control and configure the behaviour of the sensors and access the recorded data in real-time, but also to employ this data in analysis jobs executing in the high availability Grid environment.
1 Introduction Scientific activity is based in the collection of data through observation and experimentation and the formulation and testing of hypotheses [1]. After this basic and broad definition, the different fields of science present different and highly heterogenous data acquisition requirements (referring to data acquisition not only as the proper process of acquisition but also as the transport, storage and curation of the acquired data), and computing requirements in order to successfully process their data. The DORII (Deployment of Remote Instrumentation Infrastructure) project aims to deploy e-Infrastructure for new scientific communities where ICT technology is not present but is demanded to empower their community daily work [2]. The project is mainly focused on groups of scientific users with experimental equipment and instrumentation currently not, or only partially, integrated in the European e-Infrastructures. The framework established by the DORII project allows to integrate existing instrumentation in the current e-Infrastructure providing an abstraction for instruments – the Instrument Element (IE) – and a web interface called the Virtual Control Room (VCR) which provides final users a collaborative environment with access to both the infrastructure resources (storage and computing resources in the Grid) and their instrumentation by means of the IE. With the current state of the art in both communication and computing technologies, this kind of framework provides big opportunities for applications within a broad range of demands, such as multi user access to instrumentation, remote control and operation of the instrumentation, distributed networks of sensors, high data throughput, highly demanding computing applications for processing, etc. This document presents a brief (generic) overview on how to integrate one application in the DORII e-Infrastructure using two components from the DORII middleware (the IE and the VCR), and then proceeds to describe the integration of one of the applications inside the DORII project environmental community, Monitoring Inland Waters And Reservoirs, a multi sensor remotely operated autonomous mobile platform used by Ecohydros S.L [3] for monitoring the quality of water in the Cuerda del Pozo reservoir (located) in Soria, Spain.
Integrating a Multisensor Mobile System in the Grid Infrastructure
61
2 The DORII Framework In the kind of situation addressed by the DORII project objectives, a scientific community of any type would have existing instrumentation already deployed, in one of many possible working scenarios. A summary of the instrumentation setup in the scientific areas represented in the project is as follows: • The Earthquake community, present with applications which acquire data from various sensor networks. • The Experimental science community, with applications such as a synchrotron and free electron lasers requiring support for multi-user access to instrumentation and specific individual needs. • The Environmental science community, where one can find applications such as the Oceanographic and coastal observation and modelling Mediterranean Ocean Observing Network which uses data from a vast network of floating sensors, the Oceanographic and coastal observation and modelling using Imaging which processes images obtained from several digital cameras, and the Monitoring Inland Waters and Reservoirs, which will be studied deeper in the following sections, and which integrates a number of heterogeneous sensors. It can be inferred that there are a number of use cases with very different combinations of requirements regarding mobility, availability, user access control and scheduling, bandwidth of produced data, and number of sensors. Additionally, the computation requirements of the different applications range widely as the set of simulations vary depending of the kind and size of the data to be processed in each application.
2.1 The Instrument Element The key element in the process of integration of each application instrumentation is the Instrument Element (IE). A Java web service interface for client access, (using Apache Axis) which brings the possibility of including scientific instruments and sensors in the Grid environment, typically composed of computing and storage resources and additional support and structural systems as file catalogue or information systems. The IE framework provides support for secure access control (by using the gLite/GSI security model based on users proxy delegation), multiple user access through concurrency control and locking, and a simple Grid storage access through a Grid-FTP based utility that allows the Instrument Managers to save and retrieve data to and from Grid storage elements.
62
I. Coterillo et al.
Fig. 1 Instrument Element architecture (a) and Communication between IMs and instruments (b)
It also offers the possibility of using Java Message Service (JMS) for asynchronous monitoring of instrument variables, enabling the framework to use signaled alarms and events (Fig. 1). Instruments and sensors are interfaced via Instrument Managers (IM) that allow connection to physical devices, or, more precisely, their control systems. Instrument Managers consist mainly of the Java client code for managing the instrumentation and should run inside the IE installation. Summing up, the IE offers a common interface for accessing instrumentation, and can be viewed as an extension of the gLite Grid middleware laying at the bottom of the DORII architecture. A more extended description can be found in [4].
2.2 The Virtual Control Room The Virtual Control Room (VCR) is a collaborative environment based on Gridsphere [5] and web 2.0 technologies, offering a set of groupware tools supporting scientific team-work such as a logbook, chat, wiki help, or people and resource browsers, which serves as both the main frontend to the IE and a window to the Grid. The VCR allows users to search, browse, control, and manage Grid resources (e.g. allowing Job submission using a specific Computing Element (CE) or Workflow Management System (WMS), workflow submission, file browsing and transfering to a SE via gsiftp, LFC Catalog browsing and operation, credential management, etc.), and remote instrumentation, allowing the user to interact with Instrument Elements (by browsing and operating the managers offered in a certain Instrument Element). As support for application customization it includes an application manager for creating simple, application-specific forms and a scripting environment for creating and running simple workflows.
Integrating a Multisensor Mobile System in the Grid Infrastructure
63
Fig. 2 User view of Grid resources and instruments in the VCR
The purpose intent of the VCR is to present a final user of the instrumentation (which, according to the DORII objectives, would be a member of a community with no specific technical knowledge in the field of computation) with a seamless interface to both the instrumentation and the Grid. Figure 2 shows the typical user view of available resources once the user has logged to the VCR.
3 A Use Case: Monitoring of Inland Waters and Reservoirs This section of the document presents a practical process of integration of a real application in the DORII e-Infrastructure.
3.1 Motivation The Cuerda del Pozo water reservoir, in Soria, Spain was built in 1941 with a total surface of 2,176 ha. and has presented an average occupation of 99 hm3 (43.23%). Its main uses are the irrigation of 26,000 ha. of crops and the supply of drinking
64
I. Coterillo et al.
water to the province of Soria and (part of) Valladolid. It has historically presented a water quality below expected limits regarding its geographic situation and the characteristics of its watershed. A study of the trophic state of the reservoir was executed in 1997 (and later updated in 2008) confirming the eutrophication of the reservoir waters. This brought attention to the necessity of impulsing correcting actions to invert this trend and its unwanted consequences; the main one being the apparition of cyanophyceae algae blooms, which upon decay, could liberate toxins to the water rendering it useless for human use. As cyanobacteria blooms develop in short time scales (days) with a highly localized spatial differentiation inside the reservoir, monitoring via periodical sampling campaigns is not a suitable method for detecting/detection; and thus appears the need of a mobile monitoring platform with long-term deployment capabilities, multiple sensors and power autonomy. Ecohydros S.L, a Spanish SME dedicated to environmental consulting, and specialised in aquatic ecosystems in both coastal and inland areas, is in charge of developing and deploying such platform, while creating a model of the relations between the presence of different nutrients and the appearance of cyanophiceae algae.
4 Sensor Platform To fulfill the requirement of mobility of the sensors around the reservoir, the equipment is installed in a floating platform, suitable for moored deployment or for being towed with a small boat, which hosts the following sensors and probes. The sensor suite is selected to monitor a variety of environmental parameters, and the presence of different nutrients in the water (Fig. 3).
4.1 Underwater Sensors All underwater sensors are placed inside a steel cage which provides protection and a structure to fix the different sensor bodies to. The set of underwater sensors include: • An advanced CTD (Conductivity, Temperature, Depth) probe, with additional sensors for measuring also Dissolved Oxygen (DO), PH, REDOX potential and Salinity. • Fluorescence sensors for different groups of algae. • Hyperspectral irradiance sensors. A winch device holds the cage and manages the power supply and communications cable. This allows changing the depth of operation and the execution of profile measures in the 0–100 m depth range.
Integrating a Multisensor Mobile System in the Grid Infrastructure
65
Fig. 3 Frontal view of the platform and the solar panels (a), the platform after installing the wind turbine and the winch (b), and a close caption of the underwater cage (c)
4.2 Surface Sensors The following sensors are installed in the body of the platform above water surface: • A Meteorological station measuring wind speed and direction; rain, hail and snow events, air temperature, and relative humidity. • Net radiometer • A Hyperspectral irradiance sensor • A GPS
66
I. Coterillo et al.
Fig. 4 Sensor integration scheme
4.3 Sensor Integration The integration of the instruments is organized in a two-layer structure. The first layer (bottom) of the structure consist of a datalogger gathering the raw signal from all the sensors in the underwater cage plus the surface hyperspectral irradiance sensor and performing two important tasks. First, it digitalizes the raw signal measurements from the sensors while storing a two week (approximately) history of measurements which serves as a first level partial data backup; secondly, if offers a uniform interface for the different sensors in the cage using the Modbus [6] protocol over an Ethernet link. From this interface, the datalogger/cage pair can be abstracted as an individual multiprobe instrument (and will be referred as such in the next paragraphs). At the second layer of the structure, a computer using LabVIEW [7] works as the main datalogger, integrating all sensor data coming from the multiprobe instrument (via the Modbus over Ethernet link), the meteorological station (via a RS-232 serial connection, using standard ASCII as protocol), the GPS (accessed via NMEA over USB) and the net radiometer (accessed via a serial connection, and using a propietary protocol for which an ad-hoc communication module had to be developed) (Fig. 4). In a similar way as the first level datalogger, the LabVIEW software integrates the data acquisition process. Additionally, it performs a second level data backup locally with no practical storage limits, it writes all sensors output to an external database and above all, it exposes a control interface to all the sensors using the LabVIEW Web Services interface (this is the key to the Grid Integration of the whole system, as will be shown in the following sections).
Integrating a Multisensor Mobile System in the Grid Infrastructure
67
The computer is connected also to an external WiFi module for high transmission power, and this module is connected to a directional antenna used to connect to the land station (both the land station and the communications setup will be treated in the following sections).
4.4 Power System The platform is equipped with three sets of two 24-V batteries; one set connected to the computer and surface sensors, one set connected to the winch, and one set connected to the underwater sensors, depending on power consumption demand. The batteries are connected to two solar panels and a wind turbine for recharging; they support autonomous operation of the equipment for several days during winter, and continuously during spring/summer when the solar panels are able to provide their nominal power (which is also the time of the year when the monitoring of the reservoir conditions has special relevance).
4.5 Land Station The land station is placed in a CHD office building located at the head of the reservoir. This station contains: • Communication equipment linking to the floating platform, consisting of a WiFi access point and a directional antenna. • A server hosting the following services: – The Instrument Element – Monitoring software – A database storing all acquired data • A network camera used to monitor the platform for security reasons
5 Networking and Communications The main factor conditioning the design of the network architecture for the whole system was the remoteness of the location, situated in a rural area far away from any notable population. At the moment of planning and executing the initial deployment of instrumentation there was no GPRS/3G coverage in the area; and so in order to allow remote access to the instrumentation, the following setup was employed.
68
I. Coterillo et al.
Fig. 5 Platform Yagi WiFi antenna (a) and the land station patch WiFi Antenna (b)
5.1 Platform to Land Station Communication As mentioned in the previous sections, communication between the land station and the platform is carried via a WiFi link using the following equipment: • A WiFi AP connected to a WiFi outdoor patch directional antenna (C14dBi gain) placed in the land station. • A high transmission power WiFi external module connected to a Yagi directional WiFi antenna (with a C12 dBi gain) installed in the exterior of the platform housing (Fig. 5). With this setup it is possible to achieve link distances of up to 20 km, which is enough to cover points of interest in the reservoir with the platform while maintaining communication with the land station.
5.2 e-Infrastructure Network Communication For the same reason that GPRS/3G could not be used (the location being too remote) the only solution available for connecting the land station to the Internet – and thus to the DORII e-Infrastructure – at the moment of planning was to deploy a WiMAX link offering a 4 Mbps bandwidth (Fig. 6).
5.3 Data Security and Backup Protections During the development, installation and testing of the different components, possible points of failure were considered (ordered by severity):
Integrating a Multisensor Mobile System in the Grid Infrastructure
69
Fig. 6 Land station WiMAX patch antenna
1. Power supply failure for any of the platform components. Typically, in conditions where the solar panels are not receiving enough sunlight (occurring frequently during winter and autumn seasons in the location), the platform computer will drain its set of batteries before the underwater cage and cause a loss of data from the surface sensors and isolation of the underwater cage sensors, while these will remain in operation as their power consumption is low. 2. Failure in the communication link between the platform and the land station. If the link between the platform and the land station is broken, the final stage of data acquisition, when data is written to the database, fails; the platform becomes isolated from user access and control. 3. Failure in the communication link between the user and the land station. If the WiMAX link fails the platform is isolated from user access and control. During the description of instrument integration architecture in the previous section, a data backup point in each of the two levels was mentioned. The backup at the multiprobe datalogger assures that none of the data from the underwater sensors is lost in the case that the more severe error of type 1 occurs, as it is able to store up to two weeks worth of data after the power supply is lost, an event that will be detected by the monitoring system. The second backup point ensures that none of the sensor data is lost when the communication to the platform is temporarily lost. When the communication is restored the LabVIEW control software resumes operation as normally. A failure in the WiMAX link to the land station is quite improbable, but it is possible to experience long delays or bandwidth narrowing which hinders the user interaction with the system. An asynchronous synced replica of the database is hosted in the CSIC-IFCA site, and could be seen as a third level of backup. This
70
Fig. 7 Complete use case infrastructure
I. Coterillo et al.
Integrating a Multisensor Mobile System in the Grid Infrastructure
71
replica is intended to be used when accessing data for simulation and processing purposes (specifically, when launching simulations jobs to the grid, when the overall speed of the process will benefit from having the database in the same local network as one of the infrastructure sites).
6 e-Infrastructure Integration As described in Sect. 2, the DORII middleware components, IE, and VCR are the key for integrating the instruments in the existent e-Infrastructure. In this application, as mentioned in Sect. 3, a uniform interface to each of the sensors is created using LavVIEW, and this interface is exposed using a Web Services interface (Fig. 7). The Instrument Element contains three Instrument Managers (divided by function) which are basically JAVA clients for the exposed sensor interface. • myWSIM. Used to control the instruments. All actions that involve sending data to the instruments, such as starting/stopping individual sensors, setting specific parameters like sampling frequency, measurement units, etc., are carried by this IM. • myDBIM. Monitors and controls the writing of data in the land station database. In this case, the database is considered as the instrument. This IM makes possible to set alarms related to values stored in the database. • myWINCHIM. Controls the winch, allowing to remotely set the depth of operation for the underwater cage.1 The Virtual Control Room is hosted at the CSIC-IFCA site in Santander, Spain, which also hosts one of the computing and storage sites composing the DORII e-Infrastructure, and is the point of access to the system for the final user. The screenshots shown in Fig. 8 present the typical view available for managing the Instrument managers inside the VCR.
7 Conclusions This chapter presents the different components and the required architecture to integrate a set of instruments in the grid e-Infrastructure by explaining the scheme currently used in the Monitoring inland waters and reservoirs application from the DORII project environmental community; it also introduces briefly the key components from the DORII middleware that makes possible to integrate the instrumentation in the e-Infrastructure. 1
This IM is under development at the moment of writing this document.
72
I. Coterillo et al.
Fig. 8 (a) myDBIM: Generating a simulation file (b) myWSIM: control Instrument Manager (b) and (c) a close caption of myDBIM monitoring of data
It is important to remark that although this contribution focuses only on the integration of the instrumentation, the VCR is not only used to control and monitor the instrumentation. The final purpose is to present available instrumental and computational resources to the user in an environment which makes it possible to easily create and manage workflows between acquired data and the Grid computational and storage resources for successfully processing that data.
Integrating a Multisensor Mobile System in the Grid Infrastructure
73
References 1. 2. 3. 4.
http://en.wikipedia.org/wiki/Scientific method WIKIPEDIA: Scientific method http://www.dorii.eu DORII: Deployment of Remote Infrastructure Instrumentation http://www.ecohydros.com Ecohydros S.L Instrument Element User Guide V2.4 Milan Prica, Andrea del Linz Sincrotrone Trieste S.C.p.A, 2009 5. http://www.gridsphere.org/gridsphere/gridsphere Gridsphere portal framework 6. http://www.modbus.org The Modbus organization 7. http://www.ni.com/labview/ National Instruments LabVIEW
Defence in Depth Strategy: A Use Case Scenario of Securing a Virtual Laboratory Marcin Adamski, Gerard Frankowski, Marcin Jerzak, Dominik Stokłosa, and Michał Rzepka
Abstract The role of IT security is continuously growing. Complicated systems have to be protected against sophisticated attacks on different technical and logical levels. Achieving the sufficient security level becomes even more difficult for distributed heterogeneous environments, involving valuable assets and data. A virtual laboratory, which enables remote access to unique scientific instruments, is a proper example of a multi-layered environment, where security is to be introduced from the project stage, based on a number of solutions. We explain the significance of security issues in the environment, describe possible threats and suggest an approach to address the necessary issues, based on one of the threats classifying models and applied in Pozna´n Supercomputing and Networking Center for building a new Kiwi Remote Instrumentation Platform.
1 Introduction The role of security in IT infrastructures is continuously increasing. Informatization makes use of computer systems and software ubiquitous. This process is especially dynamic in developing countries. New solutions, including the whole computational paradigms (e.g. grids) are now shared also among research and scientific communities throughout the world. Every new technology is carefully investigated by people who might try to abuse it. There are numerous reports and estimations concerning the total costs of IT security breaches and of a different nature and scope. The Internet Crime Complaint Center, which is the partnership between the FBI and the National White Collar Crime Center (NW3C), received in 2009 over 336 thousands complaints
M. Adamski • G. Frankowski () • M. Jerzak • D. Stokłosa • M. Rzepka Pozna´n Supercomputing and Networking Center, Poland e-mail:
[email protected];
[email protected];
[email protected];
[email protected];
[email protected] F. Davoli et al. (eds.), Remote Instrumentation for eScience and Related Aspects, DOI 10.1007/978-1-4614-0508-5 6, © Springer Science+Business Media, LLC 2012
75
76
M. Adamski et al.
with reported losses of about 560 million USD – and they addressed only US Internet frauds [1]. As for the whole “black IT market”, in 2009, it was assessed by AT&T’s Chief Security Officer Edward Amoroso that the estimated amount of cyber criminals profit annually from hacking computers (still only in the USA) may reach a billion (1012) USD [2]. There are several factors that increase the challenges that IT security may face. First of all, IT solutions are applied on numerous new application fields in order to ease life and work of new groups of users. Not all new users are sufficiently aware of IT security issues, while their new infrastructures are often initially oriented towards functionality, not security. Moreover, those infrastructures, due to new computational paradigms like grids or clouds, appear as more powerful and thus seductive for network attackers. Botnets, groups of infected computers that may be used for sending spam or conducting DDoS (distributed denial of service) attacks of a large scale, may be sold by malicious people just with 0.50 USD per bot [3], but, e.g. single infrastructures, on the example of EGEE grid [4], may consist of 150 000 CPUs, 14 000 registered users in 55 countries, almost 70 petabytes of storage space. The resources themselves are valuable, but other types of cyber criminals might look for data that are stored on them – especially in infrastructures intended for cooperation of research communities. It is assumed that security, even if once achieved, is neither a product nor a consistent state [5]. Due to systems being changed, as well as new methods of network attacks and found vulnerabilities, it should rather be considered as a process. It should be noted that some computational paradigms have their dynamics built-in. Some parts of the infrastructure may be dynamically attached to its core and detached when unnecessary. In such cases, appropriate mechanisms that assure maintaining the desired security level (see [6]) should be implemented, but in general preserving a dynamic infrastructure from network threats is a significantly harder goal to achieve. Another factor is embedded in the software itself. The level of complexity of today’s programs, often combined with the pressure of releasing the version on time, causes that it may be expected to introduce up to 20–30 software bugs (of all types, not only security vulnerabilities) in 1,000 lines of the source code (KLOC) [7]. This number may vary due to different factors; one of them may be that software produced, for example, within the confines of R&D projects may suffer from the lack of professional testing stage (with a special respect to security tests). Indeed, there are approaches that may drastically decrease the number of bugs: e.g. NASA developers were able to achieve 0.004 bugs per KLOC for the space shuttle software, but it cost 850 000 USD per that amount of the source code (with about 5 USD for industry software) [8]. Research communities cannot afford that. Obviously, there are not only threats, vulnerabilities and attackers. IT security specialists inexhaustibly make the attackers’ goal more difficult. Although it is actually unrealistic to build perfectly secure systems, in most cases this is not required. It is enough to prepare systems that are too expensive to be attacked. If a malicious hacker is required to spend more resources in order to break the system
Defence in Depth Strategy: A Use Case Scenario of Securing a Virtual Laboratory
77
being attacked than equal to his or her expected gain from the attack, the system may be assumed as secure [9]. Although this rule of security economy may be occasionally impossible to be applied (e.g. when the value of protected assets is hard to be quantified or when the attacker is driven with personal motives), it may be simplified as “the more obstacles for a malicious hacker, the safer the system”. Introducing the desired level of security is supported with numerous methodologies, approaches and security models. We will now refer to best practices of security testing [10]. It is reasonable as the potential attacker is also actually a security tester who tends not to disclose or report his or her findings, but rather take personal advantage of them. The tested application (or, more general, a problem) should first be decomposed into basic components with differentiating interfaces between the components. Having the problem decomposed, it is a reasonable approach to apply a specific threat model in order to obtain a well-structured description of possible security threats, which are easier to be assessed and prioritized. Then the structure of data exchanged between interfaces should be identified and, finally, actual tests should be started, basing on attempts of providing unexpected data. That last stage may obviously be run no earlier than the whole system is ready. There are numerous security models and concepts. Information security is often referred to with respect to three components or objectives: confidentiality, integrity and availability (CIA) [11]. However, extremely useful as an introduction into the world of IT security, the above may not be enough for the detailed security threats specification of a complicated infrastructure. One among more granular approaches to classifying computer security threats is the STRIDE model, developed by Microsoft [12]. It provides a mnemonic for security threats in six categories: spoofing of user identity, tampering with data, repudiation, information disclosure (both privacy breach or data leak), denial of service (DoS) and elevation of privileges. To follow STRIDE, it is necessary to decompose the system into relevant components, analyze each component for susceptibility to the threats and mitigate the threats. The STRIDE model has been repeatedly applied within Pozna´n Supercomputing and Networking Center, e.g. for assuring the appropriate security level of the architecture of the National Data Storage R&D project [13]. Common security rules are helpful in mitigating the threats. Two main security principles that are considered for building sufficiently secure systems are: the minimum privileges principle and the security-in depth rule. The minimum privileges rule says that every user of a computer system must be granted only privileges strictly required in order to perform expected activities. The latter rule is actually a general guideline for introducing as perfect security as necessary, especially if the system has already been decomposed onto layers and components, suitable security measures should be applied independently on all layers. Even if an attacker is able to break into the system on one of the layers, the consecutive security solutions will stop him or her a step later.
78
M. Adamski et al.
2 Virtual Laboratories Virtual laboratory (VL) is a heterogeneous distributed environment, which allows a group of scientists from different sites to work on one project [14]. As in all other laboratories, the equipment and techniques are specific for a given field of activity. Despite some interrelations with tele-immersive applications, the virtual laboratory does not presume the need of sharing the working environment. Virtual laboratories could implement different components depending on the type of experiments which will be provided within. The common parts of all virtual laboratories are: • Access via Internet by a Web Portal • Computational server – a high-performance computer which can work with large-scale simulations and data processing • Databases which contain application-specific information – such as initial simulations, bound conditions, experimental observations, client requirements or production limitations. Databases also contain distributed, application-specific resources (e.g. repositories of human genotype). The databases content should be changed automatically. Databases could also be distributed. It should be presumed that databases will contain a large amount of information • Scientific equipment connected with the computational networks. For example, it could be data from satellites, earthquake detectors, air pollution detectors and astronomical equipment (like the distributed astronomical research program which is provided by the National Radio Astronomy Observatory [15]) • Collaboration and communication tools, such as chat, audio and video conferences • Applications. Each virtual laboratory is built on specific software which allows simulating processes, data analysis or visualization There are several important aspects which have to be taken into consideration while creating a virtual laboratory system. Delay is a critical parameter for many applications. This is the reason why a computational centre which implements the VL idea should have access to a high-throughput network. It will be helpful to connect the tasks scheduling system with the throughput reservation services. Next critical parameters are multi-cast protocols and technology reliability in collaboration with specific VL experiments where people, resources and computations are highly distributed [16, 17]. In these experiments, data streams could be divided into voice, video, computational elements and huge portions of simulation and visualization data which will be delivered in real-time from the scientific equipment. Applications should allow access to data from several heterogeneous sources. For example, such information can come from real experiments or computational simulations. The main source of information will be the mass data storage systems – they are especially important for bioinformatics tasks [16]. These data storage systems could be solved on the basis of dedicated databases implemented in national supercomputing centres and could also include small personal databases from workstations and PC. Because of virtual laboratory, data processing will be controlled by an individual scientist or a distributed research team, working in their own laboratories.
Defence in Depth Strategy: A Use Case Scenario of Securing a Virtual Laboratory
79
We will discuss a virtual laboratory (VLab) created in Pozna´n Supercomputing and Networking Center. The VLab [18] is a framework architecture aiming to provide remote access to various kinds of unique and therefore expensive scientific laboratory equipments. The main goal of the VLab was to embed remote facilities in grid environments, closely supporting the experimental processes with the computing and visualization infrastructure. Another purpose concerned the concept of Workflow Management System (WfMS) [19]. WfMS allows defining the measurement workflow in a way which is convenient for the user: from pre-processing, through executing the experiment, to the post-processing and visualization tasks. The virtual laboratory is not only a set of mechanisms to submit, monitor and execute jobs. It is also an opportunity to give access to the resources of the digital library, communication and e-Learning systems. The VLab project introduced two different instruments: NMR spectroscope and radio telescope. The VLab system has been designed as a generic system to serve different instrumentation from many domains. However, such an approach has one significant disadvantage: limited functionality. It is not possible to create one common interface for every type of equipment. The experience and lessons learned from the VLab project allowed us to propose a new approach to building remote instrumentation systems: the Kiwi Remote Instrumentation Platform [20]. It has been proposed to create it as a skeleton for the future systems. This platform consists of several components with well-defined functionalities and communication interfaces. The carefully selected components from the Kiwi Platform will become a base for building new remote instrumentation systems. The Platform provides a set of components to control and manage scientific equipment or sensors like cameras, weather, air pollution and water flow sensors and others. All the equipment connected to the Kiwi Platform can be controlled remotely with one unique user interface. It also allows users to design and run the so-called observation workflows. It is possible to set up a sequence of operations starting from data acquisition, data processing and finally visualization. This workflow can be launched periodically by the Kiwi Workflow Manager component with a desired frequency and time. For instance, the system administrator can plan a workflow with a camera taking pictures twice a day. The device can be set up so that several points of a scene are shot (Fig. 1). The interactive e-Learning system for environmental science teaching is an example of a system built on top of the Kiwi Platform. The main objective of the project is to develop an e-Learning interactive platform for areas like water management, air pollution, deadwood or phenology (Fig. 2). In order to make the study process more interesting, the Platform will be equipped with dynamic content like data collected by measurement devices and sensors or high-quality images from the observation scenes. Kiwi is also used to control and manage all the sensors deployed in the various locations in the forest near Pozna´n. The basic sensor suite measures wind speed and direction, pressure, temperature, relative humidity and precipitation, etc. Optional sensors can be added to measure, e.g. water level, soil/water temperature, global and net solar radiation. It is believed that a successful e-Learning Platform should provide means to add new devices and sensors to the existing infrastructure.
80
Fig. 1 Kiwi – remotely controlled DSLR camera
Fig. 2 A sample VL based on the Kiwi platform
M. Adamski et al.
Defence in Depth Strategy: A Use Case Scenario of Securing a Virtual Laboratory
81
An automated, remotely controlled system for fauna and flora observations has been introduced. The system is based on the Kiwi Platform. High-quality DSLR photo cameras are used for a scene comprehensive view. Photo cameras can be set up to take high-resolution pictures with a desired frequency and time. It is also possible to set up “points of interest” within an observation scene with a desired frequency.
3 Security Issues in Virtual Laboratories Besides the strict technical problems, virtual laboratories involve a number of security problems that are associated with informatization of access to scientific instruments. First of all, virtual laboratories are usually heterogeneous environments. They are usually not built from scratch, but rather based on providing remote access to already existing scientific devices. Those instruments usually have a local IT infrastructure (in most cases – a local control workstation, a storage device or measures for manipulating hardware). This infrastructure is often relatively aged and, additionally, may completely vary among controlled devices. Therefore, the infrastructure itself may not contain any built-in security solutions (as previously, IT security was not considered) or those measures may be only general and insufficient. Moreover, adjusting new security solutions to protect those aged infrastructures may be time-consuming and expensive. On the other hand, a solution able to securing one device may not be in use for the other due to their heterogeneity. Thus, applying dedicated security solutions for the whole virtual laboratory will require even more resources. Another obstacle is associated with the human factor. People often have difficulties in accepting and understanding thorough changes in their working environment, especially when those changes are not directly associated with their research area. To be more concrete, when IT solutions are introduced, they are often assumed as hostile and incomprehensible (that is why it is so significant for them to be userfriendly and fit the needs of their users, which is not always the case). Moreover, for new users of IT infrastructures, the issue of IT security is usually completely or partially new. The above may be applied especially in communities that start using networking solutions. Virtual laboratories often merge scientific devices from different countries and continents. If the researchers (and/or the administrators) are not sufficiently educated with appropriate trainings, they will not be convinced to use encrypted emails, apply strong passwords, keep their networking workstations up-to-date, avoid clicking malicious links or browsing Web pages that may contain a dangerous content. We have experienced cases where it was impossible to convince an abroad research centre to use, for example, a secure way of sending passwords through the Internet. Although trainings and applying security solutions cost effort, they are unavoidable from the point of view of security. However, IT security specialists cannot resolve this problem themselves as this is not purely technical.
82
M. Adamski et al.
As for technical threats, the goal of virtual laboratories is to provide access to rare and expensive scientific instruments. Therefore, an attacker might abuse the control infrastructure of a device and even damage it (which is a real danger for, e.g., the radio telescope in the observatory located in Piwnice near Toru´n – issuing a specific sequence of commands may be destructive for its servomotors). Although damaging a radio telescope does not result in any direct gain for the attacker, his or her goal may be to cause losses for the research centre. Technically, and with respect to the STRIDE model, it would be a classical DoS attack (not directly against the data, but the equipment). Several scenarios (like monitoring a forest) assume that the instruments are left unattended, so pure physical security matters as well. The scientific instruments are not the only valuable part of the infrastructure. Grids are often used in the virtual laboratories concept in order to increase the computational power of the whole infrastructure, which is required to be extremely large in applications like nuclear physics or weather modelling. Those resources, when successfully attacked, may be used by network attackers, e.g. to conduct DDoS attacks. A botnet built from a grid is attractive for the attackers, as a great number of different IP addresses may be used during the attack (thus, blocking them on firewalls appears overwhelming) and, moreover, each host of the botnet contributes to the size of the used network bandwidth. The computational power of the grid, especially supported with its storage facilities, may also be abused, e.g. for cracking passwords. The concept of virtual laboratories is inseparably associated with research and science. In the age of competitiveness and industrial espionage, the research results may be a significantly more attractive goal than resources or hardware. When not addressed by security solutions, they will be in danger (as well as other data stored within the same infrastructures, e.g. password hashes, personal records and internal databases). Finally, as every IT infrastructure, a virtual laboratory may suffer from all types of network attacks that are spreading through the Internet. VLs are accessed by Web portals, therefore standard Web application threats apply (see, e.g. [21] for more information). Although, as an entry point to the VL structure, a Web access portal appears as the most apparent vector of attack, it must be remembered that another opportunity for an attacker may be a misconfigured server, offering unnecessary and/or insecure services – like Telnet, FTP or even SSH with account names that are easy to be guessed and are protected with weak passwords.
4 Applying Defence In-Depth Strategy As it has already been discussed, in order to cope with protecting a sophisticated (large, distributed, heterogeneous) IT infrastructure, it should be logically decomposed into smaller, distinct subsystems. All subsystems should then be secured appropriately, and additionally interfaces between the subsystems have to be considered.
Defence in Depth Strategy: A Use Case Scenario of Securing a Virtual Laboratory
83
Fig. 3 Decomposition of a virtual laboratory into smaller layers
According to the general structure of virtual laboratories, we will decompose it into separate layers: instruments layer, VLAB platform and access platform. The first and last layers are the most distinct, while the VLAB platform, being a consistent subsystem, still has a relatively sophisticated structure where subcomponents may be differentiated with interactions between them. Therefore, the VLAB platform has been divided into four lower-level layers, addressing systems, network, computational resources and digital libraries. Figure 3 shows the conceptual decomposition of the infrastructure. What is worth noting, there may exist security solutions that address even more than one logical layer (examples of such systems are described later).
4.1 Instruments Layer The scientific instruments accessible in virtual laboratories are rare; otherwise they would not have to be accessible in that manner. As it has been already mentioned, one specific threat is damaging a scientific device (either accidentally or intentionally). The instruments are usually remotely accessible via a dedicated interface. There are scenarios (e.g. for the NMR use case), where the issued commands may have to be manually applied by a local operator of the device. This decreases the threat, although the local operator has to be exactly aware, which commands or parameters may be inappropriate for the device. However, in other scenarios (radio astronomy), the interface may allow to directly operate the device. In the majority of cases, the interface of the instrument does not contain any built-in remote access facilities. The interfaces are usually complicated for inexperienced users. Moreover, they are closely tied to the instrument and it is unrealistic or unjustified to build alternative ones. Remote access
84
M. Adamski et al.
may thus be obtained by controlling the local management station, not the device itself (e.g. by a virtual desktop). Therefore, at least the following threats must be considered: • The direct interface of the device may be too complicated for a remote user and an inappropriate command may be issued by accident (not all interfaces contain appropriate software limitations prohibiting that) • A malicious remote user may abuse the remote control interface in order to gain control over an inappropriately (insecurely) configured management station, thus, e.g. intercepting the research results of others, leaving a backdoor or switching off the device • A malicious attacker may intercept the network traffic and read the sent commands or modify them to be inappropriate; the legitimate user would then be accused of malicious activities In scenarios applied several years ago within the VLAB project, one of the solutions applied based on RealVNC software, supported with Zeebeedee Secure Tunnel software for encrypting the communication channel. Other remote control software may be used as well, e.g. UltraVNC that is free of charge, and – supported with an additional plug-in – provides encryption as well. The above, however, protects only against intercepting and potentially influencing the network traffic. Protecting against abusing the functionality of remote control software is a complex task. It has to be started with an appropriate selection of remote control application; first, it should provide functionality of controlling only a single application (the device interface). Unfortunately, this will only make the attack more difficult – a single Windows desktop application may offer functionality that might be abused: e.g. by using File j Open main menu commands one could browse directories and delete system files. However, restricting remote control to a single application conforms with minimum privileges principle and thus should be applied. The device management station itself should be configured according to the mentioned rule. For example, the remote control application server should be run with minimum possible privileges (never under the administration account!). Significant operating system files (and also those of other users) must not be accessible for that account. In addition, appropriate and detailed logging and accounting facilities must be implemented. They might be helpful in detecting some attacks in their early stage and then be suitable for forensics. There still remains a problem of denying potentially dangerous parameters or sequences of commands issued to the device. Without implementing dedicated solutions, however, it may be only partially solved. For scenarios with access portals (or any other interface between VLAB users and instruments), the entered data should be carefully filtered on the server side. The sets of commands or values that may damage the device must be known for security specialists and the developers (local instrument operators must be interviewed and the values should be put into an appendix to the suitable security policy). In addition, an intrusion detection system (IDS) might be applied.
Defence in Depth Strategy: A Use Case Scenario of Securing a Virtual Laboratory
85
4.2 VLAB Platform Layer This layer covers components that are directly associated with the core of the VL concept: remote access to scientific devices and maintaining the research results. The instruments themselves are not included as they function also separately and therefore should be treated as a distinct layer. However, the described functionality is compound enough to be divided into separate subcomponents. Those smaller modules will correspond with different facets of security – both in terms of the VL structure and functionality: systems, communication channels, authorization policies and digital libraries (or more general: storing research data). This structure is mainly based on our experience concerning conducted projects oriented towards virtual laboratories.
4.2.1 Systems and Communications Security issues associated with those subcomponents are the most general of all discussed in this paper. The virtual laboratory application does not introduce any specific considerations here, and therefore the provided description will be exceptionally brief. However, those issues still have to be remembered and addressed during the process of building the whole infrastructure. Each system (server, computer, network hardware unit, etc.) has to be secured according to known security best practices for the given system type and provided functionality. Security measures must be introduced on the system itself – its environment must not be trusted. Even if the environment provides sufficient security measures, it may change in the future. In the case of the Therac 25 device [22], in previous versions of the device, the controlling software was supported with many hardware safety interlocks. Unfortunately, after releasing a new version of Therac, the software was not sufficiently improved to handle the whole control over the system, which finally resulted in cases of radiation overdose (including at least five lethal cases) [23]. As the systems are secured, interactions (interfaces) between them must be taken into consideration, especially addressing the protection of the public communication channels. An appropriate level of confidentiality must be assured (encrypting the network traffic where necessary) and proper authentication mechanisms must be provided (e.g. PKI based on X.509 certificates). It is recommended that mutual authentication is applied. Server to client authentication, which is mandatory [24], protects the user against connecting to an inappropriate remote system (the owner of which, could, e.g. intercept the parameters sent to the scientific device, which may be a precious piece of information on research being conducted by the user). On the other hand, client to server authentication (which is optional and thus often omitted) would protect the management station: it would facilitate logging and accounting, and obstruct attacks from unknown sources (e.g. unauthorized “switch off” requests). There are appropriate open solutions (e.g. OpenSSL [25], OpenCA
86
M. Adamski et al.
[26]) that may be used without increasing the price of the whole infrastructure (although, naturally, effort is required to install and configure them properly). Finally, each system (and also the infrastructure as a whole) has to be the subject of a security audit, conducted by independent security specialists. “Independent” means there that the auditors should not have been involved with the project or implementation of the analyzed system – not necessarily that they must not be from the same company or research unit.
4.2.2 Identities, Authorization and Grid In order to work properly, virtual laboratories must provide sufficient identity management facilities (authorization and authentication mechanisms). This does not only mean differentiating between distinct users and their activities, but also implementation of the minimum privileges principle: differentiating between separate roles in the infrastructure, and granting those roles only strictly necessary rights. During our work with virtual laboratories, the applied scenarios usually have assumed utilization of the grid computational model, especially to enable distributed processing of large amounts of data. There are several specific authorization mechanisms and scenarios used in grids, which are discussed below. We have developed three main use cases defined for virtual laboratories, all strongly connected to a grid infrastructure. It implicates using a sophisticated authorization policy, and therefore usually an extended authorization system. Below connections between use cases (based on their layered structure) and authorization policies are explained in details. For the first use case – the Spectroscopy NMR – three separated functional layers are defined: Access, Grid and Resources. This scenario uses an external authorization system: the Grid Authorization Service (GAS) [27]. The access layer provides necessary interaction facilities with the system for its users. The internal security model has been prepared for this layer, which contains items like: laboratory groups (groups of users), user roles (superadmin/supervisor, admin, common user) and objects (sets of devices which are accessible for a specific laboratory group). All such policies are stored in the VL server database. In turn, the next layer – the grid layer – is connected to GAS. This service is responsible for storing grid polices and for evaluating an authorization decision based on known users, grid services and resource requests. Figure 4 presents a sample authorization policy for the spectrometry laboratory. Two object definitions are created: laboratory and device. Several instances of these definitions are shown with a set of operations which can be performed on the object. Each instance of the object has its unique name. In addition, several groups and subgroups of users are defined in one structure, making it possible to distinguish user rights to a specific device. The GAS policy stores authorization knowledge as triplets: “user A can perform operation Op on object Ob”. The last layer – resources – has an internal security policy, which is compared with the access rights of the operation system to which a device is connected. Access is granted based on local
Defence in Depth Strategy: A Use Case Scenario of Securing a Virtual Laboratory
87
Fig. 4 Authorization policy for NMR spectrometry scenario
user rights to run applications. The first scenario treats authorization processes in an especially serious way, i.e. on each layer authorization is one of the most critical security aspects. One disadvantage that can be found covers missing communication between layers from the authorization perspective. It results in separate management actions on each layer and a need of manual synchronization of policies between layers. The second use case, the EXPReS Radioastronomy [28], treats authorization in a different way. It is based on trust between users in a limited community. It covers three functional layers: Portal (an internal, Web-based system), Platform and Devices. As it is mentioned above, full trust inside the community is required. The platform (grid) layer is organized as a separate computing cluster, accessed only by the community members. The most critical layer – Devices – makes it possible to use instruments like telescopes by the community. Each action on such a device requires a manual confirmation to be made by the telescope operator. Due to the fact that it is possible to damage a radio telescope by executing a specific set of operations, this scenario is limited only to a closed and trusted community. The authorization policies are similar for each community member. There is only
88
M. Adamski et al.
Fig. 5 Authorization policy for the forest environment scenario
one authorization advantage important for this scenario: each operation or set of operations on a device has to be accepted (and thus verified) by the device operator as even a random set of operations can damage the device. As the last use case, the forest environment is described, applied to support a multi-media nature lesson in a school. This scenario takes into consideration several authorization aspects. Therefore, it can be assumed as a proper reference solution from the authorization perspective. This type of solution will also have a multilayered architecture, which makes it possible to prepare interaction between end users and devices. The extended authorization system will be used not only by one layer but also by the whole system. The authorization system will contain several services connected together (for exchanging policy processes). In addition, on the lower layers (resource, grid layers), it will be possible to put the policy inside – e.g. proxy certificates for the components not connected directly to the authorization system. The policy for the topmost layer (portal layer) will be defined inside the portal in the RBAC (role-based access control) [29] model and automatically put to the authorization system which will facilitate easy managing the authorization polices. There are several portal implementations which can be used in this scenario: an example to be recommended as containing extended authorization functionality may be the Liferay portal with its role management facilities [30]. A policy similar to the first described use case can be defined: for a specific user or for a group of users (e.g. a virtual organization). Each device and organization will have its own definition and a set of operations which may be performed in its context. Also, as shown in Fig. 5, it will be possible to define complex authorization scenarios which will make the authorization decision dependent on results obtained from the previous steps or other modules (e.g. a pupil will be able to enter the next stage of work only after he or she has completed the previous one).
Defence in Depth Strategy: A Use Case Scenario of Securing a Virtual Laboratory
89
However, authorization and roles are not only security-related facets of the grid. At least two other threats should be considered (excluding a standard requirement of properly securing each node of the grid). The first of them is the opportunity of executing applications on remote grid nodes. Scenarios where a grid user may send his or her own application to be executed should be absolutely excluded (the VLab project did not accept the above). Moreover, even if the user is merely able to provide input parameters and/or data to one application out of a strictly defined set, those parameters should be exceptionally carefully sanitized. Otherwise a malicious user could be able to execute remote code on grid nodes. For example, if the user parameters were directly appended to a shell script call or application, one could provide legitimate parameters, followed by a semicolon metacharacter (;) and a next, malicious command to be executed. The other issue, although probably significant only in a limited set of scenarios, is a potential indirect information disclosure. Grid infrastructures base on the distributed computing model, where one large task is divided into subtasks (jobs) that are executed on different systems. It is potentially possible that a single job, received by a node controlled by potentially distrusted virtual organization, may reveal sensitive information about the research (e.g. symbols of investigated chemical substances or coordinates of reviewed space area). Information about the fact of conducting specified research may also be confidential for a researcher. In such cases, it must be considered whether a kind of encoding of sensitive data may be applied; if not, leaving the distributed computational model should be considered. 4.2.3 Research Data and Digital Libraries Virtual laboratories are built in order to provide access to valuable scientific devices also for those research centres that cannot afford an expensive infrastructure. Therefore, especially in a distributed computational model of grid, it is possible that VL users will tend to store their research results in external repositories (digital libraries), at least the part that is going to be finally published (but not necessarily expected to be accessible during the research itself). Such type of repositories usually are an excellent solution for (intended) sharing the research results: e.g. a researcher may check whether similar work had already been conducted or use some earlier results as a reference. The described situation requires applying suitable security solutions, although none of them is especially related to digital libraries. Authorization policies should be applied to differentiate between data that are accessible to anyone (world readable, potentially also world writable), data that may be available by members of the same research group (or any other defined subset of users) and confidential data (readable and writable only by their owner). In addition, strict accounting and logging policies must be applied. Authorization solutions based on access control lists (ACLs) or RBAC should meet those requirements. Communication channels should be protected against eavesdropping (by encrypting network connections) and
90
M. Adamski et al.
impersonation (e.g. by an appropriate PKI implementation). Finally, the systems that compose the digital library must undergo an independent security audit as well as the computational nodes.
4.3 Access Platform The access platform is a key component of virtual laboratory infrastructure in terms of security, because this is where the users enter their data in order to conduct their research. As it has been mentioned, Web technologies are commonly used to provide access to VLs. This is an excellent approach in terms of interoperability and accessibility of rare instruments. Web standards are relatively well established and exceptionally widespread. On the other hand, the basic Web standards (like HTTP) were developed several years ago and, although relatively simple, are not securityaware. Therefore, additional security measures have to be applied, addressing Web applications vulnerabilities. The Web access platform (an example of which is the Kiwi Remote Instrumentation Platform access portal) is intended to be deployed for numerous users and communities. Therefore, “security through obscurity” has to be especially excluded from considerations. On the other hand, opening the platform to the world will expose the VL to more potential attackers. There are many types of well-known security vulnerabilities of Web applications (see [21]). Addressing them, it would be difficult to differentiate virtual laboratories from other types of portals and applications. However, it is sure that the opportunity of identity stealing and compromising remote systems has to be emphasized. The former may lead to impersonation and stealing research results of another user, while the latter will result in total taking control over users, instruments and data (the whole infrastructure may also be used for further attacks). HTTP is a stateless protocol which adds great simplicity to implement applications based on it, but requires additional mechanisms in order to differentiate between distinct users. Usually, HTTP session facilities are used, with information on users stored in files or databases on the server side, and cookies (which may be understood as simple, short text files) are maintained by the browsers on the client side. Although the whole mechanism is widely understood, in many cases it is implemented in its simplest form. Session state is stored in files on the server side and the cookies are not associated with any particular attributes. In order to implement an HTTP session not only for differentiating between the users, but also to increase the security level of the application, at least several good practices should be applied. First, the session identifier should be random and long enough (e.g. current PHP implementations already assure that). The session lifetime should be, on the contrary, short enough – that is, the session should expire within a reasonable amount of time. The amount should depend on a particular case – for sensitive applications (like, e.g. remote administration of the radio telescope) it should be no longer than 5–10 min. To loosen the limitations, a sliding expiration
Defence in Depth Strategy: A Use Case Scenario of Securing a Virtual Laboratory
91
window may be considered (the session lifetime counts not from its beginning, but from the last activity of a user). Logging out by a user must absolutely cause immediate session invalidation on the server side. As an additional security measure, association of a session identifier with the IP address of the user might be considered. For applications like Polish e-commerce portals it appeared as rarely implemented (only 2 out of 50 investigated [31]). It may be partially justified with the willingness to help mobile users; however, in scientific communities users are rather expected to have a static IP within a single session expiration time interval (possibly with an exception of a remote system administrator who is travelling while handling a user request). To make the compromise between functionality and security even better, the association between session ID and IP address may not cause the session expiration, but, e.g. only increase the inspection level of these particular user activities by an IDS. Furthermore, additional attributes should be applied for cookies: httponly and secure [32]. Although not all versions of popular browsers (especially the more aged ones) support them, applying the attributes will never make any harm. The httponly attribute disables access to cookies from active script languages (which is the main idea of XSS attacks), while secure enables this access only for a secure HTTPS protocol. The latter is equivalent to spending an additional effort devoted to providing HTTPS services, but for applications that deal with user identities (which is apparently desired for virtual laboratories) this should be one of the very first requirements to be defined. To be finished with a session and cookies, it is recommended to consider developing a custom session handling mechanism on the server side. Instead of simply storing the session data in file system objects, database facilities should rather be used. For example, the PHP scripting language provides highly configurable custom session handling routines (session set save handler library function plus additional callback functions invoked when necessary) that allow, e.g. to implement additional logging functionality or IP address verification [33]. It has been noted that applying security attributes to cookies allows to protect against Cross Site Scripting attacks. This is true, although the Web application developers must be aware that this is only partial and supplementary protection. The main countermeasure against not only XSS attacks, but also many other types (e.g. SQL Injection, Cross Site Request Forgery and Command Injection) is careful sanitization of all user input data. It should never be assumed that the infrastructure being built will never be attacked (or has never been attacked). The detailed explanations of those (and several others) excuses that may be intended to justify little emphasis put on security, may be found in [34]. Functionality is crucial for the application users, but it has to be supported with built-in routines that increase the security level of the software. The designers of an application must include data sanitization functions and the developers must implement them properly. Sanitizing of input data (which, in general, must be treated as malicious) allows to avoid the majority of common Web attacks. Moreover, sanitization cannot be limited to the client side. It is useful for
92
M. Adamski et al.
the user convenience (e.g. to point out that an entered value is improperly formatted or lacking), but it may be easily omitted with a proxy allowing to modify a sent HTTP packet. Several approaches may be applied in data filtering. Blacklisting bases on enumerating all known malicious sequences, the existence of which is verified in the input data. If at least one sequence is found, the data will be rejected, otherwise – accepted. This approach is easier to apply with data that are hard to be structured (e.g. comments on a forum where a number of HTML tags are allowed), but may become inefficient as the number of malicious strings grows, e.g. because a new type of network attacks is introduced. Whitelisting appears to be a better approach: the definition of properly formatted data is known (e.g. a postal code in Poland: two digits – a dash – three digits) and every input string not conforming the pattern is rejected. This type of input data processing is faster, more convenient and easier to be implemented; however, not all sorts of data fit into it. Regular expressions are a great help for the developer. They may be used to build the definition of both accepted and rejected types of data. However, even those definitions must be designed with care as there are attacks aimed to DoS regular expression parsers [35]. Finally, even if the format of the data is proper, their value is still to be investigated. For instance, if a three-digit number denoted the angle at which a radiotelescope should be turned, one should consider if there is a risk of damage if a malicious user enters, say, 720. Probably, it should be assured that there is an appropriate sanitization routine on the server side (at the device management station). A detailed description how to defend against all particular types of attacks is beyond the scope of this work. The last security issue associated with the Web access platform that will be mentioned here is information disclosure. Network attackers, before actually trying to compromise the system, attempt to gather as much information about it as possible. Therefore, all misconfigurations that cause leaking even the smallest pieces of information (like server signatures) may be harmful. Other examples are: switching of indexing directories (which allows to see the directory contents in the user browser), configuring your scripting language to display error messages in generated pages or leaving unnecessary contents under the directories with names that are easy to be guessed (like old, test, backup – or even leaving infamous phpinfo output under /.phpinfo.php). More detailed information about avoiding common Web attacks may be found in numerous articles, papers and portals. The OWASP project may be recommended as a general and broad source of information, and as a condensed source of useful advices the best security practices set for EGEE-II/EGEE-III project may be suggested [36]. And once again, as the final stage of securing the access portal, a complex security audit should be conducted. The recommended way of the audit would be a combination of penetration tests and review of the configuration and source code.
Defence in Depth Strategy: A Use Case Scenario of Securing a Virtual Laboratory
93
4.4 Multi-layer Security Components In the previous sections, a virtual laboratory has been decomposed into layers and security measures have been suggested for all layers. In this section, we would like to present additional security systems whose range may cover several layers. An intrusion detection system (MetaIDS) and the SARA solution for administration, inventory and automatic reporting are described. Both systems have been designed and are being implemented and deployed in Pozna´n Supercomputing and Networking Center within the confines of national research projects.
4.4.1 MetaIDS: Intrusion Detection System To deal with the problem of dynamically changing security conditions, often IDSs are used. This additional security layer may react to the most recent threats in areas where standard security countermeasures could not be effectively applied. An IDS will rather detect and report malicious behaviour of an attacker than block the preconditions of the attack, but apparently this increases the security level of the infrastructure being protected. Several usage scenarios of an IDS in the VL structure may be developed. In the previous chapters, we have described many potential threats and attack scenarios. As mentioned before, the attack vector is broad and includes numerous entry points. In order to monitor all of them properly, an infrastructure administrator should constantly perform a log analysis for many applications simultaneously. The logfiles are stored in separate files, located all over the hard drive (or even on several systems). This complicates the task and makes information correlation extremely difficult and time consuming. An IDS may help the administrator to tackle with a large amount of distributed information, and – moreover – to recognize patterns of attack, often complicated to be detected manually. In addition, the IDS analyzes data all the time and may issue an e-mail or SMS alert in an emergency case. By then all the important data regarding a particular incident are normalized and stored in one place, making it really easy to maintain (e.g. for further forensics purposes). A standard host-based IDS works on a single machine and therefore is unable to detect more sophisticated attacks, which could have been performed using many machines located in different parts of a corporate network or a distributed environment like a VL. For instance, an IDS installed on a Samba server will not be able to correlate events and detect the following attack: 1. Virtual laboratory Web access server scanned from a particular IP address. 2. Malicious network traffic sent to the VL Web access server. 3. A successful login to the Samba server from the VL Web access server. Port scanning is an activity which may be considered malicious, but not necessarily implicates a successful break in. Considering every port scanning event as a serious alert, an administrator would end up with numerous false positives and much effort
94
M. Adamski et al.
spent on verifying them. A successful login to the Samba server is not suspicious at all, but once a machine is marked as “compromised” (in this case – VL Web access server), all its outgoing connections, whether it is a Samba Server, an FTP Server or any other machine, are considered malicious and should be immediately investigated. The distributed IDS “MetaIDS”, developed in Pozna´n Supercomputing and Networking Center within the confines of the Polish Platform for Homeland Security project [37], is able to detect intrusions and intrusion attempts based on information from many resources located all over the network being protected. In order to successfully monitor a particular environment – in this case a VL – it is crucial to link several information sources. All the important elements that work within the monitored infrastructure and store data in log files should be plugged in to MetaIDS [38]. Because MetaIDS has a modular architecture, it is possible to implement, e.g., an additional module for a new and unique log source that was not included originally [38]. Basically, the system consists of a central management and analysis server and a set of agents installed on protected systems. The agents send specified information to the server, where all data are analyzed and appropriate decisions are undertaken. An agent (also called “sensor”) itself is a lightweight, stateless monitoring program, working on an unprivileged account. The detailed explanations are out of the scope of this chapter, and may be obtained, e.g., in [38]. An analysis and combining different events are described there as well. What is especially significant for distributed infrastructures is that MetaIDS offers facilities of cooperation with other IDSs. Therefore, it appears reasonable to implement it in distributed infrastructures wherein some parts are already protected with another IDS (e.g. when a new subnet with another scientific device is attached to the VL structure). That approach will assure a better coverage of the overall attack vector. The communication may be bidirectional: IDS ! MetaIDS and MetaIDS ! IDS. In the first case (see Fig. 6), MetaIDS reads events detected by another IDS, understood there as a “rich sensor”. That type of configuration is especially valuable, when MetaIDS cooperates with a network-based IDS like Snort [39]. MetaIDS sensors monitor events occurring in services running on servers, so acting like a host-based IDS. Snort monitors events on a lower level, and therefore it is able to detect other types of events which are not available for host-based systems. By combining information from host and network sensors, MetaIDS may detect significantly more compound attacks. A special MetaIDS module monitoring another IDS is required that filters generated events and translates them to the form used by MetaIDS. Currently, events reported from Snort [39], Ossec [40] and any other IDS supporting events in IDMEF [41] format (e.g. Prelude [42]) may be read. Communication in the opposite direction is also possible. In such case, MetaIDS stores detected incidents in a file, using the IDMEF format. The foreign IDS may read that file and use it for further analysis. If there is a need to “push” events
Defence in Depth Strategy: A Use Case Scenario of Securing a Virtual Laboratory
95
Fig. 6 MetaIDS in cooperation with another IDS system
directly to the foreign IDS (without the intermediate IDMEF file), it is possible to configure a MetaIDS action, which will automatically send an IDMEF message using an external program. Applying MetaIDS as an additional protecting mechanism for virtual laboratories, besides its ability to protect or participate in protecting distributed environments, may also be based on preparing dedicated modules for sensors that monitor particular VL systems. For instance, it may be hard to apply external protection software on outdated systems that appear in research centres as device management stations. Preparing a simple agent that will control logs or system calls may be a surrogate. A similar sensor could also, e.g., intercept values of parameters directed to the scientific device and check whether they do not become dangerous for the instrument. 4.4.2 SARA System for Support of Inventory and Monitoring Because security is a process, in distributed and heterogeneous environments like VLs, it is especially meaningful to maintain appropriate versions of software products and suitable, hardened configurations on all currently available nodes as even a single vulnerable system may act as a broad input for an attacker. Dynamic environments, including VLs, are especially affected with this issue; as new nodes (computers, networking devices, laboratory equipment) may appear within the infrastructure, they may introduce new attack vectors. Built-in security mechanisms seem to be not enough for assuring the desired security level (again, with a special emphasis on heterogeneous environments). Therefore, as an additional security measure, we would like to propose a system for constant monitoring of a dynamic infrastructure with respect to individual
96
M. Adamski et al.
security of its subsystems. Both active and passive control mechanisms are necessary, and on different granularity levels. A fast reaction that is able to inform security officers about found vulnerabilities is required. The control should cover known (publicly disclosed) security holes and – whenever possible – take into account data taken from repositories of software vendors. On the other hand, it seems to be extremely useful to maintain a detailed registry of systems used within the infrastructure. The structure and rules managing that registry must anyway be based on a consistent and detailed security policy. The problem of an accurate, formal and consistent description of security state is itself non-trivial. The provided information should be appropriately structured but also easy to be understood and exchanged (e.g. between different organizations within the virtual laboratory). There are a number of standards, relatively new or even still being established, specifications and repositories that may be utilized for those purposes. CVE [43], CPE [44], CVSS [45] or NVD [46] may serve as examples. Currently, it is difficult to find a system offering the ability to control all systems within a highly heterogeneous infrastructure. The virtual laboratories are specific: the systems which are in use may be outdated, due to software compatibility issues. Also, in such an environment, uncommon operating systems may exist. Nagios software [47] with its plug-ins allows to monitor systems, but it requires installation of plug-ins on each of them. An additional difficulty for the monitoring systems is the lack of agents (local programs controlling security on the nodes) for certain operating systems. The problem of agents was partially eliminated in the software Pakiti [48]; however, it uses vulnerability repositories provided by some of the operating system vendors, which can significantly restrict the application area. The SARA system leverages a slightly different approach to the problem of correlation of known security vulnerabilities with the system configurations. A universal solution helps to minimize the significance of the human factor in controlling the security level under changing conditions (i.e. the infrastructure contents and a set of known attacks). SARA combines information on vulnerabilities stored with the help of several mentioned standards in the National Vulnerability Database (which is updated every 2 h, and SARA regularly checks for those updates). The standards allow to determine which systems may be affected and how critical the known vulnerabilities are. The system automatically issues alerts to administrators and security officers. Currently, SARA requires a certain amount of initial human work: information on systems has to be inserted and maintained manually. This overhead will be minimized in the future versions, although probably it will never be avoided at all, especially in environments containing heterogeneous, uncommon and outdated systems (as for those machines there will be no solution reporting installed software versions automatically). On the other hand, the system is an excellent tool for inventory with determining the physical location and responsibility of each system. Contrary to the mentioned existing solutions, SARA may also handle not only servers, but also e.g., network devices (Fig. 7).
Defence in Depth Strategy: A Use Case Scenario of Securing a Virtual Laboratory
97
Fig. 7 SARA system general architecture overview
The current version of the SARA system has been accepted for use in the national project Pl-Grid [49] as an optional solution for appliance in various clusters. Furthermore, there are outlined promising visions of development, expanding its capabilities and making easier the interaction with the users, as well as other security systems.
4.5 Security Considerations on Dynamic and Heterogeneous VLAB Environments As it has already been mentioned, security is actually dynamic even in a static, stable environment, because the knowledge on security of particular systems or software is dynamic. The reason why grid paradigm introduces another dimension of this dynamism is because new systems may be attached to the protected environment. In the case of a VLAB environment, the laboratory may be, e.g. extended with a new scientific device with an appropriate access point. The new element of the environment should be incorporated by the security systems as well – at least to a certain degree.
98
M. Adamski et al.
The KIWI remote instrumentation platform is intended to be accessible from the Internet (although this is not a problem to provide the installation limited to a corporate or a campus network). However, the part of the environment that holds the devices should be granted a higher trust level than an arbitrary computer of a remote user of the KIWI platform. Therefore this “internal” part, which may also be dynamic, must be strongly protected (anyway, the authors of the platform have no impact on what level of security is applied by its users). If the trusted part of a virtual laboratory is going to be extended, a trade-off problem between security and usability (understood as the easiness of attaching the new systems, potentially only temporally) must be resolved. Especially, it may be not allowed or not applicable to introduce significant (or even any) changes in the attached systems, like installing Nagios plug-ins or requiring using specific properties of the infrastructure or applying defined attachment procedures as proposed in the Clusterix grid project [20]. The multi-layer security components, described in the Sect. 4.4, address this issue in a different way. It must be noted that the deployment of MetaIDS in the new part of the infrastructure may be found unacceptable by the owner of this part. In addition, a plug-in for the particular operating system may not be available (however, installing only plug-ins on some additional systems to extend the set of computers protected by the current MetaIDS server would be relatively fast and easy). On the other hand, it is obvious that MetaIDS will be immediately able to detect known attacks originating from the attached part of the environment. The SARA system has rather been intended to implement the static approach to monitoring security and to complement other, dynamic, security systems (or to assure a basic security level where the dynamic systems are not applied). A huge advantage of this approach is that for securing the attached systems, SARA requires neither software installation nor specific configuration routines to be made on them. On the other hand, a certain amount of interaction with their administrator would be necessary: the administrator would have to register his or her systems on the SARA internal webpage and provide the basic data on the used software, as well as the email address. The above would be enough to start receiving notifications about known security vulnerabilities. Finally, the security systems may become outdated themselves – either because of a security vulnerability made by their developers or because their knowledge bases should be extended with the most recently known threats. Both proposed multi-layer security components provide software updating facilities. However, please note that no SARA components would be installed on the attached systems, and MetaIDS plug-ins would probably be also avoided on those systems. However, if ultimately installed, they may be updated by the MetaIDS server (that is controlled, the same as the SARA server, by the KIWI platform administrator). As for updating knowledge bases, SARA in general does not require that facility as it uses NVD [46] – a public repository of software security vulnerabilities. However, there is a possibility of providing the descriptions of vulnerabilities for software not considered by the NVD (which may be important for highly specific
Defence in Depth Strategy: A Use Case Scenario of Securing a Virtual Laboratory
99
and rarely used systems that may occur in a virtual laboratory environment). MetaIDS plug-ins do not contain their internal knowledge bases, they just send the descriptions of the events to the MetaIDS server. With respect to the requirements of the owners of external systems, the described facilities may seem to be oriented more towards usability than security. On the other hand, a virtual laboratory is not a security service. A KIWI platform administrator may have no impact on organizational details in the dynamically attached part of the environment, but the platform offers basic security facilities with minimal (or even zero) changes made to the attached systems.
5 Summary This chapter attempts to aggregate all necessary security considerations that should be taken into account during the process of building a distributed IT infrastructure – on a practical example of a virtual laboratory, based on the authors’ experience. The reasons for which security has to be considered thoroughly and during all stages of the project development have been introduced. Then virtual laboratories have been described, as heterogeneous and distributed environments, together with their general and specific security threats. In order to cope with possible attacks, the problem of securing VLs has been decomposed onto several smaller pieces (layers) and the most significant security problems to consider have been described. This chapter does not aim to be a complete review of all existing security issues, but numerous references have been provided in order to provide more knowledge for those who would like to apply the described recommendations in practice. It has been emphasized that security should not be applied in a single place. The best results are obtained when security measures are present in different functional layers, and when solutions of a different nature are multiplied. Even though a potential attacker might be able to bypass or break one or two of them, the other(s) will still stop (or at least limit) the further damages. The described solutions and practices are and will be used to assure the appropriate security level of the KIWI Remote Instrumentation Platform developed in Pozna´n Supercomputing and Networking Center within the confines of the WLIN (Virtual Laboratory of Interactive Teaching) research project [50]. The described security measures have been applied at an early stage of the lifecycle of the KIWI platform; therefore, no direct comparisons between the unprotected and secured versions of KIWI could be provided. On the other hand, providing built-in security from the very beginning allows to stop the vast majority of network attacks immediately after the protected infrastructure is released. Acknowledgement The authors thank Łukasz Olejnik (PSNC) for valuable advice and support in writing this paper.
100
M. Adamski et al.
References 1. Internet Crime Complaint Center, 2009 Internet Crime Report, pp. 2, http://www.ic3.gov/ media/annualreport/2009 IC3Report.pdf. 2. Eric Chabrow, Cyber Attacks Cost U.S. $1 Trillion a Year, 24 March 2009, http://blogs. govinfosecurity.com/posts.php?postID=159. 3. Yuri Namestnikov, The economics of botnets – Kaspersky Lab, 2009, page 10, http://www. securelist.com/en/downloads/pdf/ynam botnets 0907 en.pdf. 4. EGEE Project – EGEE grid in numbers: Infrastructure Status, October 2009, http://project.euegee.org/index.php?id=417 5. Bruce Schneier, Crypto-Gram Newsletter, May 15, 2000, http://www.schneier.com/cryptogram-0005.html 6. Jan Kwiatkowski, Marcin Pawlik, Gerard Frankowski, Kazimierz Balos, Roman Wyrzykowski, Konrad Karczewski, Dynamic clusters available under Clusterix grid, Lecture Notes in Computer Science, 2007, Volume 4699/2007, pp. 819–829 7. Steven McConnell, Code Complete – A Practical Handbook of Software Construction (Second Edition) – Microsoft Press, 2004, page 548. 8. Naresh Jain, Bala, Agile Overview – Embrace Uncertainty, http://www.slideshare.net/nashjain/ agile-overview. 9. Andrew Odlyzko, Economics, Psychology, and Sociology of Security, Lecture Notes in Computer Science, 2003, Volume 2742, pp. 182–189 10. Michael Howard, David LeBlanc, Writing Secure Code, Microsoft Press 2002, pp. 347 11. FIPS PUB 199, Standards for Security Categorization of Federal Information and Information Systems, National Institute of Standards and Technology, US, February 2004, pp. 2 12. Shawn Herman, Scott Lambert, Tomasz Ostwald, Adam Shostack, Threat Modeling – Uncover Security Design Flaws Using The STRIDE Approach, MSDN Magazine – November 2006, http://msdn.microsoft.com/en-us/magazine/cc163519.aspx 13. National Data Storage project, http://nds.psnc.pl 14. Virtual Laboratory, http://edutechwiki.unige.ch/en/Virtual laboratory 15. National Radio Astronomy Observatory, http://www.nrao.edu 16. Adamiak, R.W., Gdaniec, Z., Lawenda, M., Meyer, N., Popenda, Ł., Stroi´nski, M., Zieli´nski, Laboratorium Wirtualne w srodowisku gridowym (in Polish) http://vlab.psnc.pl/pub/ Laboratorium Wirtualne w srodowisku gridowym ver.1.0.pdf 17. Lawenda, M., Meyer, N., Rajtar, T., Okon, M., Stoklosa, D., Stroinski, M., Popenda, L., Gdaniec, Z., Adamiak, R.W. General Conception of the Virtual Laboratory. International Conference on Computational Science 2004, LNCS 3038, pp. 1013–1016 18. VLab project home page, http://vlab.psnc.pl 19. Lawenda, M., Meyer, N., Rajtar, T., Okon, M., Stoklosa, D., Kaliszan, D., Kupczyk, M., Stroinski, M.: Workflow with Dynamic Measurement Scenarios in the Virtual Laboratory http://vlab.psnc.pl/pub/Workflow With Dynamic Measurement Scenarios In The Virtual Laboratory.pdf 20. Kiwi Platform, http://kiwi.man.poznan.pl 21. Open Web Application Security Project, OWASP Top 10 – 2010, The Ten Most Critical Web Application Security Risks, http://owasptop10.googlecode.com/files/OWASP%20Top%2010 %20-%202010.pdf 22. Nancy Leveson, Clark S. Turner, An Investigation of the Therac-25 Accidents, IEEE Computer, Vol. 26, No. 7, July 1993, pp. 18–41. 23. Therac 25 Case Materials – System Safety, ComputingCases.org service, http:// computingcases.org/case materials/therac/supporting docs/therac case narr/System Safety. html 24. Pravir Chandra, Matt Messier, John Viega, Network Security in OpenSSL, O’Reilly 2002, pp.109 25. OpenSSL: The Open Source toolkit for SSL/TLS, http://www.openssl.org
Defence in Depth Strategy: A Use Case Scenario of Securing a Virtual Laboratory 26. 27. 28. 29.
101
OpenCA PKI Research Labs, http://www.openca.org GAS – Grid Authorization Service, http://www.gridlab.org/WorkPackages/wp-6 Express Production Real-time e-VLBI Service (EXPREeS), http://www.expres-eu.org David F. Ferraiolo, D. Richard Kuhn, Role-Based Access Controls, 15th National Computer Security Conference (1992), Baltimore MD, pp. 554–563, http://csrc.nist.gov/rbac/ferraiolokuhn-92.pdf 30. Pawan Modi, Liferay Portal Authorization & Role Management, http://www.scribd.com/doc/ 16804928/Liferay-Authorization-Role-Management 31. PSNC Security Team, E-commerce security: session and cookies, http://security.psnc.pl/ reports/sklepy internetowe cookies.pdf (in Polish) 32. Ryan Barnett, Fixing Both Missing HTTPOnly and Secure Cookie Flags, December 2008, http://blog.modsecurity.org/2008/12/fixing-both-missing-httponly-and-secure-cookieflags.html 33. Chris Shiftlett, Storing Sessions in a Database, http://shiflett.org/articles/storing-sessions-ina-database 34. Michael Howard, David LeBlanc, Writing Secure Code, Microsoft Press 2002, pp. 434–437 35. Bryan Sullivan, Regular Expressions Denial of Service Attacks and Defenses, MSDN Magazine, May 2010, http://msdn.microsoft.com/en-us/magazine/ff646973.aspx 36. PSNC Security Team, Security best practices for administrators, developers and users of EGEE infrastructure, 2008, https://edms.cern.ch/file/926685/1/EGEE best practices.pdf 37. PSNC in Polish Platform for Homeland Security, http://ppbw.pcss.pl/en 38. Jerzak, M., Wojtysiak, M., Distributed Intrusion Detection Systems – MetalDS case study, Computational Methods in Science in Technology, Special Issue 2010, pp. 135–145. 39. Snort intrusion detection system, http://www.snort.org 40. OSSEC – Open Source Security intrusion detection system, http://www.ossec.net 41. RFC 4765, The Intrusion Detection Message Exchange Format (IDMEF) – http://www.ietf. org/rfc/rfc4765.txt 42. Prelude intrusion detection system, http://prelude.sourceforge.net 43. Common Vulnerabilities and Exposures standard, http://cve.mitre.org 44. Common Platform Enumeration standard, http://cpe.mitre.org 45. Common Vulnerabilities Scoring System standard, http://www.first.org/cvss 46. National Vulnerability Database, http://nvd.nist.gov 47. Nagios monitoring system, http://www.nagios.org 48. Pakiti monitoring software, http://pakiti.sourceforge.net 49. Polish Infrastructure for Information Science Support in the European Research Space – PLGrid, http://www.plgrid.pl/en 50. Virtual Laboratory of Interactive Teaching (WLIN), http://www.man.poznan.pl/online/pl/ projekty/113/WLIN.html (in Polish)
Part II
Software Platforms
Performance Analysis Framework for Parallel Application Support on the Remote Instrumentation Grid Alexey Cheptsov and Bastian Koller
Abstract In the recent years, the Grid has become the most progressive IT trend that has enabled the high-performance computing for a number of scientific domains. The large-scale infrastructures (such as Distributed European Infrastructure for Supercomputing Applications setup in the frame of DEISA or Remote Instrumentation Infrastructure deployed within the DORII EU project) enabled the Grid technology on practice for many application areas of e-Science and have served as a testbed for performing challenging experiments, often involving the results acquired from complex technical and laboratory equipments. However, as the Grid technology has matured, the attention is largely shifted towards optimization of Grid resource utilization by the applications. The performance analysis module setup within the DORII project offers scientific applications an advanced tool set for the optimization of performance characteristics on the Grid. The performance analysis tools adapted and techniques elaborated within DORII for parallel applications, implemented for example by means of Message-Passing Interface (MPI), are presented in this chapter and might be of great interest for the optimization of a wide variety of parallel scientific applications.
1 Motivation and Introduction The widespread use of Internet and Web technology has resulted in leading-edge innovations allowing researches to exploit advanced computational technology in a wide range of application areas of modern science and technology. Grid technology is a fundamental aspect of e-Science that has enabled many scientific societies to get access to high-performance e-Infrastructures in which virtual communities share, federate and exploit the collective power of scientific facilities.
A. Cheptsov () • B. Koller High-Performance Computing Center (HLRS), University of Stuttgart, Germany e-mail:
[email protected];
[email protected] F. Davoli et al. (eds.), Remote Instrumentation for eScience and Related Aspects, DOI 10.1007/978-1-4614-0508-5 7, © Springer Science+Business Media, LLC 2012
105
106
A. Cheptsov and B. Koller
Computation and data Grids are highly beneficial for getting high application scalability characteristics as they offer virtually unlimited computation resources and storage facilities. Moreover, the e-Infrastructures enable the Grid for a great variety of new scientific domains, which pose new challenging requirements to the e-Infrastructure. For example, the e-Infrastructure that is setup by the DORII (Deployment of the Remote Instrumentation Infrastructure)1 project offers a promising way how the modern Grid technology can enhance the level of scientific applications usability, allowing them to have shared access to unique and/or distributed scientific facilities (including data, instruments, computing and communications), regardless of their type and geographical location. The DORII infrastructure consolidates 2,200 CPU cores for the computation and offers a total of 147 TB for storing the data. However, as our experience of porting scientific applications to the Grid e-Infrastructures [1] has revealed, the application performance suffers heavily on a single node due to poor performance characteristics of a standard Grid resource – generic cluster of workstations as compared with the dedicated high-performance computers. This especially concerns the network interconnect between the compute nodes and the file I/O system characteristics, which can considerably degrade the performance of many applications. Message-passing interface (MPI) is a widespread standard for the implementation of parallel applications [2]. MPI applications constitute an important part of the pilot applications that are ported to the e-Infrastructure in the frame of DORII, too. Realization of the message-passing mechanism allows the application to share the computation work over the nodes of the parallel computing system. The efficiency of this sharing is straightforward for gaining high application performance on a single node as well as scalability when running on many nodes. In order to make running the MPI applications on the Grid even more efficient, special performance improvement techniques can be applied for the MPI applications. These techniques rely mainly on in-depth analysis of the implemented communication patterns. The most efficient way of performance characteristics analysis is application’s source code instrumentation. With regard to the MPI applications, instrumentation means mainly collecting the time stamps of the main communication events occurred in the application during run-time. There are many tools facilitating the application instrumentation as well as GUIs intended for analysis of the application run profile. However, support of those tools on the currently available infrastructures is very poor. On the other hand, there is no clear methodic of different tools’ consolidated use for performance analysis of an application. In consequence, performance analysis of the parallel MPI applications is quite awkward for the developers. In the most cases, the complexity of the performance analysis techniques often prevents application providers from in-depth analysis and performance optimization of their applications. In order to facilitate performance analysis of applications ported to a Grid e-Infrastructure, a performance analysis module has been setup within the mid-
1
http://www.dorii.eu.
Performance Analysis Framework for Parallel Application Support on...
107
dleware architecture of the DORII project [3]. In this chapter, we introduce this module and its main components. We begin with a description of a practical use case coming from the DORII project, which is the OPATM-BFM application [4]. We then describe how to use the tools of the performance analysis module for collecting the communication profile in terms of the test application. Finally, we generalize our experience obtained for the tested use case application and present some general performance improvement proposals that might also be of great interest for other parallel applications implemented by means of MPI.
2 Use Case OPATM-BFM is a physical–biogeochemical simulation model developed at Istituto Nazionale di Oceanografia e di Geofisica Sperimentale (OGS) and applied for shortterm forecasts of key biogeochemical variables (among others, chlorophyll and salinity) for a wide range of coastal areas, in particular for the Mediterranean Sea, and currently explored within the DORII project. The model solves the transport-reaction equation (1): @ci @ci @ci @ kz C Rbio .ci ; c1 : : : cN ; T; I : : :/; (1) C v rci D wi C kh rh ci C @t @z @z @z where v is the current velocity, wi is the sinking velocity, kh and kz are the eddy diffusivity constants and Rbio is the biogeochemical reactor that depends, in general, on the other concentrations and on temperature T , short-wave radiation I and other physical variables. The complexity of OPATM-BFM consists in the great number of prognostic variables to be integrated, dimension of analysed ecosystems and steps of the numerical solution according to the forecasted period. In this context, OPATM-BFM poses several challenging scenarios for the efficient usage of modern HPC systems for the application. OPATM-BFM is parallelized by means of MPI, based on domain decomposition over longitudinal elements. MPI implementation enables OPATM-BFM to utilize massive-parallel computing resources. The number of domains corresponds to the number of computing nodes the application is running on. Consistency of the computation by the parallelization and domain decomposition is ensured by the inter-domain communication pattern implemented for exchange of the data cells residing on the domain’s bounds. Obtaining maximal performance (in terms of the execution time) and scalability (in terms of speed-up due to running on the increasing number of computing nodes) is mandatory for OPATM-BFM’s practical usability for the tasks of the real complexity (in particular long-term simulation). Moreover, the application poses great challenge for different HPC architectures with regard to both optimal
108
A. Cheptsov and B. Koller
utilization of resources for performing the identified complex tasks of environmental simulation and development of algorithms enabling such an efficient utilization [5]. The application’s standard run consists of three major phases: initialization (where the model is initialized and input files are read), main simulation loop (iterative solver, the number of iterations depends on the forecasted period’s length, each step corresponds to a half an hour of the simulated system’s behaviour) and data storage (file storage operations, at the end of simulation or after each 48th step of the numerical solution). Analysis of the application time characteristics in the identified phases with regard to computation on a single node and communication between the nodes (notably, MPI functions duration), performed in [1], revealed that scalability on different number of nodes (in the current experiment, execution on 32 and 64 nodes has been analysed) is quite poor for some operations (see Table 1). However, it is highly important to highlight that time distribution among the phases of execution in the tested use case (second column in Table 1) can hugely differ as compared with a real long-term simulation use case (third column in Table 1) due to different impact of iteratively repeated operations on the total time (the last row in Table 1). Application performance speed-up when running on bigger number of nodes is also changing according to the use case. Nevertheless, the internal characteristics of the iterative phase are iteration-independent and valid not only for the test one but for the all use cases. There are several software tools that facilitate instrumentation of the source code and collection of the details about the occurred events as well as further analysis of those events. Some of the tools (e.g. Valgrind tool suite [6]) are useful for static analysis of the single application’s process on a single computing node, while others aim at parallel applications [7] and focus on analysis of the MPI communication between the nodes of the parallel computer (e.g. Vampir [8] and Paraver [9]). The DORII project’s applications, including presented OPATM-BFM, can greatly benefit from the use of both mentioned categories of tools. Whereas profiling a single MPI process will allow the user to identify the source code regions where the most time-consuming communication takes place, the detailed information about the interactions (messages for the MPI applications) between the different processes of a parallel program is provided by the communication analysis tools (Fig. 1).
3 Performance Analysis Tools and Techniques The practical attempts to design a scalable and easy-to-use performance measurement and monitoring software environment for supercomputing applications running on the remote instrumentation infrastructure resulted in a special module provided within the middleware architecture of the DORII project. In the following, we give a brief description of the main tools comprised in the module – Valgrind and Vampir/VampirTrace – and highlight the main usage scenarios of those for the exemplar use case from the previous section – the OPATM-BFM application.
Phases of execution 1. Initialization, input 2. Main simulation loop 3. Data storage Total time, (s)
# Iterations, test case 1
3
1
# Iterations, real case 1
816
17
7 949
3
Comput. duration 939
204 226
5
MPI calls duration 6
32 nodes, with one process per node, (s)
Table 1 Application performance on changing the number of computing nodes
213 1,175
8
Total time 945
3 2,243
2
Comput. duration 2,238
170 181
5
MPI calls duration 6
64 nodes, with one process per node, (s)
173 2,424
7
Total time 2,244
0.4 2.4
0.7
1.25 0.8
1
Comput MPI calls 2.4 1
Scalability coefficient, t64 =t32 ; .s/
0.8 2
0.9
Total 2.4
Performance Analysis Framework for Parallel Application Support on... 109
110
A. Cheptsov and B. Koller
Fig. 1 Using performance analysis tools for OPATM-BFM
Valgrind [6] is an instrumentation framework for building dynamic analysis tools for debugging and profiling sequential applications (or single processes of a parallel application), aiming at speeding-up application performance. Valgrind’s distribution is an open source and currently includes several production-quality tools, including a memory error and thread error detectors, a cache and branch-prediction profiler, a call-graph generating cache profiler and others. In order to proceed with the parallel application analysis efficiently, the phases of application execution should firstly be identified. The localization of the most computation- and communication-intensive phases without the help of any profiling tools is a non-trivial and quite complicated task that requires deep understanding of the application source code as well as implemented algorithms. However, Valgrind is capable of building a so-called application call-graph, which is sufficient for basic understanding of the dependencies between the application’s code regions as well as time characteristics of the communication events in those regions. This is done by means of the Callgrind tool, which is included to the current Valgrind distribution. Moreover, details on the I1, D1 and L2 CPU caches are provided by Callgrind as well. In addition, Valgrind provides a powerful visualizer for the data produced by Callgrind, which is KCachegrind. For example, in Fig. 2 a fragment of the OPATMBFM’s call-graph is shown, visualized through KCachegrind, based on which the main phases of application run (Table 1) have been identified.
Performance Analysis Framework for Parallel Application Support on...
111
Fig. 2 The OPATM-BFM’s call-graph’s fragment (visualized with KCachegrind)
Analysis of the single process’ run profile is also an excellent starting point for profiling communication between the MPI processes in the parallel application. There are several tools designed for large-scale application analysis. In the frame of the remote instrumentation infrastructure, we chose the VampirTrace tool [10] because of its tight integration to the Open MPI library, used for the implementation of parallel applications by the DORII consortium. However, the introduced performance analysis module does not eliminate the use of other famous tools, including Paraver [9], or Scalasca [11], for post-mortem analysis or Periscope, which is used for analysis at run-time. VampirTrace is an application tracing package that collects a very fine grained event trace of a sequential or parallel program. The traces can be visualized with the Vampir or any other tool that reads the open trace format (Fig. 3). Another advantage of VampirTrace is that no changes in the application’s code are required for switching on profile collection functionality, only recompilation with corresponding VampirTrace libraries (or using the wrapper provided with open MPI).
112
A. Cheptsov and B. Koller
Fig. 3 Example of the visualized application’s communication profile (performed with Vampir tool). Communication between the nodes is shown with arrows. The operations are presented with different colours
To sum up, analysis of the MPI application’s communication patterns as well as time distribution among the main phases of a single process is largely based on the profile collected at application’s run-time. The profile is stored in the trace files of the special format and can be visualized by means of the dedicated graphical front-ends after execution is completed. When using performance analysis tools, it is highly important to note that the size of the obtained trace files is proportionate to the number of the events occurred. For the large-scale parallel applications, which are often implemented with a significant number of function calls, in particular to the MPI library, the large size of the collected trace files poses a serious drawback for their further exploration with the visualization front-ends. In such cases, a preliminary analysis step is required, which aims at detecting the application execution’s regions with the communications having the maximal impact on execution time. In general, the low-priority communication events should be discarded from profiling that in turn leads to decreasing the size of the trace files with the collected events. This can be done by either filtering the events in the defined regions or launching the application for special use cases with a limited number of iterations. However, this task is not trivial and approaches that allow the user to filter the events in the trace files differ from application to application. Nevertheless, it can greatly be supported by the tools of the first category, in particular, based on the call-graph analysis. For example, even for a standard (short-term) use case, the OPATM-BFM application presented above performs more than 250,000 calls to the MPI library. In a consequence, the size of the trace data grew up to several tens
Performance Analysis Framework for Parallel Application Support on...
113
of gigabytes. Due to preliminary analysis of the call-graph, the trace file size was minimized down to only 200 Mb, which allowed us to proceed with post-processing analysis even on the low-performance machine. Therefore, the combination of those tools presented above on the joint basis, provided by the performance analysis module, offers the user a complete environment for efficient and effective application performance analysis.
4 Some Optimization Results There are several techniques that could be applied for MPI applications aiming at their performance improvement. As an outcome of exemplarily use of the performance analysis module for OPATM-BFM, the following main improvement proposals have been specified: • Use of MPI-IO operations for parallel access/storage of NetCDF data (currently the application I/O has been ported to the PNetCDF library) • Use of the collective MPI communication for inter-domain communication • Encapsulation of the data chunks transmitted inside a single MPI communication The realization of the optimization proposals for the application communication and I/O patterns elaborated in [1] for OPATM-BFM allowed us to improve dramatically the performance for the test case (three steps of numerical solution), as shown in Table 2. Whereas file I/O operations are dominating for the application execution for a small number of simulation steps (three steps for the test case), the overall performance improvement due to optimization of the MPI communication becomes significant only for a long-term simulation (816 steps for the real case). As Table 2 shows, the total amount of realized optimizations allowed us to reduce the duration of the application execution for the real case by 64% (from initially measured 1,175 down to 5,145 s). Furthermore, the optimization for the increasing number of nodes
Table 2 Comparison of application time characteristics before and after optimization 32 nodes 64 nodes Phases of execution 1. Initialization, input 2. Main simulation loop 3. Data storage Total
Total duration (s)
Total duration (s)
Initial
Initial
Optimized
Optimized
945
300
2,244
360
8
7
7
6
213 1,175
2,075 5,145
173 2,424
172 538
114
A. Cheptsov and B. Koller
from 32 up to 64 grew up to 88% (by reduction of the execution time from 2,424 down to only 538 s). The application speed-up when scaling the number of compute nodes from 32 to 64 was improved as well.
5 Conclusions and Future Directions The main aim of this chapter was to show how different performance analysis tools and strategies can be consolidated and applied for supercomputing applications. The DORII project enables Grid technology for many new applications, whose performance properties play an important role in practical usability within the modern Grid environments. Performance analysis is, therefore, straightforward for improved run of the applications, in particular, for use in production. Integration of all the described tools in the common performance analysis module introduced in this chapter allowed us to holistically investigate the performance of one of the most challenging applications deployed on the Remote Instrumentation Infrastructure – OPATM-BFM. The performance improvement techniques applied for the application allowed us to optimize the application characteristics when running on a standard EGEE Grid site. The performance analysis module will be used to tune the performance of academic and industrial simulation applications, not limited on the ones coming from the DORII consortium.
References 1. Alexey Cheptsov, Kiril Dichev, Rainer Keller, Paolo Lazzari and Stefano Salon: Porting the OPATM-BFM Application to a Grid e-Infrastructure – Optimization of Communication and I/O Patterns, Computational Methods in Science and Technology, 15(1), 9–19 2009. 2. The MPI standard http://www.mcs.anl.gov/research/projects/mpi/standard.html 3. Adami, D., Cheptsov, A., Davoli, F., Liabotis, I., Pugliese, R., and Zafeiropoulos, A.: The DORII Project Test Bed: Distributed eScience Applications at Work, in: Proceedings of the 5th International Conference on Testbeds and Research Infrastructures for the Development of Networks & Communities and Workshops (TridentCom 2009), Washington DC, USA, 6–8 April 2009, 1–4, doi:10.1109/TRIDENTCOM.2009.4976247, 2009. 4. A. Crise, P. Lazzari, S. Salon, and A. Teruzzi. MERSEA deliverable D11.2.1.3 - Final report on the BFM OGS-OPA Transport module, 21 pp., 2008. 5. Alexey Cheptsov: Enabling grid-driven supercomputing for oceanographic applications – theory and deployment of hybrid OpenMP C MPI parallel model for the OPATM-BFM application. Proceeding of HPC-Europa project’s Transnational Access Meeting, Montpellier, October 14th–16th 2009. 6. Nicholas Nethercote and Julian Seward. algrind: A Framework for Heavyweight Dynamic Binary Instrumentation. Proceedings of ACM SIGPLAN 2007 Conference on Programming Language Design and Implementation (PLDI 2007), San Diego, California, USA, June 2007. 7. F. Wolf: Performance Tools for Petascale Sysems. inSiDE, Vol. 7, No. 2, pp. 38–39 2009. 8. A. Kn¨upfer, H. Brunst, J. Doleschal, M. Jurenz, M. Lieber, H. Mickler, M.S. M¨uller, and W.E. Nagel. The Vampir Performance Analysis Tool-Set. Tools for High Performance Computing, Springer, 2008, 139–156.
Performance Analysis Framework for Parallel Application Support on...
115
9. Vincent Pillet, Jes´us Labarta, Toni Cortes, Sergi Girona. PARAVER: A Tool to Visualize and Analyze Parallel Code. In WoTUG-18, 1995. 10. A.G. Sunderland, R.J. Allan. An Example of Parallel Performance Analysis using VAMPIR. Retrieved from http://www.cse.scitech.ac.uk/arc/vampir.shtml 11. Markus Geimer, Felix Wolf, Brian J. N. Wylie, Daniel Becker, David B¨ohme, Wolfgang Frings, Marc-Andr´e Hermanns, Bernd Mohr, Zolt´an Szebenyi: Recent Developments in the Scalasca Toolset. In Tools for High Performance Computing 2009 of 3rd International Workshop on Parallel Tools for High Performance Computing, pages 39–51, Dresden, ZIH, Springer, 2010.
New Technologies in Environmental Science: Phenology Observations with the Kiwi Remote Instrumentation Platform Dominik Stokłosa, Damian Kaliszan, Tomasz Rajtar, Norbert Meyer, ´ Filip Koczorowski, Marcin Procyk, Cezary Mazurek, and Maciej Stroinski
Abstract Advances in e-learning create a base for creative use of new technologies in virtually any field of science. Whether or not this results in improvements in the quality of education is mostly up to the attractiveness of the final solution. This chapter describes a possible way of increasing the attractiveness of learning tutorials by the Kiwi remote instrumentation platform with the main concern on environmental science domains like phenology, meteorology, water circulation and its chemistry or, finally, air pollution. This chapter is focussed on the technical aspects of conducting phenology observations with a remotely controlled DSLR camera managed by the Kiwi platform.
1 Introduction The continuous development of human civilization, explorations of natural resources or pollution of the environment have an enormous impact on the balance of the biological life of fauna and flora. However, over the past decades, people have tried to minimise the harmful impact on the environment and to repair all errors made while exploring it. A variety of means, like advanced and eco-friendly production technologies, establishment of new forms of nature conservation and wide-ranging environmental education among children are used nowadays to protect the ecosystem and spread out the knowledge. The main objective of the project is to develop a virtual laboratory of interactive teaching, which will be controlled by the Kiwi remote instrumentation platform.
D. Stokłosa () • D. Kaliszan • T. Rajtar • N. Meyer • F. Koczorowski • M. Procyk • C. Mazurek • M. Stroi´nski Pozna´n Supercomputing and Networking Center Pozna´n, Poland e-mail:
[email protected];
[email protected];
[email protected];
[email protected];
[email protected];
[email protected];
[email protected];
[email protected] F. Davoli et al. (eds.), Remote Instrumentation for eScience and Related Aspects, DOI 10.1007/978-1-4614-0508-5 8, © Springer Science+Business Media, LLC 2012
117
118
D. Stokłosa et al.
Kiwi has been designed to provide the environment where different scientific, video or measurement pieces of equipment can be easily attached to the system and also to build an interactive e-learning platform for students. We are planning to build a pilot installation of the system with a set of different measuring devices installed in a forest in two selected locations near Pozna´n. All devices will be integrated and remotely controlled by the Kiwi platform. The data from devices and sensors will be stored automatically in the system and will be available within the e-learning platform. Moreover, data from the observation equipment will be incorporated into e-learning lessons, which will be available for students and citizens.
2 Scope The project addresses five selected areas: air pollution, water management, meteorology, dead wood and, finally, phenology (see Fig. 1). Phenology is described in detail in Sect. 4. However, the automated, remotely controlled observation system is the main focus of this chapter. The description of the e-learning platform is not the scope of this chapter.
Fig. 1 List of addressed areas
New Technologies in Environmental Science: Phenology Observations...
119
Fig. 2 Old mill pond
(a) Air Pollution Two weather stations will be installed near Pozna´n and integrated with the Kiwi platform. Educational content about air pollution will be enriched with live data coming from air quality sensors. This will allow us to visualise the influence of city-generated air pollution on the nearby region. (b) Water Management Since the water circulation is strictly connected with weather conditions, online content will be enriched with data from weather stations, as well as from a water flow measurement device installed at the estuary of a river coming through a pond located near an old water mill close to Pozna´n. This solution will help us to observe the relationship between changing amounts of atmospheric precipitation, isolation, temperature and the water flows. Moreover, it is planned to build an educational path next to the pond (see Figs. 2 and 3) explaining the role of the water in the environment. Students and pupils will have a chance to see devices and sensors integrated with the Kiwi platform in action on-site, and also to expand their knowledge on water management area. (c) Meteorology Weather stations will be installed in two locations near Pozna´n. A weather station is a compact system for hydro-meteorological monitoring. A station is a flexible and modular system which can be extended with different sensors. The sensors allow us to measure wind speed and direction, air pressure, air and ground temperature or humidity.
120
D. Stokłosa et al.
Fig. 3 Pond estuary
(d) Role of Deadwood By popular knowledge, deadwood is ranked close to waste. That, however, cannot be further from the truth. Dozens of species live only in deadwood microenvironments. Deadwood is even considered to be the richest habitat in a healthy forest [5]. Learning materials concerning deadwood will be accompanied by a live video stream from a site inhabited by Osmoderma eremita, a European beetle in the family Scarabaeidae. The video stream will be supplied by a remotely controlled panoramic video camera equipped with a large zoom range lens, that will allow showing both the site as a whole and single hollows.
3 Kiwi Platform As it was presented before, the main objective of the project is to develop an e-learning interactive platform for areas like water management, air pollution, deadwood or phenology. In order to make the study process more interesting, the platform will be equipped with dynamic content like data collected by measurement devices or high quality images from the observation scenes. We believe that a successful e-learning platform should provide means to add new devices and sensors to the existing infrastructure. The platform should also allow us to manage and monitor devices and control data acquisition processes. In order to achieve these goals, a special system is required which will be responsible
New Technologies in Environmental Science: Phenology Observations...
121
Fig. 4 Kiwi platform
for all the work behind the scenes. We propose to split the system into two parts, with an access layer in between two layers (see Fig. 4). • Layer I: e-learning platform – user interface and access point to the learning tutorials • Layer II: Kiwi remote instrumentation platform Layer I is responsible for providing the interface for online lessons and tutorials. The platform is based on a custom learning management system built by extending open source content management system software. The e-learning platform layer is only focussed on providing lessons and tutorials. It does not deal with issues related to accessing and controlling devices or sensors, nor has the knowledge how to do it. All the instruments are managed and controlled by the Kiwi remote instrumentation platform (see Fig. 4). The Kiwi platform is a framework for building remote instrumentation systems. The platform provides a set of components to control and manage scientific equipment or sensors like cameras, weather, air pollution and water flow sensors and others. All the equipment connected to the Kiwi platform can be controlled remotely with one unique user interface. It also allows users to design and run the so-called observation workflows. It is possible to set up a sequence of operations starting from data acquisition, data processing and finally visualisation. This workflow can be launched periodically by the Kiwi workflow manager component with a desired frequency and time. For instance, the system administrator can plan a
122
D. Stokłosa et al.
workflow with a camera taking pictures twice a day at noon time. The device can be set up so that several points of a scene are being shot. The Kiwi remote instrumentation platform is a successor of a distributed workgroup environment, the virtual laboratory system [8], which has been developed by Pozna´n Supercomputing and Networking Center. Devices and sensors produce different output data. Most of the sensors produce numerical data which will be visualised by the e-learning platform. However, the IP cameras or photo-cameras produce images and videos. In order to make the system flexible, we have designed one common and generic interface for retrieving data from the instruments – Kiwi access layer. This is the only place where the two layers interact with each other. The same interface is used when dealing with different equipment, i.e. when the temperature from the given location is required, as well as images from the photo camera are downloaded.
4 Phenology Observations Phenology is the study of the timing of life cycle events of plants and animals [1]. Common examples include the date migrating birds return, the first flower dates for plants, and the date on which a lake freezes in the autumn or opens in the spring. The presence of advanced weather forecasting systems pushed aside this area of research. However, phenological records can be a useful proxy for a temperature analysis in historical climatology [2], especially in the study of climate change and global warming. Phenological observations are one of the simplest methods of observing climate changes that do not require any special skills nor equipment. The only rule is to select a site, then select a plant or an animal and record whether or not the so-called phenophases are occurring. The hard requirement is to record observations regularly and as often as every two or three days. Additionally, the observation sites are ideally far from urban areas. This means even simple phenology observations require quite a bit of time. It also means that the observations may benefit from new technologies such as remotely controlled cameras and video streaming. The Chinese are thought to have kept the first written records dating back to around 974 BC. For the past 1,200 years, observations of the timing of peak cherry blossoms in Japan have been recorded. Phenological observations have a long tradition in Poland and are one of the oldest in Europe. First networks of stations and outposts were found in Poland in the middle of the nineteenth century [9]. Phenological observations are nowadays still carried out. Action 725 of the European Cooperation in the field of Scientific and Technical Research [6] and the USA National Phenology Network [7] are the biggest scale observations.
New Technologies in Environmental Science: Phenology Observations...
123
5 Automated Phenology Observations Fauna and flora are reacting rapidly to climate changes. Therefore, plants and animals observations can be valuable in nature science education. However, they are very time consuming and require a special attitude during the observation phase. It seems that regular, stationary phenology observations are very hard to conduct with a group of students. We have introduced an automated, remotely controlled system for phenology observations based on the Kiwi remote instrumentation platform. High quality photo-cameras are used for a scene comprehensive view. Cameras can be set up to take high-resolution pictures with a desired frequency and time. The photo equipment has been integrated with the Kiwi platform. The platform allows controlling the cameras remotely and also provides tools to manage the observation process, i.e. changing the time and frequency of pictures being shot. The system allows controlling and monitoring all the observation equipment remotely. All photos taken during nature surveillance will be available for students and pupils for a study process or further analysis in the e-learning portal.
6 Surveillance Equipment The most important factors of successful digital phenology observations are: high quality images and a high level of details. The observation scenario is as follows: an operator defines a set of scenes. The operator has to set up cameras to shoot certain points of interest on each scene. Next, the operator has to set up the time and frequency of pictures being shot. After that, the entire process is automated. The photo-cameras will get triggered by the Kiwi platform at the specified time. The resulting photos will be stored in the Kiwi platform and will be available for students through the e-learning portal. We have to make sure that the resulting pictures are of high quality allowing users to zoom in or out and observe tiny details of their target of interest like a leaf or a bud. This is why we have conducted a series of tests allowing us to choose the best equipment. A maple tree has been chosen as our test object. We have shot three types of pictures: wide angle, medium and finally maximum zoom pictures using such devices as video cameras, video cameras with high quality lens and photocameras. The comparison is shown in Figs. 5–7. The comparison presents pictures taken with a compact camera, digital reflex camera and finally with a high-definition video camera. We have divided the pictures into three categories: wide angle (if a user wants to see the entire scene and its surroundings), medium zoom (please note that for some devices this was the maximum zoom available) and maximum zoom.
124
D. Stokłosa et al.
Fig. 5 Image quality comparison – wide angle
Fig. 6 Image quality comparison – medium zoom
The user is mostly interested in the maximum zoom quality comparison which is presented in Fig. 7. For instance, for a daily increase of a leaf size measurement, a strong close-up is required. Our tests have shown that even expensive, high-quality video cameras did not turn out to be useful in the phenology observations. As one can see in the comparison, the resulting photos are not detailed enough. Moreover, the Sony
New Technologies in Environmental Science: Phenology Observations...
125
Fig. 7 Image quality comparison – max zoom
Fig. 8 Image quality comparison – compact cameras
PMW-Ex3 Full-HD Camcorder, which has been used in out tests, reaches its maximum zoom very quickly. The maximum zoom available is presented in the medium comparison. Photo-cameras, as our tests show, are much better suited for this kind of usage. Tests in the field have shown that even consumer-level compact cameras deliver satisfactory image quality. Figure 8 shows the quality comparison between two different compact cameras: Panasonic DMC TX3 and Nikon Coolpix L12. However, photo-cameras have one significant disadvantage compared to video cameras: lack of remote control functionality and software required to control the camera remotely using a web browser. This feature is provided even by low-end video cameras but is usually not available in digital SLR or compact cameras. We have proposed a solution where a digital reflex camera is integrated with a turnplate device and can be controlled remotely using the Kiwi platform.
126
D. Stokłosa et al.
7 Remotely Controlled Reflex Camera As it was mentioned before, a digital reflex camera allows us to obtain high quality and high resolution pictures. However, the remote control functionality is missing. In order to base the phenology observations on reflex cameras, we had to implement a set of missing functionalities like camera movements, setting picture properties, shooting pictures and, finally, storing a camera position. In order to achieve this, we had to integrate the camera with other devices like turnplate, heater, control computer and other controllers. This is presented in Fig. 9. We have designed a special casing to hold a camera and other necessary equipment. The casing is resistant to water and other outdoor conditions. The casing is also equipped with a temperature sensor and a heater, which is triggered by the sensor. Since there is no option to connect a digital camera directly to the network, a small control computer is required. The camera is connected to the computer and managed by the Kiwi components, which are installed there as well. The Kiwi components are responsible for controlling the camera, taking pictures and downloading them from a device. The camera case is mounted on top of a turnplate device – see Fig. 9. This means that we will not be moving the camera itself, but the entire casing with all the equipment installed in it. The turnplate is connected to the control computer and managed by the Kiwi component. We have also prepared a graphical user interface for end users – Kiwi Instrument, which allows users to take advantage of the implemented functionality (Fig. 10). The interface integrates camera control, turnplate control and, finally, camera live preview functionality at one place.
Fig. 9 Remotely controlled camera components
New Technologies in Environmental Science: Phenology Observations...
127
Fig. 10 Kiwi instrument – camera control interface
8 Summary At this point the project is under development. The first release was completed at the beginning of 2011. All devices from areas presented in this article like air pollution, water management, meteorology or deadwood will get integrated with the Kiwi platform. Moreover, two remotely controlled photo-cameras will be deployed in the Zielonka Forest near Pozna´n. The cameras will be set up to observe the same type of tree to give students the opportunity to compare the results and broaden their knowledge about phenology.
References 1. About Phenology. Available on-line at http://www.usanpn.org/about/phenology 2. M. Molga. Meteorologia rolnicza. PWRiL. 1983. 3. K. Piotrowicz. Historia obserwacji fenologicznych w Galicji. IMiGW. 2007. 4. Water cycle. Available on-line at http://en.wikipedia.org/wiki/Water circulation 5. Deadwood – living forests. The importance of veteran trees and deadwood to biodiversity. WWF Report. 2004. Available on-line at http://wwf.panda.org/about our earth/all publications/? 15899/Deadwood-living-forests-The-importance-of-veteran-trees-and-deadwood-tobiodiversity 6. COST 725: Establishing a European Phenological Data Platform for Clima tological Applications. Available on-line at www.fsd.nl/cost725 7. USA National Phenology Network. Available on-line at http://www.usanpn.org/home 8. Virtual Laboratory PSNC. Available on-line at http://vlab.psnc.pl 9. Phenology History, http://www.budburst.ucar.edu/phenology history.php
An Agent Service Grid for Supporting Open and Distance Learning Alberto Grosso, Davide Anghinolfi, Antonio Boccalatte, and Christian Vecchiola
Abstract This chapter describes ExpertGrid, a software infrastructure for the development of decision support tools to train crisis managers. The system handles a rich variety of realistic critical scenarios addressed by the trainees, who, by means of the system, will develop effective countermeasures. ExpertGrid is based on agent oriented technology to be dynamic, flexible, and open, and it relies on a distributed computing grid to support large scale scenario computations and incorporate a larger body of knowledge. This contribution presents the design of the system together with its prototypal implementation and initial results.
1 Introduction Decision support systems [1] constitute a valuable tool for helping humans in taking decisions during critical conditions. Critical situations happen unexpectedly and require immediate response where the outcome of a wrong decision can be extremely costly. In these scenarios, the ability to react promptly is fundamental; the support given by an automated tool filtering out the inapplicable or not opportune courses of action can help reducing the decision time, thus making the reaction more effective. Moreover, response to crisis scenarios often requires a heterogeneous expertise to fully address the critical situation. It generally includes knowledge of regulatory bodies, psychology, medicine, structural engineering, chemical and electrical hazards, and specific knowledge about the crisis site (e.g., urban, maritime, or airborne). Such expertise is both human, matured over years of experience, and technical. In these conditions, a crisis executive committee will rely on a large
A. Grosso () • D. Anghinolfi • A. Boccalatte • C. Vecchiola Department of Communication, Computer and System Sciences, University of Genova, Via Opera Pia 13, 16145 Genova, Italy e-mail:
[email protected];
[email protected];
[email protected];
[email protected] F. Davoli et al. (eds.), Remote Instrumentation for eScience and Related Aspects, DOI 10.1007/978-1-4614-0508-5 9, © Springer Science+Business Media, LLC 2012
129
130
A. Grosso et al.
number of experts to cover the full range of skills required for an effective and appropriate decision. This might not be a practical solution given the stringent time constraints, which do not leave room for large scale debates on the best course of action. A training system able to prepare crisis managers to actively respond to emergency situations could help speeding up the decision process, thus making the reaction more effective and prompt. In this paper, we propose a distributed e-learning tool, called ExpertGrid, for training crisis managers to act readily in critical conditions. The tool integrates statistical analysis, crisis modelling, and scenarios management. More precisely, it leverages the body of knowledge of real experts and provides a simulation environment where crisis scenarios designed and validated by experts are used to train crisis managers. ExpertGrid uses a “learn by mistake” approach: the solutions devised by the trainees are compared to the ones given by the experts and crisis managers can refine their skills incrementally. The system is intended to help municipalities, regional authorities, and state governments to face unexpected emergencies. ExpertGrid enables local response, recovery, and optimized damage control. Moreover, by relying on a distributed and widely accessible infrastructure it can be used as a general repository of knowledge: pre-existing and already tested solutions in similar scenarios can be adapted to the current crisis condition. Moreover, mutual feedback from local crisis handling creates a robust, versatile, and fast reacting crisis response network managed by highly trained crisis managers. Fundamental for the effectiveness of the system is the creation of a community of experts and crisis managers around it. In order to make this happen, ExpertGrid has been designed to be widely accessible and distributed. This allows the system to incorporate a larger body of knowledge as a result of the contribution of all the community of experts around the world. Essential for the tool is the ability to scale as the number of users increases and to support large scenario simulations. To address this issue, ExpertGrid relies on Grid computing, which provides the ability to scale and to leverage multiple processing nodes for large scenario simulations, and agent technology, which confer the flexibility, openness, and dynamicity required to continually keeping updated the system and integrate new policies and attributes to evaluate. The rest of the chapter is organized as follows: Sect. 2 provides a brief description of the background technologies used to develop the system, such as grid computing and software agents; Sect. 3 discusses the motivations, functionalities, and design solutions of ExpertGrid. Conclusions along with the description of ongoing activities follow.
2 Background Technologies 2.1 Grid Computing Grid Computing [2] provides access to large computational power, huge storage facilities, and a variety of additional services by harnessing geographically disparate computing sites connected through high capable network connect