Algorithms and Architectures for Parallel Processing. ICA3PP 2011 Workshops Melbourne Proceedings Part II (Lecture Notes in Computer Science)

Lecture Notes in Computer Science Commenced Publication in 1973 Founding and Former Series Editors: Gerhard Goos, Juris Hartmanis, and Jan van Leeuwen

Editorial Board David Hutchison Lancaster University, UK Takeo Kanade Carnegie Mellon University, Pittsburgh, PA, USA Josef Kittler University of Surrey, Guildford, UK Jon M. Kleinberg Cornell University, Ithaca, NY, USA Alfred Kobsa University of California, Irvine, CA, USA Friedemann Mattern ETH Zurich, Switzerland John C. Mitchell Stanford University, CA, USA Moni Naor Weizmann Institute of Science, Rehovot, Israel Oscar Nierstrasz University of Bern, Switzerland C. Pandu Rangan Indian Institute of Technology, Madras, India Bernhard Steffen TU Dortmund University, Germany Madhu Sudan Microsoft Research, Cambridge, MA, USA Demetri Terzopoulos University of California, Los Angeles, CA, USA Doug Tygar University of California, Berkeley, CA, USA Gerhard Weikum Max Planck Institute for Informatics, Saarbruecken, Germany

7017

Yang Xiang Alfredo Cuzzocrea Michael Hobbs Wanlei Zhou (Eds.)

Algorithms and Architectures for Parallel Processing 11th International Conference, ICA3PP 2011 Melbourne, Australia, October 24-26, 2011 Proceedings, Part II

13

Volume Editors Yang Xiang Wanlei Zhou Deakin University, School of Information Technology Melbourne Burwood Campus, 221 Burwood Highway Burwood, VIC 3125, Australia E-mail: {yang, wanlei}@deakin.edu.au Alfredo Cuzzocrea ICAR-CNR and University of Calabria Via P. Bucci 41 C, 87036 Rende (CS), Italy E-mail: [email protected] Michael Hobbs Deakin University, School of Information Technology Geelong Waurn Ponds Campus, Pigdons Road Geelong, VIC 3217, Australia E-mail: [email protected]

ISSN 0302-9743 e-ISSN 1611-3349 e-ISBN 978-3-642-24669-2 ISBN 978-3-642-24668-5 DOI 10.1007/978-3-642-24669-2 Springer Heidelberg Dordrecht London New York Library of Congress Control Number: 2011937820 CR Subject Classification (1998): F.2, H.4, D.2, I.2, G.2, H.3 LNCS Sublibrary: SL 1 – Theoretical Computer Science and General Issues

© Springer-Verlag Berlin Heidelberg 2011 This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, re-use of illustrations, recitation, broadcasting, reproduction on microfilms or in any other way, and storage in data banks. Duplication of this publication or parts thereof is permitted only under the provisions of the German Copyright Law of September 9, 1965, in its current version, and permission for use must always be obtained from Springer. Violations are liable to prosecution under the German Copyright Law. The use of general descriptive names, registered names, trademarks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. Typesetting: Camera-ready by author, data conversion by Scientific Publishing Services, Chennai, India Printed on acid-free paper Springer is part of Springer Science+Business Media (www.springer.com)

Message from the ADCN 2011 Chairs

We are happy to welcome you to the 2011 International Symposium on Advances of Distributed Computing and Networking (ADCN 2011). ADCN 2011 is held in conjunction with the 11th International Conference on Algorithms and Architectures for Parallel Processing (ICA3PP 2011), Melbourne, Australia, October 24-26, 2011. ADCN 2011 contains 16 full papers selected from those submitted to the ICA3PP 2011 main track. All the papers were peer reviewed by members of the ICA3PP 2011 Program Committee. The symposium covers a broad range of topics in the field of parallel and distributed computing such as cluster; distributed and parallel operating systems and middleware; cloud, grid, and services computing; reliability and fault-tolerant computing; multi-core programming and software tools; distributed scheduling and load balancing; high-performance scientific computing; parallel algorithms; parallel architectures; parallel and distributed databases; parallel I/O systems and storage systems; parallel programming paradigms; performance of parallel and distributed computing systems resource management and scheduling; tools and environments for parallel and distributed software development; software and hardware; reliability testing, verification and validation; security, privacy, and trusted computing; self-healing, self-protecting and fault-tolerant systems, information security on internet, multimedia in parallel computing parallel computing in bioinformatics dependability issues in computer networks and communications; dependability issues in distributed and parallel systems; dependability issues in embedded parallel systems; industrial applications; and scientific applications. We thank the authors for submitting their work and the members of the ICA3PP 2011 Program Committee for managing the reviews of the ACDN 2011 symposium papers in such short time. We firmly believe this symposium complements perfectly the topics covered by ICA3PP 2011, and provides additional breadth and depth to the main conference. Finally, we hope you enjoy the symposium and have a fruitful meeting in Melbourne, Australia. August 2011

Wanlei Zhou Alfredo Cuzzocrea Michael Hobbs

Message from the IDCS 2011 Chairs

It is our great pleasure that the accepted papers of the 4th International Workshop on Internet and Distributed Computing Systems (IDCS 2011) included in the proceedings of the 11th International Conference on Algorithms and Architectures for Parallel Processing (ICA3PP 2011), held in Melbourne, Australia during October 24–26, 2011. Following the previous three successful IDCS workshops – IDCS 2008 in Dhaka, Bangladesh; IDCS 2009 on Jeju Island, Korea; and IDCS 2010 in Melbourne, Australia – IDCS 2011 is the fourth in its series to promote research in diverse fields related to Internet and Distributed Computing Systems. In this workshop, we are interested in presenting innovative papers on emerging technologies related to Internet and distributed systems to support the effective design and efficient implementation of high-performance computer networks. The areas of interest for this year’s event are the following: – – – – – – – – –

Internet architectures and protocols modeling and evaluation of internet-based systems Internet quality of service grid, cloud, and P2P computing middleware for wireless sensor networks security of network-based systems network-based applications (VoIP, streaming) network management and traffic engineering tools and techniques for network measurements

The target audience of this event includes researchers and industry practitioners interested in different aspects of the Internet and distributed systems, with a particular focus on practical experiences with the design and implementation of related technologies as well as their theoretical perspectives. We received 23 submissions from 7 different countries. Each submission was reviewed by three members of the international Program Committee. After a rigorous review process, we selected 10 papers for inclusion in the workshop program. We plan to invite extended and enhanced versions of top-quality selected papers for submission on a fast-track basis for the Springer Journal of Internet Services and Applications (JISA) and International Journal of Internet and Distributed Computing Systems (IJIDCS). In addition, selected papers in the information security area will be recommended for publication in the International Journal of Risk and Contingency Management. The organization of IDCS 2011 includes direct or indirect contributions from many individuals, including program chairs, Program Committee members, external reviewers, logistics personnel and student volunteers. We would like to thank Dr Wen Tao Zhu and Dr Muhammad Khurram Khan for accepting the

VIII

Message from the IDCS 2011 Chairs

IDCS 2011 workshop proposal within ICA3PP. Special thanks to ICA3PP general chairs Andrzej Goscinski and Peter Brezany, as well as program chairs Yang Xiang, Alfredo Cuzzocrea, and Michael Hobbs for their continuous support in making IDCS 2011 a success. Last but not least, we express our gratitude to all authors of the accepted and submitted papers. Their contribution has made these proceedings a scholarly compilation of exciting research outcomes. August 2011

Jemal Abawajy Giancarlo Fortino Ragib Hasan Mustafizur Rahman

IDCS 2011 Organizing Committee

Workshop Chairs Jemal Abawajy Giancarlo Fortino Ragib Hasan Mustafizur Rahman

Deakin University, Australia University of Calabria, Italy Johns Hopkins University, USA IBM, Australia

Web, Publicity and Logistics Chair Al-Sakib Khan Pathan Mukaddim Pathan

International Islamic University, Malaysia CSIRO, Australia

International Program Committee Joaqu´ın Garc´ıa-Alfaro Doina Bein Rajkumar Buyya Antonio Coronato Mustafa Mat Deris Zongming Fei S.K. Ghosh Victor Govindaswamy Jaehoon Paul Jeong Syed Ishtiaque Ahmed Tarem Ahmed Mohammad Mehedi Hassan Dimitrios Katsaros Fahim Kawsar Ram Krishnan Hae Young Lee Ignacio M. Llorente Carlo Mastroianni Jaime Lloret Mauri Sudip Misra Muhammad Mostafa Monowar Manzur Murshed Marco Netto George Pallis Rajiv Ranjan

´ ECOM ´ TEL Bretagne, France Pennsylvania State University, USA University of Melbourne, Australia ICAR-CNR, Italy Universiti Tun Hussein Onn, Malaysia University of Kentucky, USA IIT-Kharagpur, India Texas A&M University-Texarkana, USA University of Minnesota, USA BUET, Bangladesh Brac University, Bangladesh Kyung Hee University, South Korea University of Thessaly, Greece Bell Labs, BE and Lancaster University, UK University of Texas at San Antonio, USA ETRI, South Korea Universidad Complutense de Madrid, Spain ICAR-CNR, Italy Universidad Polit´ecnica de Valencia, Spain IIT-Kharagpur, India University of Chittagong, Bangladesh Monash University, Australia IBM Research, Brazil University of Cyprus, Cyprus University of New South Wales, Australia

X

IDCS 2011 Organizing Committee

Thomas Repantis Riaz Ahmed Shaikh Ramesh Sitaraman Mostafa Al Masum Shaikh Paolo Trunfio Christian Vecchiola Spyros Voulgaris Anwar Walid Lizhe Wang Bin Xie Norihiko Yoshida

Akamai Technologies, USA University of Quebec in Outaouais, Canada University of Massachusetts, USA University of Tokyo, Japan University of Calabria, Italy University of Melbourne, Australia Vrije Universiteit, The Netherlands Alcatel-Lucent Bell Labs, USA Indiana University, USA InfoBeyond Technology, USA Saitama University, Japan

M2A2 Foreword

It is with great pleasure that we present the proceedings of the Third International Workshop on Multicore and Multithreaded Architectures and Algorithms (M2A2 2011) held in conjunction with the 11th International Conference on Algorithms and Architectures for Parallel Processing (ICA3PP 2011) in Melbourne, Australia. Multicore systems are dominating the processor market, and it is expected that the number of cores will continue to increase in most of the commercial systems, such as high-performance, desktops, or embedded systems. This trend is driven by the need to increase the efficiency of the major system components, that is, the cores, the memory hierarchy, and the interconnection network. For this purpose, the system designer must trade off performance versus power consumption, which is a major concern in current microprocessors. Therefore new architectures or architectural mechanisms addressing this trade-off are required. In this context, load balancing and scheduling can help to improve energy saving. In addition, it remains a challenge to identify and productively program applications for these architectures with a resulting substantial performance improvement. The M2A2 2011 workshop provided a forum for engineers and scientists to address the resulting challenge and to present new ideas, applications, and experience on all aspects of multicore and multithreaded systems. This year, and because of the high quality of the submitted papers, only about 38% of the papers were accepted for the workshop. We would like to express our most sincere appreciation to everyone contributing to the success of this workshop. First, we thank the authors of the submitted papers for their efforts in their research work. Then, we thank the TPC members and the reviewers for their invaluable and constructive comments. Finally, we thank our sponsors for their support of this workshop. August 2011

Houcine Hassan Julio Sahuquillo

XII

M2A2 Foreword

General Co-chairs Houcine Hassan Julio Sahuquillo

Universidad Politecnica de Valencia, Spain Universidad Politecnica de Valencia, Spain

Steering Committee Laurence T. Yang Jong Hyuk Park

St Francis Xavier University, Canada Seoul National University of Technology, Korea

Program Committee Hideharu Amano Hamid R. Arabnia Luca Benini Luis Gomes Antonio Gentile Zonghua Gu Rajiv Gupta Houcine Hassan Seongsoo Hong Shih-Hao Hung Eugene John Seon Wook Kim Jihong Kim Chang-Gun Lee Sebastian Lopez Yoshimasa Nakamura Sabri Pllana Julio Sahuquillo Zili Shao Kenjiro Taura

Keio University, Japan The University of Georgia, USA University of Bolonia, Italy Universidade Nova de Lisboa, Portugal Universit` a di Palermo, Italy University of Science and Technology, Hong Kong University of California, Riverside, USA Universidad Politecnica de Valencia, Spain Seoul National University, Korea National Taiwan University, Taiwan University of Texas at San Antonio, USA Korea University, Korea Seoul National University, Korea Seoul National University, Korea Universidad Las Palmas, Spain Kyoto University, Japan University of Vienna, Austria Universidad Politecnica de Valencia, Spain The Hong Kong Polytechnic University, Hong Kong University of Tokyo, Japan

HardBio 2011 Foreword

It gives us great pleasure to introduce this small collection of papers that were presented at the First International Workshop on Parallel Architectures for Bioinformatics Systems (HardBio 2011), October 23–26, 2011, Melbourne, Australia. Bioinformatics is a research field that focuses on algorithms and statistical techniques that allow efficient interpretation, classification and understanding of biological datasets. These applications are to the general benefit of mankind. The datasets typically consist of huge numbers of DNA, RNA, or protein sequences. Sequence alignment is used to assemble the datasets for analysis. Comparisons of homologous sequences, gene finding, and prediction of gene expression are the most common techniques used on assembled datasets. However, analysis of such datasets have many applications throughout all fields of biology. The down-side of bioinformatics-related applications is that they need a humongous computational effort to be executed. Therefore, a lot of research effort is being channeled towards the development of special-purpose hardware accelerators and dedicated parallel processors that allow for efficient execution of this kind of applications The Program Committee received 12 submissions, from which it selected 4 for presentation and publication. Each paper was evaluated by three referees. Technical quality, originality, relevance, and clarity were the primary criteria for selection. We wish to thank all these who submitted manuscripts for consideration. We also wish to thank the members of the Program Committee who reviewed all of the submissions. I hope that many more reserachers will submit the results of their work to next year’s workwhop. August 2011

Nadia Nedjah Luiza de Macedo Mourelle

XIV

HardBio 2011 Foreword

Program Committee Felipe Maia Galv˜ ao Fran¸ca Nader Bagherzadeh Leandro dos Santos Coelho Jurij Silc Heitor Silv´erio Lopes Lech J´ ozwiak Zhihua Cui Hamid Sarbazi-Azad

Federal University of Rio de Janeiro, Brazil University of California, Irvine, USA Pontifical Catholic University of Paran´ a, Brazil Jozef Stefan Institute, Slovenia Federal Technological University of Paran´ a, Brazil Eindhoven University of Technology, The Netherlands Taiyuan University of Science and Technology, China Sharif University of Technology, Iran

Table of Contents – Part II

ADCN 2011 Papers Lightweight Transactional Arrays for Read-Dominated Workloads . . . . . . Ivo Anjo and Jo˜ ao Cachopo Massively Parallel Identification of Intersection Points for GPGPU Ray Tracing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Alexandre Solon Nery, Nadia Nedjah, Felipe M.G. Fran¸ca, and Lech Jozwiak Cascading Multi-way Bounded Wait Timer Management for Moody and Autonomous Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Asrar Ul Haque and Javed I. Khan World-Wide Distributed Multiple Replications in Parallel for Quantitative Sequential Simulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Mofassir Haque, Krzysztof Pawlikowski, Don McNickle, and Gregory Ewing Comparison of Three Parallel Point-Multiplication Algorithms on Conic Curves . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yongnan Li, Limin Xiao, Guangjun Qin, Xiuqiao Li, and Songsong Lei Extending Synchronization Constructs in OpenMP to Exploit Pipeline Parallelism on Heterogeneous Multi-core . . . . . . . . . . . . . . . . . . . . . . . . . . . . Shigang Li, Shucai Yao, Haohu He, Lili Sun, Yi Chen, and Yunfeng Peng

1

14

24

33

43

54

Generic Parallel Genetic Algorithm Framework for Protein Optimisation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Lukas Folkman, Wayne Pullan, and Bela Stantic

64

A Survey on Privacy Problems and Solutions for VANET Based on Network Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Hun-Jung Lim and Tai-Myoung Chung

74

Scheduling Tasks and Communications on a Hierarchical System with Message Contention . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Jean-Yves Colin and Moustafa Nakechbandi

89

Spiking Neural P System Simulations on a High Performance GPU Platform . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Francis George Cabarle, Henry Adorna, Miguel A. Mart´ınez–del–Amor, and Mario J. P´erez–Jim´enez

99

XVI

Table of Contents – Part II

SpotMPI: A Framework for Auction-Based HPC Computing Using Amazon Spot Instances . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Moussa Taifi, Justin Y. Shi, and Abdallah Khreishah

109

Investigating the Scalability of OpenFOAM for the Solution of Transport Equations and Large Eddy Simulations . . . . . . . . . . . . . . . . . . . . Orlando Rivera, Karl F¨ urlinger, and Dieter Kranzlm¨ uller

121

Shibboleth and Community Authorization Services: Enabling Role-Based Grid Access . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Fan Gao and Jefferson Tan

131

A Secure Internet Voting Scheme . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Md. Abdul Based and Stig Fr. Mjølsnes

141

A Hybrid Graphical Password Based System . . . . . . . . . . . . . . . . . . . . . . . . Wazir Zada Khan, Yang Xiang, Mohammed Y. Aalsalem, and Quratulain Arshad

153

Privacy Threat Analysis of Social Network Data . . . . . . . . . . . . . . . . . . . . . Mohd Izuan Hafez Ninggal and Jemal Abawajy

165

IDCS 2011 Papers Distributed Mechanism for Protecting Resources in a Newly Emerged Digital Ecosystem Technology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ilung Pranata, Geoff Skinner, and Rukshan Athauda Reservation-Based Charging Service for Electric Vehicles . . . . . . . . . . . . . . Junghoon Lee, Gyung-Leen Park, and Hye-Jin Kim Intelligent Ubiquitous Sensor Network for Agricultural and Livestock Farms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Junghoon Lee, Hye-Jin Kim, Gyung-Leen Park, Ho-Young Kwak, and Cheol Min Kim

175 186

196

Queue-Based Adaptive Duty Cycle Control for Wireless Sensor Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Heejung Byun and Jungmin So

205

Experimental Evaluation of a Failure Detection Service Based on a Gossip Strategy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Leandro P. de Sousa and Elias P. Duarte Jr.

215

On the Performance of MPI-OpenMP on a 12 Nodes Multi-core Cluster . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Abdelgadir Tageldin Abdelgadir, Al-Sakib Khan Pathan, and Mohiuddin Ahmed

225

Table of Contents – Part II

XVII

A Protocol for Discovering Content Adaptation Services . . . . . . . . . . . . . . Mohd Farhan Md Fudzee and Jemal Abawajy

235

Securing RFID Systems from SQLIA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Harinda Fernando and Jemal Abawajy

245

Modeling QoS Parameters of VoIP Traffic with Multifractal and Markov Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Homero Toral-Cruz, Al-Sakib Khan Pathan, and Julio C. Ram´ırez-Pacheco Hybrid Feature Selection for Phishing Email Detection . . . . . . . . . . . . . . . Isredza Rahmi A. Hamid and Jemal Abawajy

255

266

M2A2 2011 Papers On the Use of Multiplanes on a 2D Mesh Network-on-Chip . . . . . . . . . . . . Cruz Izu A Minimal Average Accessing Time Scheduler for Multicore Processors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Thomas Canhao Xu, Pasi Liljeberg, and Hannu Tenhunen

276

287

Fast Software Implementation of AES-CCM on Multiprocessors . . . . . . . . Jung Ho Yoo

300

A TCM-Enabled Access Control Scheme . . . . . . . . . . . . . . . . . . . . . . . . . . . . Gongxuan Zhang, Zhaomeng Zhu, Pingli Wang, and Bin Song

312

Binary Addition Chain on EREW PRAM . . . . . . . . . . . . . . . . . . . . . . . . . . . Khaled A. Fathy, Hazem M. Bahig, Hatem M. Bahig, and A.A. Ragb

321

A Portable Infrastructure Supporting Global Scheduling of Embedded Real-Time Applications on Asymmetric MPSoCs . . . . . . . . . . . . . . . . . . . . . Eugenio Faldella and Primiano Tucci Emotional Contribution Process Implementations on Parallel Processors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Carlos Dom´ınguez, Houcine Hassan, Jos´e Albaladejo, Maria Marco, and Alfons Crespo A Cluster Computer Performance Predictor for Memory Scheduling . . . . M´ onica Serrano, Julio Sahuquillo, Houcine Hassan, Salvador Petit, and Jos´e Duato

331

343

353

XVIII

Table of Contents – Part II

HardBio 2011 Papers Reconfigurable Hardware Computing for Accelerating Protein Folding Simulations Using the Harmony Search Algorithm and the 3D-HP-Side Chain Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . C´esar Manuel Vargas Ben´ıtez, Marlon Scalabrin, Heitor Silv´erio Lopes, and Carlos R. Erig Lima Clustering Nodes in Large-Scale Biological Networks Using External Memory Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ahmed Shamsul Arefin, Mario Inostroza-Ponta, Luke Mathieson, Regina Berretta, and Pablo Moscato Reconfigurable Hardware to Radionuclide Identification Using Subtractive Clustering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Marcos Santana Farias, Nadia Nedjah, and Luiza de Macedo Mourelle

363

375

387

A Parallel Architecture for DNA Matching . . . . . . . . . . . . . . . . . . . . . . . . . . Edgar J. Garcia Neto Segundo, Nadia Nedjah, and Luiza de Macedo Mourelle

399

Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

409

Table of Contents – Part I

ICA3PP 2011 Keynote Keynote: Assertion Based Parallel Debugging . . . . . . . . . . . . . . . . . . . . . . . . David Abramson

1

ICA3PP 2011 Regular Papers Secure and Energy-Efficient Data Aggregation with Malicious Aggregator Identification in Wireless Sensor Networks . . . . . . . . . . . . . . . . Hongjuan Li, Keqiu Li, Wenyu Qu, and Ivan Stojmenovic

2

Dynamic Data Race Detection for Correlated Variables . . . . . . . . . . . . . . . Ali Jannesari, Markus Westphal-Furuya, and Walter F. Tichy

14

Improving the Parallel Schnorr-Euchner LLL Algorithm . . . . . . . . . . . . . . Werner Backes and Susanne Wetzel

27

Distributed Mining of Constrained Frequent Sets from Uncertain Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Alfredo Cuzzocrea and Carson K. Leung

40

Set-to-Set Disjoint-Paths Routing in Recursive Dual-Net . . . . . . . . . . . . . . Yamin Li, Shietung Peng, and Wanming Chu

54

Redflag: A Framework for Analysis of Kernel-Level Concurrency . . . . . . . Justin Seyster, Prabakar Radhakrishnan, Samriti Katoch, Abhinav Duggal, Scott D. Stoller, and Erez Zadok

66

Exploiting Parallelism in the H.264 Deblocking Filter by Operation Reordering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Tsung-Hsi Weng, Yi-Ting Wang, and Chung-Ping Chung Compiler Support for Concurrency Synchronization . . . . . . . . . . . . . . . . . . Tzong-Yen Lin, Cheng-Yu Lee, Chia-Jung Chen, and Rong-Guey Chang

80 93

Fault-Tolerant Routing Based on Approximate Directed Routable Probabilities for Hypercubes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Dinh Thuy Duong and Keiichi Kaneko

106

Finding a Hamiltonian Cycle in a Hierarchical Dual-Net with Base Network of p -Ary q-Cube . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yamin Li, Shietung Peng, and Wanming Chu

117

XX

Table of Contents – Part I

Adaptive Resource Remapping through Live Migration of Virtual Machines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Muhammad Atif and Peter Strazdins LUTS: A Lightweight User-Level Transaction Scheduler . . . . . . . . . . . . . . . Daniel Nic´ acio, Alexandro Baldassin, and Guido Ara´ ujo Verification of Partitioning and Allocation Techniques on Teradata DBMS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ladjel Bellatreche, Soumia Benkrid, Ahmad Ghazal, Alain Crolotte, and Alfredo Cuzzocrea Memory Performance and SPEC OpenMP Scalability on Quad-Socket x86 64 Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Daniel Molka, Robert Sch¨ one, Daniel Hackenberg, and Matthias S. M¨ uller Anonymous Communication over Invisible Mix Rings . . . . . . . . . . . . . . . . . Ming Zheng, Haixin Duan, and Jianping Wu Game-Based Distributed Resource Allocation in Horizontal Dynamic Cloud Federation Platform . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Mohammad Mehedi Hassan, Biao Song, and Eui-Nam Huh

129

144

158

170

182

194

Stream Management within the CloudMiner . . . . . . . . . . . . . . . . . . . . . . . . . Yuzhang Han, Peter Brezany, and Andrzej Goscinski

206

Security Architecture for Virtual Machines . . . . . . . . . . . . . . . . . . . . . . . . . . Udaya Tupakula, Vijay Varadharajan, and Abhishek Bichhawat

218

Fast and Accurate Similarity Searching of Biopolymer Sequences with GPU and CUDA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Robert Pawlowski, Bo˙zena Malysiak-Mrozek, Stanislaw Kozielski, and Dariusz Mrozek Read Invisibility, Virtual World Consistency and Probabilistic Permissiveness are Compatible . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Tyler Crain, Damien Imbs, and Michel Raynal Parallel Implementations of Gusfield’s Cut Tree Algorithm . . . . . . . . . . . . Jaime Cohen, Luiz A. Rodrigues, Fabiano Silva, Renato Carmo, Andr´e L.P. Guedes, and Elias P. Duarte Jr. Efficient Parallel Implementations of Controlled Optimization of Traffic Phases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Sameh Samra, Ahmed El-Mahdy, Walid Gomaa, Yasutaka Wada, and Amin Shoukry

230

244

258

270

Table of Contents – Part I

Scheduling Concurrent Workflows in HPC Cloud through Exploiting Schedule Gaps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . He-Jhan Jiang, Kuo-Chan Huang, Hsi-Ya Chang, Di-Syuan Gu, and Po-Jen Shih Efficient Decoding of QC-LDPC Codes Using GPUs . . . . . . . . . . . . . . . . . . Yue Zhao, Xu Chen, Chiu-Wing Sham, Wai M. Tam, and Francis C.M. Lau

XXI

282

294

ICA3PP 2011 Short Papers A Combined Arithmetic Logic Unit and Memory Element for the Design of a Parallel Computer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Mohammed Ziaur Rahman Parallel Implementation of External Sort and Join Operations on a Multi-core Network-Optimized System on a Chip . . . . . . . . . . . . . . . . . . . . Elahe Khorasani, Brent D. Paulovicks, Vadim Sheinin, and Hangu Yeo STM with Transparent API Considered Harmful . . . . . . . . . . . . . . . . . . . . . Fernando Miguel Carvalho and Joao Cachopo A Global Snapshot Collection Algorithm with Concurrent Initiators with Non-FIFO Channel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Diganta Goswami and Soumyadip Majumder An Approach for Code Compression in Run Time for Embedded Systems – A Preliminary Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Wanderson Roger Azevedo Dias, Edward David Moreno, and Raimundo da Silva Barreto Optimized Two Party Privacy Preserving Association Rule Mining Using Fully Homomorphic Encryption . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Md. Golam Kaosar, Russell Paulet, and Xun Yi SLA-Based Resource Provisioning for Heterogeneous Workloads in a Virtualized Cloud Datacenter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Saurabh Kumar Garg, Srinivasa K. Gopalaiyengar, and Rajkumar Buyya ΣC: A Programming Model and Language for Embedded Manycores . . . Thierry Goubier, Renaud Sirdey, St´ephane Louise, and Vincent David Provisioning Spot Market Cloud Resources to Create Cost-Effective Virtual Clusters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . William Voorsluys, Saurabh Kumar Garg, and Rajkumar Buyya

306

318

326

338

349

360

371

385

395

XXII

Table of Contents – Part I

A Principled Approach to Grid Middleware: Status Report on the Minimum Intrusion Grid . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Jost Berthold, Jonas Bardino, and Brian Vinter

409

Performance Analysis of Preemption-Aware Scheduling in Multi-cluster Grid Environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Mohsen Amini Salehi, Bahman Javadi, and Rajkumar Buyya

419

Performance Evaluation of Open Source Seismic Data Processing Packages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Izzatdin A. Aziz, Andrzej M. Goscinski, and Michael M. Hobbs

433

Reputation-Based Resource Allocation in Market-Oriented Distributed Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Masnida Hussin, Young Choon Lee, and Albert Y. Zomaya

443

Cooperation-Based Trust Model and Its Application in Network Security Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Wu Liu, Hai-xin Duan, and Ping Ren

453

Performance Evaluation of the Three-Dimensional Finite-Difference Time-Domain(FDTD) Method on Fermi Architecture GPUs . . . . . . . . . . . Kaixi Hou, Ying Zhao, Jiumei Huang, and Lingjie Zhang

460

The Probability Model of Peer-to-Peer Botnet Propagation . . . . . . . . . . . . Yini Wang, Sheng Wen, Wei Zhou, Wanlei Zhou, and Yang Xiang

470

A Parallelism Extended Approach for the Enumeration of Orthogonal Arrays . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Hien Phan, Ben Soh, and Man Nguyen

481

Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

495

Lightweight Transactional Arrays for Read-Dominated Workloads Ivo Anjo and Jo˜ ao Cachopo ESW INESC-ID Lisboa/Instituto Superior T´ecnico/Universidade T´ecnica de Lisboa Rua Alves Redol 9, 1000-029 Lisboa, Portugal {ivo.anjo,joao.cachopo}@ist.utl.pt

Abstract. Many common workloads rely on arrays as a basic data structure on top of which they build more complex behavior. Others use them because they are a natural representation for their problem domains. Software Transactional Memory (STM) has been proposed as a new concurrency control mechanism that simplifies concurrent programming. Yet, most STM implementations have no special representation for arrays. This results, on many STMs, in inefficient internal representations, where much overhead is added while tracking each array element individually, and on other STMs in false-sharing conflicts, because writes to different elements on the same array result in a conflict. In this work we propose new designs for array implementations that are integrated with the STM, allowing for improved performance and reduced memory usage for read-dominated workloads, and present the results of our implementation of the new designs on top of the JVSTM, a Java library STM. Keywords: Parallel Programming, Software Transactional Memory.

1

Introduction

Software Transactional Memory (STM) [10, 15] is a concurrency control mechanism for multicore and multiprocessor shared-memory systems, aimed at simplifying concurrent application development. STM provides features such as atomicity and isolation for program code, while eliminating common pitfalls of concurrent programming such as deadlocks and data races. During a transaction, most STMs internally work by tracking the memory read and write operations done by the application on thread-local read and write-sets. Tracking this metadata adds overheads to applications that depend on the granularity of transactional memory locations. There are two main STM designs regarding granularity: Either word-based [4, 8] or object-based [7, 11]. Wordbased designs associate metadata with either each individual memory location, or by mapping them to a fixed-size table; whereas object-based designs store 

This work was supported by FCT (INESC-ID multiannual funding) through the PIDDAC Program funds and by the RuLAM project (PTDC/EIA-EIA/108240/2008).

Y. Xiang et al. (Eds.): ICA3PP 2011 Workshops, Part II, LNCS 7017, pp. 1–13, 2011. c Springer-Verlag Berlin Heidelberg 2011 

2

I. Anjo and J. Cachopo

transactional information on each object or structure’s header, and all of the object’s fields share the same piece of transactional metadata. Arrays, however, are not treated specially by STM implementations. Thus, programmers either use an array of transactional containers in each position, or they wrap the entire array with a transactional object. Neither option is ideal, if we consider that array elements may be randomly but infrequently changed. Because arrays are one of the most elemental data structures on computing systems, if we hope to extend the usage of STM to provide synchronization and isolation to array-heavy applications, minimizing the imposed overhead is very important. In this paper, we describe how existing transactional arrays are implemented, and explore new approaches that are integrated with the STM, achieving better performance and reducing memory usage for read-dominated workloads. Our work is based on the Java Versioned Software Transactional Memory (JVSTM) [2, 3], a multi-version STM. The rest of this work is organized as follows. Section 2 introduces the JVSTM transactional memory. Section 3 describes current black-box approaches to arrays. Section 4 introduces the new proposals for handling arrays. In Section 5, we compare the different array implementations. Experimental results are presented in Section 6, followed, in Section 7, by a survey of related work. Finally, in Section 8, we finish by presenting the conclusions and future research directions.

2

The JVSTM Software Transactional Memory

The Java Versioned Software Transactional Memory (JVSTM) is a pure Java library implementing an STM [3]. JVSTM introduces the concept of versioned boxes [2], which are transactional locations that may be read and written during transactions, much in the same way of other STMs, except that they keep the history of values written to them by any committed transaction. Programmers using the JVSTM must use instances of the VBox class to represent the shared mutable variables of a program that they want to access transactionally. In Java, those variables are either class fields (static or not) or array components (each element of an array). As an example, consider a field f of type T in a class C whose instances may be accessed concurrently. To access f transactionally, the programmer must do two things: (1) transform the field f in C into a final field that holds an instance of type VBox, and (2) replace all the previous accesses to f by the corresponding operations on the contents of the box now contained in f. JVSTM implements versioned boxes by keeping a linked-list of VBoxBody instances inside each VBox: Each VBoxBody contains both the version number of the transaction that committed it and the value written by that transaction. This list of VBoxBody instances is sorted in descending order of the version number, with the most recent at the head. The key idea of this design is that transactions typically need to access the most recent version of a box, which is only one indirection-level away from the box object.

Lightweight Transactional Arrays for Read-Dominated Workloads

3

Yet, because the JVSTM keeps all the versions that may be needed by any of the active transactions, a transaction that got delayed for some reason can still access a version of the box that ensures that it will always perform consistent reads: The JVSTM satisfies the opacity correctness criteria [9]. In fact, a distinctive feature of the JVSTM is that read-only transactions are lock-free and never conflict with other transactions. They are also very lightweight, because there is no need to keep read-sets or write-sets: Each read of a transactional location consists only of traversing the linked-list to locate the correct VBoxBody from which the value is to be read. These two characteristics make the JVSTM especially suited for applications that have a high read/write transaction ratio. Currently there are two versions of the JVSTM that differ on their commit algorithm. The original version of the JVSTM uses a lock-based commit algorithm, described below, whereas more recently Fernandes and Cachopo described a lock-free commit algorithm for the JVSTM [6]. Unless otherwise stated, the approaches described in this paper apply to both versions of the JVSTM. To synchronize the commits of read-write transactions, the lock-based JVSTM uses a single global lock: Any thread executing a transaction must acquire this lock to commit its results, which means that all commits (of read-write transactions) execute in mutual exclusion. After the lock acquisition, the committing transaction validates its read-set and, if valid, writes-back its values to new VBoxBody instances, which are placed at the head of each VBox’s history of values. To prevent unbounded growth of the memory used to store old values for boxes, the JVSTM implements a garbage collection algorithm, which works as follows: Each committing transaction creates a list with all the newly created instances of VBoxBody and stores this list on its descriptor. The transaction descriptors themselves also form a linked-list of transactions, with increasing version numbers. When the JVSTM detects that no transactions are running with version number older than some descriptor, it cleans the next field of each VBoxBody instance in the descriptor, allowing the Java GC to clean the old values.

3

Current Black-Box Array Implementations

In this section, we describe the two most common alternatives to implement transactional arrays with the JVSTM if we use only its provided API — that is, if we use the JVSTM as a black-box library. 3.1

Array of Versioned Boxes

The most direct and commonly used way of obtaining a transactional array with the JVSTM is the array of VBoxes. A graphical representation of the resulting structure is shown in Figure 1. One of the shortcomings of this approach is the array initialization: All positions on the array need to be initialized with a VBox before they are used, typically as soon as the array is created and before it is published. Trying to perform lazy initialization highlights one of the issues of implementing such a data-structure outside the STM: the underlying native Java array is

4

I. Anjo and J. Cachopo





  







































Fig. 1. Array of versioned boxes

Fig. 2. Versioned box with array

not under the control of the STM, and as such the programmer must provide his own synchronization mechanism for this operation. Side-stepping the synchronization provided by the STM while at the same time using the STM must be done carefully, or key STM characteristics might be lost, such as lock-freedom and atomicity, and common concurrent programming issues such as deadlocks might arise again. We will see in Section 4.1 a variant of this approach that uses lazy initialization and knowledge of the JVSTM’s internals. Since all VBoxes and their associated VBoxBody instances are normal Java objects, they still take up a considerable amount of memory when comparing to the amount needed to store each reference on the VBox array. As such, it is not unexpected for the application to spend more than twice the space needed for the native array to store these instances in memory. 3.2

Versioned Box with Array

The other simple implementation of a transactional array is one where a single VBox keeps the entire array, as shown in Figure 2. Creation of this kind of array is straightforward, with overheads comparable to a normal non-transactional array. Array reads are the cheapest possible, only adding the cost of looking up the correct VBoxBody to read from; but writes are very expensive, as they need to duplicate the entire array just to change one of the positions. In addition, a single array write conflicts with every other (non read-only) transaction that is concurrently accessing the array, as the conflict detection granularity is the VBox holding the entire array. Moreover, there is a very high overhead in keeping the history of values: For each version, an entire copy of the array is kept, even if only one element of the array was changed. This may lead the system to run out of memory very quickly, if writes to the array are frequent and some old running transaction prevents the garbage collector from running. In conclusion, this approach is suited only for very specific workloads, with zero or almost-zero writes to the array. On the upside, for those workloads, it offers performance comparable to native arrays, while still benefiting from transactional properties. It is also the only approach that allows the underlying array to change size and dimensions dynamically with no extra overhead.

Lightweight Transactional Arrays for Read-Dominated Workloads

5

Type value = getVBox(index).get(); // Reading from a VBoxArray getVBox(index).put(newValue); // Writing to a VBoxArray VBox getVBox(int index) { // Helper method getVBox VBox vbox = transArray[index]; if (vbox == null) { vbox = new VBox((VBoxBody) null); vbox.commit(null, 0); if (!unsafe.compareAndSwapObject(transArray, ..., null, vbox)) vbox = transArray[index]; } return vbox; }

Fig. 3. Code for the

4

VBoxArray

approach

New Array Proposals

In this section, we describe three proposals to implement transactional arrays that improve on the black-box approaches presented in the previous section. 4.1

VBoxArray and VBodyArray

The VBoxArray approach is obtained by adding lazy creation and initialization of VBoxes to the approach presented in Section 3.1. The main operations for this implementation are shown in Figure 3. The getVBox() helper method first tries to obtain a VBox from the specified array position. If it exists, it is returned; otherwise a new one is created using an empty body, that is immediately written back, and tagged with version 0. This is conceptually the same as if the VBox was created by a transaction that ran before every other transaction and initialized all the boxes. The VBox is then put into the array in an atomic fashion: Either the compareAndSwap1 operation succeeds, and the box is placed on the underlying array, or it fails, meaning that another thread already initialized it. We can take the VBoxArray one step further and obtain the VBodyArray by doing away with the VBoxes altogether. The insight is that a VBox is needed only to uniquely identify a memory location on which we can transactionally read and write. If we provide our transactional array inside a wrapper VBodyArray class, we can use another method to identify uniquely a memory position: a pair . Using this pair, we no longer need the VBoxes, because the underlying array can directly contain the VBoxBody instances that would normally be kept inside them; initialization can still be done lazily. The VBodyArray saves a considerable amount of memory for larger arrays, and also lowers overhead on reads, as less memory reads need to be done to reach the values. 1

Available in the

sun.misc.Unsafe

class included in most JVM implementations.

6

I. Anjo and J. Cachopo







 




Fig. 4. The




 


VArray

transactional array

Type value = array.values.get(index); // Read value from array (volatile read!) int version = array.version; // Read array version // If the array did not change, return the value read, otherwise check the log if (version <= currentTransactionVersion) return value; Type logValue = array.log.getLogValue(index, currentTransactionVersion); return logValue != null ? logValue : value;

Fig. 5. Reading from a

4.2

VArray

VArray

The VArray approach, shown in Figure 4, does away entirely with the normal storage mechanisms in the JVSTM: No VBoxes and no VBoxBodies are needed. It is designed to have the upsides of the VBox with Array approach described in Section 3.2, but to eliminate or minimize the downsides.2 The main design idea is to have an array that keeps both a set of values tagged with a version and a log containing the remaining versions. Based on this design, two strategies are possible: – The underlying array keeps the oldest values for each array position, and newer values are kept in the log; there must be a strategy to decide when to transfer values from the log to the main array. – The underlying array keeps the latest values for each array position, and older values are kept in the log; there must be a strategy to allow garbage collection of older values from the log. We argue that the second choice is more in line with the spirit of the JVSTM’s design, because newer transactions find their values quickly, while older longrunning transactions have to search through the log to find their older values. Additionally, JVSTM’s existing garbage collection algorithm can, with minor modifications, be used to perform garbage collection of the log. Reading from a VArray. We start by reading the value directly from the array and then we check the array version: If it is older than the current transaction version number, we may return the value that we read directly from the array. If, instead, an older value is needed, we have to check the log to find the value corresponding to our current version, and return it, if found; otherwise, we may safely return the value originally read from the array because that position was never changed, although other array positions were. Figure 5 shows the code to read from a VArray. 2

The full source-code for the JVSTM with the VArray class is available on the jvstmlock-free branch at http://groups.ist.utl.pt/esw-inesc-id/git/jvstm.git/

Lightweight Transactional Arrays for Read-Dominated Workloads

7

int txNumber; // Version of transaction being committed VArrayEntry[] writesToCommit; // Sorted list of writes to be committed // Create and initialize logEntryIndexes to be used in the log int[] logEntryIndexes = new int[writesToCommit.length]; for (int i = 0; i < writesToCommit.length; i++) logEntryIndexes[i] = writesToCommit[i].index; // Create and place log node Type[] logEntryValues = (Type[]) new Object[writesToCommit.length]; array.log = new VArrayLogNode(logEntryIndexes, logEntryValues, txNumber - 1, array.log); // Bump array version array.version = txNumber; // Writeback values int i = 0; for (VArrayEntry entry : writesToCommit) { // Read old value from the array, and copy it to the log logEntryValues[i++] = array.values.get(entry.index); // Write the new value array.values.lazySet(entry.index, entry.object); // Volatile write! }

Fig. 6. Committing changes to a
  

VArray


  







      



  







   

Fig. 7. Structure of a

     


VArrayLogNode

Writing and Committing to a VArray. Writing to a VArray is similar to writing to a VBox: the value to be written is added to the transaction’s write-set. During the commit, the write-back to a VArray proceeds as follows: 1. Create a new log entry with the indexes of the array positions that are going to be overwritten and add that entry to the head of the log; 2. Update the array version; 3. Finally, backup to the log and write-back each changed array position. Each write operation is done with volatile semantics. These steps need to be done while inside the commit lock. On the lock-free version of the JVSTM, because there is no commit lock, each array is locked individually; note that this approach eliminates the property of lock-freedom from the commit of transactions that wrote to a VArray, but lock-freedom is restored when no active transactions are writing to VArray instances. Figure 6 shows a simplified version of the VArray commit code. Accessing the Log. Figure 7 shows the structure of a VArrayLogNode. The VArray log is a linked list of VArrayLogNodes containing the older values of array positions that were overwritten by newer transactions. Inside each VArrayLogNode, a version

8

I. Anjo and J. Cachopo

field keeps the last version when the values contained in it were valid. This means that if the log contains two nodes, with versions 50 and 40, then the node with version 50 is valid for transactions with a version in the range ]40, 50] and the older entry is valid for transactions with versions ]0, 40]. Moreover, each VArrayLogNode maintains two arrays: logEntryIndexes and logEntryValues. The first keeps an ordered list of indexes that were changed by the transaction that created the log node, and the second keeps the values that were at those indexes, and were overwritten in the main array. When a transaction with version n needs to look up the log for the value in index index, it first traverses the log nodes until it finds the log node with the smallest version >= n. It then checks that node for the index, by performing a binary search on the logEntryIndexes array. If this search finds the index, it returns the corresponding value. Otherwise, the search is resumed from the previous node, until a value is found, or the beginning of the log is reached — meaning that the requested value should be read from the main array. Synchronization. As we saw, the read algorithm first reads the value from the array, and then reads its version. To commit a new value we reverse this order: First the committer updates the version, and then writes back the new values. Yet, without additional synchronization, we have a data race and the following can happen: The update of the array value may be reordered with the update of the version, which means that a reader may read the new value written by the committing transaction, but still read the old version value, causing the algorithm to return an invalid (newer) value to the application. To solve this issue, and taking into account the Java memory model [12] we might be inclined to make the field that stores the array version volatile. Unfortunately, this will not work: If the committing thread first does a volatile write on the array version, and then updates the array, if the reading thread does not observe the write to the array version, no synchronizes-with 3 relation happens, and so the update to the array value may be freely reordered before the version write, making a reader read the new value, and miss the new version. The other possible option would be for the committing thread to first write-back the value, and then update the array version with a volatile write; in this case, a simple delay or context switch between the two writes would cause issues. As such, we can see that no ordering of writes to update both the array value and version can work correctly if just the version is declared volatile. As it turns out, the commit algorithm works correctly if only the array value is read and written with volatile semantics (through the usage of the AtomicReferenceArray class), and the version as a normal variable. This way, the reader can never read a newer value and an old version, because by volatile definition, if we observe a value, we at least observe the correct version for that value, but may also 3

The volatile keyword, when applied to a field states that if a thread t1 writes to normal field f1 and then to volatile field f2 ; then if other thread observes the write on f2, it is guaranteed that it will also see the write to f1, and also every other write done by t1 before the write to f2. This is called a synchronizes-with [12] relationship.

Lightweight Transactional Arrays for Read-Dominated Workloads

9

 % &  

Table 1. Comparison of array implementations. The memory overheads are considered for two workloads: a workload where only a single position is ever used after the array is created, and one where the entire array is used.


 
             


                !  "    "    !    ! 
 # 
 $
 # $
 
 # 
 $
 # $


    "       $
 # $
 
 # 


 

     " "     $
 


    " " ! "  

observe a later version, which poses no problem: In both cases the algorithm will correctly decide to check the log. Garbage Collection. We also extended the JVSTM garbage collection algorithm to work with the VArray log. As the linked list structure of the array log is similar to the linked list of bodies inside a VBox, new instances of VArrayLogNode that are created during transaction commit are also saved in the transaction descriptor, and from then the mechanism described in Section 2 is used.

5

Comparison of Approaches

Table 1 summarizes the key characteristics of the multiple approaches described in this paper. The single position memory overhead test case considers an array of n positions, where, after creation, only one of those positions is ever used during the entire program; conversely the entire array test case considers one where every position of the array is used. The memory overheads considered are in addition to a native array of size n, which all implementations use. The main objective of this work was the creation of an array implementation that provided better performance for read-only operations, while minimizing memory usage and still supporting write operations without major overheads. We believe VArray fulfills those objectives, as it combines the advantages of the “VBox with Array” approach, such as having a very low memory footprint and read overhead, with advantages from other approaches, notably conflict detection done at the array position level, and low history overhead. Writes to a VArray are still more complex than most other approaches, but as we will see in Section 6 they can still be competitive.

6

Experimental Results

We shall now present experimental results of the current implementation of VArray. They were obtained on two machines: one with two Intel Xeon E5520 processors (8 cores total) and 32GB of RAM, and another with four AMD Opteron 6168 processors (48 cores total) and 128GB of RAM, both running Ubuntu

10

I. Anjo and J. Cachopo   
















  





    



!"#$% #!

  



 











  
















 





Fig. 8. Comparison of VArray versus the Array of VBoxes approach for the array benchmark, with a read-only workload on our two test systems     




     
  
     
  

   

  




















 

Fig. 9. Comparison of VArray versus the Array of VBoxes approach for the array benchmark, with varying number of read-write transactions (10%, 50% and 100%) on the 48-core AMD machine

10.04.2 LTS 64-bit and Oracle Java 1.6.0 22. For our testing, we compared VArray to the Array of VBoxes approach, using the array benchmark,4 which can simulate multiple array-heavy workloads. Before each test, the array was entirely initialized— note that after being fully initialized, the Array of Versioned Boxes and VBoxArray behave similarly. Each test was run multiple times, and the results presented are the average over all executions. Figure 8 shows the scaling of VArray versus the Array of VBoxes approach for a read-only workload, with a varying number of threads. Each run consisted of timing the execution of 1 million transactions, with an array size of 1,000,000 on the 8-core machine, and 10,000,000 on the 48-core machine. Due to the reduced overheads imposed on array reads, VArray presents better performance. Figure 9 shows the scaling of VArray versus a Array of VBoxes approach for a workload with a varying percentage of read-only and read-write transactions. Each read-only transaction reads 1000 (random) array positions, and each 4

http://web.ist.utl.pt/sergio.fernandes/darcs/array/

Lightweight Transactional Arrays for Read-Dominated Workloads

11

read-write transaction reads 1000 array positions and additionally writes to 10. Each run consisted of timing the execution of 100,000 transactions. As we can see, the increased write overhead of VArray eventually takes its toll and beyond a certain number of cores (that depend on the percentage of read-write transactions), VArray presents worse results than the Array of VBoxes approach. These results show that while VArray is better suited for read-only workloads, if needed it can still support a moderate read-write workload. To test the memory overheads of VArray, we measured the minimum amount of memory needed to run a read-only workload in the array benchmark, on a single CPU, for an array with 10 million Integer objects. Due to its design, VArray was able to complete the benchmark using only 57MB of RAM, 10% of the 550MB needed by the Array of VBoxes approach. Finally, we measured, using a workload comprised of 10% read-write transactions and 90% read-write transactions, and 4 threads, the minimum memory needed for both approaches to present acceptable performance, when compared with a benchmark run with a large heap. In this test, VArray took approximately 25% longer to execute with a 256MB heap, when compared to a 3GB heap; runs with an Array of VBoxes needed at least 800MB and also took 25% longer.

7

Related Work

Software Transactional Memory (STM) [15] is an optimistic approach to concurrency control on shared-memory systems. Many implementations have been proposed — Harris et al.’s book [10] provides a very good overview of the subject. CCSTM [1] is a library-based STM for Scala based on SwissTM [5]. Similarly to the JVSTM, the programmer has to explicitly make use of a special type of reference, that mediates access to a STM-managed mutable value. Multiple memory locations can share the same STM metadata, enabling several levels of granularity for conflict detection. The CCSTM also provides a transactional array implementation that eliminates some of the indirections needed to access transactional metadata, similar to our VBodyArray approach. The DSTM2 [11] STM framework allows the automatic creation of transactional versions of objects based on supplied interfaces. Fields on transactional objects are allowed to be either scalar or other transactional types, which disallows arrays; to work around this issue, the DSTM2 includes the AtomicArray class that provides its own specific synchronization and recovery, but no further details on its implementation are given. Another approach to reducing the memory footprint of STM metadata on arrays and other data structures is changing the granularity of conflict detection. Word-based STMs such as Fraser and Harris’s WSTM [8] and TL2 in per-stripe mode [4] use a hash function to map memory addresses to a fixed-size transactional metadata table; hash collisions may result in false positives, but memory usage is bounded to the chosen table size. Marathe et al. [13] compared word-based with object-based STMs, including the overheads added and memory usage; one of their conclusions is that the studied systems incur significant bookkeeping overhead for read-only transactions.

12

I. Anjo and J. Cachopo

Riegel and Brum [14] studied the impact of word-based versus object-based STMs for unmanaged environments, concluding that object-based STMs can reach better performance than purely word-based STMs. Our VArray implementation is novel because it presents the same memory overheads of word-based schemes, while still detecting conflicts for each individual array position. Processing overhead for read-write transactions is still larger than with word-based approaches, because the transaction read-set must contain all individual array positions that were read, and all of them must be validated at commit-time, which is something word-based STMs can further reduce.

8

Conclusions and Future Work

Software transactional memory is a very promising approach to concurrency. Still, to expand into most application domains, many research and engineering issues need to be examined and solved. The usage of arrays is one such issue. In this work we presented the first comprehensive analysis of transactional array designs, described how arrays are currently implemented on top of the JVSTM, and presented two implementations that improve on previous designs. In particular, the VArray implementation has memory usage comparable to native arrays, while preserving the lock-free property of JVSTM’s read-only transactions. In addition, our experimental results show that VArray is highly performant for readdominated workloads, and competitive for read-write workloads. Future research directions include researching the possibility of a lock-free VArray commit algorithm, and exploring the usage of bloom filters for log lookups.

References 1. Bronson, N., Chafi, H., Olukotun, K.: CCSTM: A library-based STM for Scala 2. Cachopo, J., Rito-Silva, A.: Versioned boxes as the basis for memory transactions. Science of Computer Programming 63(2), 172–185 (2006) 3. Cachopo, J.: Development of Rich Domain Models with Atomic Actions. Ph.D. thesis, Technical University of Lisbon (2007) 4. Dice, D., Shalev, O., Shavit, N.: Transactional locking II. In: Dolev, S. (ed.) DISC 2006. LNCS, vol. 4167, pp. 194–208. Springer, Heidelberg (2006) 5. Dragojevi´c, A., Guerraoui, R., Kapalka, M.: Stretching transactional memory. ACM SIGPLAN Notices 44, 155–165 (2009) 6. Fernandes, S., Cachopo, J.: Lock-free and scalable multi-version software transactional memory. In: Proceedings of the 16th ACM Symposium on Principles and Practice of Parallel Programming, pp. 179–188. ACM, New York (2011) 7. Fraser, K., Harris, T.: Practical lock-freedom. Tech. rep. (2004) 8. Fraser, K., Harris, T.: Concurrent programming without locks. ACM Trans. Comput. Syst. 25 (2007) 9. Guerraoui, R., Kapalka, M.: On the correctness of transactional memory. In: PPoPP 2008: Proceedings of the 13th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, pp. 175–184. ACM, New York (2008) 10. Harris, T., Larus, J., Rajwar, R.: Transactional memory. Synthesis Lectures on Computer Architecture 5(1), 1–263 (2010)

Lightweight Transactional Arrays for Read-Dominated Workloads

13

11. Herlihy, M., Luchangco, V., Moir, M.: A flexible framework for implementing software transactional memory. ACM SIGPLAN Notices 41(10), 253–262 (2006) 12. Manson, J., Pugh, W., Adve, S.: The Java Memory Model 13. Marathe, V.J., Scherer, W.N., Scott, M.L.: Design tradeoffs in modern software transactional memory systems. In: Proceedings of the 7th Workshop on Workshop on Languages, Compilers, and Run-Time Support for Scalable Systems, LCR 2004, pp. 1–7. ACM, New York (2004) 14. Riegel, T., Brum, D.B.D.: Making object-based STM practical in unmanaged environments. In: TRANSACT 2008: 3rd Workshop on Transactional Computing (2008) 15. Shavit, N., Touitou, D.: Software transactional memory. Distributed Computing 10(2), 99–116 (1997)

Massively Parallel Identification of Intersection Points for GPGPU Ray Tracing Alexandre S. Nery1,3 , Nadia Nedjah2 , Felipe M.G. Fran¸ca1 , and Lech Jozwiak3 1

LAM – Computer Architecture and Microeletronics Laboratory, Systems Engineering and Computer Science Program, COPPE, Universidade Federal do Rio de Janeiro 2 Department of Electronics Engineering and Telecommunications, Faculty of Engineering, Universidade do Estado do Rio de Janeiro 3 Department of Electrical Engineering – Electronic Systems, Eindhoven University of Technology, The Netherlands

Abstract. The latest advancements in computer graphics architectures, as the replacement of some fixed stages of the pipeline for programmable stages (shaders), have been enabling the development of parallel general purpose applications on massively parallel graphics architectures (Streaming Processors). For years the graphics processing unit (GPU) is being optimized for increasingly high throughput of massively parallel floating-point computations. However, only the applications that exhibit Data Level parallelism can achieve substantial acceleration in such architectures. In this paper we present a parallel implementation of the GridRT architecture for GPGPU ray tracing. Such architecture can expose two levels of parallelism in ray tracing: parallel ray processing and parallel intersection tests, respectively. We also present a traditional parallel implementation of ray tracing in GPGPU, for comparison against the GridRT-GPGPU implementation.

1

Introduction

High-fidelity computer generated images is one of the main goals in the Computer Graphics field. Given a 3-D scene, usually described by a set of 3-D primitives (e.g. triangles), a typical rendering algorithm creates a corresponding image by several matrix computations and space transformations applied to the 3-D scene, together with many per-vertex shading computations [1]. All these computations are organized in pipeline stages, each one performing many SIMD floating-point operations, in parallel. The Graphics Processing Unit (GPU) is also known as a Stream Processor, because of such massively parallel pipeline organization, that continuously processes a stream of input data through pipeline stages. In the final stage, all primitives are rasterized to produce an image (a.k.a. frame). In order to achieve real-time rendering speed it is necessary to produce at least 60 frames per second (fps), so that the change between frames is not perceived and interactivity is ensured. The Streaming Processor model of current GPU architectures can deliver enough throughput of frame rates for most 3-D scenarios, but at the cost Y. Xiang et al. (Eds.): ICA3PP 2011 Workshops, Part II, LNCS 7017, pp. 14–23, 2011. c Springer-Verlag Berlin Heidelberg 2011 

Massively Parallel Identification of Intersection Points

15

of a lower degree of realism in each produced frame. For example, important Global Illumination effects like shadows and reflections must be handled at the application level, because the hardware is based on a Local Illumination model and, thus, is especialized in processing 3-D primitives only [4]. Although the ray tracing algorithm [10] is also a computer graphics application for rendering 3-D scenes, the algorithm operates particularly in opposition to traditional rendering algorithms [1]. For instance, instead of projecting the primitives to the viewplane, where the final image is produced, the ray tracing algorithm fires rays towards the scene and traces their path in order to identify what are the visible objects, their properties and the light trajectory within the scene, through several intersection computations. In the end, all these informations are merged to produce the final image. For that reason, ray tracing is a high computation cost application that can produce high-fidelity images of a 3-D scene, with shadow and reflection effects. Besides, the algorithm has a very high parallelization potential, because each ray can be processed independently from the others, usually achieving almost linear acceleration for a parallel implementation [4]. Thus, there are parallel implementations on Clusters [11] and Shared Memory Systems [2], using spatial subdivision of the 3-D scene. Parallel implementations in GPGPUs (General Purpose Graphics Processing Units) have also achieved substantial results [9]. Some stages of the pipeline, such as the Vertex and Geometry processing stages, have recently evolved to programmable Shaders, that can be programmed to perform different algorithms [6]. So, the GPU is no longer dedicated to run graphic related algorithms, but also general purpose parallel algorithms that can benefit from the massively parallel architectue of modern GPGPUs. For instance, Data Level parallel applications in general achieve high acceleration when mapped to GPGPUs, because these applications perform well in SIMD machines [9]. However, if control flow and recursion are strongly required, which is often the case for ray tracing, then existing Von Neumann architectures may be a better option for Task Level parallel applications. In ray tracing, every ray can be processed independently, in parallel, but each ray must be tested for intersections against the primitives of the 3-D scene and, if there is such intersection, the computation may proceed in many different ways. So, the work granularity is in the task level and each task may execute through different branches, which makes control flow and recursion a big issue for ray tracing. So, there are consistent approaches to accelerate ray tracing with custom parallel architectures in hardware, as in [8,13], operating at low frequencies. Hence, the low frequency of operation is compensated by the parallelism of the custom design and several limitations can be overcome by a custom hardware design. In general, the target device is a Field Programmable Gate Array (FPGA), which can be used to prototype the design, and later an Application Specific Integrated Circuit (ASIC) can be produced, operating at much higher frequencies. Throughout this paper we briefly describe our GridRT parallel architecture for ray tracing and we present a GPGPU implementation of the architecture

16

A.S. Nery et al.

in CUDA, exhibiting Task Level parallelism of rays and Data Level parallelism of intersection computations. The CUDA kernel algorithm, which corresponds to the GridRT parallel architecture with some minor modifications, is also described. In the end, we present performance results for two modern NVidia Fermi GPUs, the GTX 460 and GTX 465. Furthermore, we describe a traditional parallel ray tracer implementation in GPGPU, for comparison with the GridRTGPGPU. The rest of this paper is organized as follows. First, Section 2 briefly explains the ray tracing algorithm. Then, Section 3 shows a traditional parallel implementation of the algorithm in GPGPUs. After that, Section 4 presents the GridRT architecture before the GPGPU implementation is presented in Section 5. Finally, Section 6 presents performance results, while Section 7 draws the conclusion of this work.

2

Ray Tracing

The ray tracing algorithm is briefly explained in this section, while further details can be found in [10]. Thus, the first step of the algorithm is the setup of a virtual camera, so that primary rays can be fired towards the scene. Each primary ray passes through a pixel of the camera’s viewplane, where the final image is going to be captured. For every primary ray, a simple and straightforward ray tracing algorithm usually computes intersection tests against all the 3-D primitives of the scene, looking for the primitives (objects) that are visible from the camera’s perspective. If an intersection is encountered, the object properties are used to determine wether the ray will be reflected, refracted or completely absorbed. For instance, if the ray is reflected or refracted, the algorithm is recursively executed to determine the objects that are visible from the previous intersection point perspective, which is why the algorithm can naturally produce mirror like effects in the final image. On the other hand, if the ray is absorbed, the processing ends and all the information that has been gathered until that point is merged to compose the color of the corresponding pixel of the viewplane. This ray tracing style is known as Whitted-Style ray tracing [12]. The program main entry is presented in Algorithm 1, in which the primary rays are being traced. The trace procedure in Algorithm 1 is responsible for determining the closest intersection point, while the shade procedure (called by the trace procedure) is responsible for coloring the pixel and recursively calling the trace procedure in case the intersected object surface is specular or transparent. For the sake of simplicity and brevity, these two procedures are not described in this work. Further details on shading algorithms can be found in [1]. In order to avoid intersection computations between each ray and the whole scene, a spatial subdivision of scene can be applied to select only those objects that are in the direction of a given ray, avoiding unnecessary computation. There are several spatial subdivision techniques, such as Binary Space Partitioning Trees, Boundary Hierarchical Volumes and KD-Trees and Uniform Grids [10,1], each one of them with its own advantages and disadvantages. For instance, the

Massively Parallel Identification of Intersection Points

17

Algorithm 1. Ray Tracing primary rays 1 2 3 4 5 6 7 8

3-D scene = load3DScene(file); viewplane = setupViewplane(width,height); camera = setupCamera(viewplane,eye,view direction); depth = 0; for i = 1 to viewplane’s width do for j = 1 to viewplane’s height do ray = getPrimaryRay(i,j,camera); image[i][j] = trace(3-D scene, ray, depth);

KD-Tree structure adapts very well to the 3-D scene and, hence, selects fewer objects than the other techniques. However, the KD-Tree building time is more expensive and complex, as well as the algorithm that is used to traverse the tree structure [5]. On the other hand, the Uniform Grid structure is less expensive to build and the traversal algorithm is very fast [10], but such structure is not adaptive and, because of that, may select a few more objects for intersection tests or perform extra traversal steps through empty areas of the 3-D scene. In this work we use the Uniform Grid structure, which is the base of the GridRT parallel architecture [8].

3

Traditional Parallel GPGPU Ray-Tracer

The ray tracing algorithm exhibits parallelism at the task level. Each ray can be processed independently from the others and each one can be assigned to an execution process or thread across different computing nodes or processing elements. So, one ray is going to be processed as a task, producing one pixel at the end of the computation. The main idea is to spread tasks (rays) across different processes. Also, it is possible to assign a group of tasks per process, instead of one task only. In the end, each task or group of tasks will produce the color of one or more pixels of the final image, respectively. Modern general purpose GPUs are capable of executing many thousands of threads in parallel [6], achieving peaks of 1TFLOPs or more. Thus, in modern GPGPUs, each thread can be assigned to a primary ray that crosses a pixel of the viewplane. The result is that a portion of the final image is going to be produced by a block of threads (one pixel per thread). The number of blocks of threads corresponds to as many subdivisions the image is going to be split into, which corresponds to distributing primary rays among threads. The given CUDA Kernel is presented in Algorithm 2, considering that all data transfers between the host and the GPGPU have already been performed. Note that this algorithm does not have the loop construction presented in lines 5 and 6 of the sequential Algorithm 1 version, because now each ray has been assigned to a thread of a block of threads. Every block of threads has its own identifier, as every thread too. In that way each thread can access its own data to process. So, in Algorithm 2, the given thread uses its own identifiers to select the corresponding ray that

18

A.S. Nery et al.

Algorithm 2. Traditional parallel GPGPU ray tracer CUDA-Kernel 1 2 3 4 5

int i = blockDim.x * blockIdx.x + threadIdx.x; int j = blockDim.y * blockIdx.y + threadIdx.y; ray = rays[i][j]; color = trace(3-D scene,ray,depth); image[i][j] = color;

will be traced, resulting in one pixel color. Depending on the configuration that is set on Kernel launch, the identifiers can have up to three coordinates. In the case of Algorithm 2, only two coordinates are used (i, j), because the data (primary rays) is organized in two dimensions. In the end, the whole image will have been produced by parallel threads that processed one primary ray each, together with any secondary rays that may have been generated for each intersection test.

4

The GridRT Parallel Architecture

Before explaining the GridRT implementation in GPGPU we explain the GridRT parallel architecture, which can be implemented in any kind of multiprocessor system, such as Clusters, Chip-Multiprocessors, GPGPUs or custom parallel designs in FPGA. The GridRT architecture is strongly based on the Uniform Grid structure. In such spatial subdivision scheme the 3-D scene is split into regions of equal size, called voxels. Each voxel has a list of primitives (triangles) that are inside it or almost inside it. Thus, only those voxels that are pierced by a ray are going to be sequentially accessed for intersection tests, from the voxel that is closest to the ray origin to the furthest. Therefore, if an intersection is found, no more tests are required for the given ray, because it is already the closest to the ray origin. In the example depicted in Fig. 1a, three voxels were accessed until an intersection t1 was found in voxel v5 . On the other hand, the GridRT parallel model maps each voxel onto a Processing Element (PE), as depicted in Fig. 1b. So, intersection tests are performed in

v0

Uniform Grid 4x4x1 v1 v2 v3

PE0

Uniform Grid 4x4x1 PE1 PE2 PE3 block of threads

voxel v4

v5

v6

v7

PE4

PE5

t1 v8

v9

v10

v11 ray origin

v12

v13

v14

v15

(a) Uniform Grid

PE8

PE9

B4

PE13

B5

PE10

PE14

B6

PE11

PE15

(b) Parallel GridRT

B7 t2

t1 ray origin

PE12

Uniform Grid 4x4x1 B1 B2 B3

t2

t1

ray origin

PE7

PE6

B0

B8

B9

B10

B11

B12

B13

B14

B15

(c) GridRT-GPGPU

Fig. 1. Sequential Uniform Grid, Parallel GridRT model and GridRT in GPGPU

Massively Parallel Identification of Intersection Points

19

parallel by those PEs that are pierced by a ray and, because of that, it becomes necessary to decide which PE holds the result that is closest to the ray origin. At first, one solution is to exchange the results between the PEs, but it would require every PE to wait for the others to finish their computation on the given ray before deciding which one holds the correct results. Thus, the GridRT parallel model uses the order of traversal for each ray to determine the correct result. For instance, in Fig. 1b, P E5 and P E6 have found an intersection each, t1 and t2 . According to the ray traversal order, P E5 is closer to the ray origin. Thus, P E5 may send an interrupt message to the next PE in the list, so it can abort its computation and forward the interrupt message to the next one, until every following PE is aborted. The computation is now reduced between the remaining PEs. If one of them finds an intersection within its piece of scene data, it can also proceed in the same way, sending interruption messages to the following ones in the list. Otherwise, if none of them finds an intersection, a feedback message is sent from the first to the last remaining PEs. Such message is used to ensure that none of the previous PEs in the list have found intersection tests. Then, the remaining PE holds the correct result, like P E5 of Fig. 1b, or none of them. Note that each PE needs to communicate such messages between their direct neighbors, which depends on the target architecture that the parallel model is going to be implemented. For example, if each PE is mapped onto a Process running on different computation nodes, the messages can be exchanged via Message Passing Interface (MPI) [7]. But if the target architecture is a FPGA parallel design, then the neighborhood of PEs can be connected by interrupt signals. Further details on the GridRT architecture and its communication model can be found in [8].

5

GridRT-CUDA Parallel Ray-Tracer

Following the GridRT parallel architecture presented in the previous section, the GridRT-CUDA implementation maps each voxel onto a block of threads, as depicted in Fig. 1c. Thus, every block of threads is performing intersection tests along a ray, in parallel. Also, the intersection tests are performed in parallel inside a block of threads. Thus, two levels of parallelism are exhibited in such organization. The first one is in the task level parallelism, while the second one is in the data level parallelism. For instance, if a given block has n triangles and n or more threads at disposal, then the intersection tests are performed in parallel by the threads of the block. Otherwise, parallel intersection tests can be performed in chunks of n triangles by n threads. In this work, the 3-D scene is such that there are always enough threads to process all the triangles in parallel inside the block, as will be presented in Section 6. However, in order to determine the correct result among the blocks of threads, a different approach from the one presented in Section 4 had to be developed, because threads from different blocks of threads cannot coordinate their activities. Only threads inside the same block can coordinate their activities through a Shared Memory. Thus, a given block of threads cannot inform the next block in the traversal list about its computation results.

20

A.S. Nery et al.

Algorithm 3. GridRT-CUDA Kernel 1 2 3 4 5

shared float res[max. number of triangles per voxel]; foreach ray i do foreach element j of the corresponding traversal list do if this blockId is in the list then if there are triangles in this block then

6

res[threadIdx.x ] = intersectTriangles(ray,vertices [threadIdx.x ]);

7

syncthreads();

8

if threadIdx.x = 0 then /* Finds the smallest result /* Copy the result to global memory

*/ */

Therefore, we let the host processor determine the correct result at the end of the whole computation, also according to the order of traversal presented in Section 4. Hence, each ray has an array of results associated to it and the size of an array corresponds to the maximum number of PEs, i.e. blocks, that can be traversed by a given ray. The size is given by the total number of subdivisions applied to each of the three axis (nx , ny , nz ) of the GridRT spatial structure, as defined in Eq. 1. For instance, considering the grid of Fig. 1c, the maximum size of the array is N = 7, since the uniform grid subdivision is nx = 4, ny = 4 and nz = 1. N = nx + (ny − 1) + (nz − 1)

(1)

When each block of threads have finished the intersection checks with respect to the corresponding voxel, the result is stored in the array at the block associated entry. Thereafter, the block can proceed with the computation of a different ray, which also has a different array of results associated to it. In the end, the matrix of results is copied from the GPU to the host processor, so it can proceed with further analysis of results. Each row of the matrix corresponds to the array of results computed by the block for a given ray, while each column contains the result that was computed by a block. The algorithm that is executed by a block of threads is shown in Algorithm 3. Each block of threads takes as input an array of rays, which has also an array of results associated to each row, thus yielding a matrix. The 3-D scene is already copied to the GPU before the kernel execution. The scene is stored according to the uniform grid structure as an unidimensional array. Each position of the array points to the list of triangles that belongs to the corresponding voxel (i.e. block of threads). Once the necessary data has been copied to the GPU, the kernel is launched. According to Algorithm 3, the first step taken by each block is to declare an array of shared results, as in line 1. This shared array is used to store the results from the parallel intersection tests, as in line 6. For each ray in the input data, the block will search for its own identifier in the traversal list, as in lines 3 and 4. Then, if there are any triangles in the block, parallel intersection tests are performed by the threads. Finally, one of the threads (the

Massively Parallel Identification of Intersection Points

21

Table 1. GridRT-CUDA kernel execution times in GTX 460 and GTX 465 Blocks of threads GridRT GTX 460 GridRT GTX 465 TradRT GTX 465

1 2 4 8 12 18 27 64 - 1.69 0.87 0.92 0.97 1.07 2.39 - 1.34 0.63 0.61 0.59 0.55 1.2 0.94 0.70 0.47 0.35 0.28 0.23 0.21 0.17

125 3.85 2.41 0.14

216 7.03 4.02 0.12

*All times are in seconds. Low-res Stanford Bunny 3-D scene.

one with identifier zero) searches the smallest result (that is closest to the ray origin, in respect to that block) from the array of shared results, as in line 8.

6

Results

In this section we present the comparison results between our GridRT-CUDA implementation in two different Nvidia GPU’s (GTX 460, 465) and also the results for our traditional parallel ray-tracer in GPGPU. These results are summarized in Table 1, respectively. The execution times for configurations of 1 and 2 blocks of threads are not available for the GridRT implementation, because the execution was terminated due to kernel execution timeout. A second dedicated GPGPU graphics card could have been used to avoid this limitation. Otherwise, the same GPU has to be shared between the host Operating System applications and thus cannot execute long time CUDA kernels (up to tens of seconds). Also, because of kernel execution timeout limitation, we could not use higher resolution 3-D scenes. As we can observe from Table 1, the GridRT-CUDA implementation achieves acceleration in both GPU models. However, the performance starts to degenerate when more than 8 blocks of threads are used by the GTX 460 or more than 27 blocks of threads are used by the GTX 465. The latter scales better because it has 11 Streaming Multiprocessors (SMs), while the first has 7 SMs. In essence, a block of threads is executed by a SM. The more SMs are available, the more blocks of threads can be executed in parallel. The results from Table 1 for the GridRT-CUDA are depicted in Fig. 2a. If more SMs were available, more acceleration was likely to be achieved. The kernel execution times for a traditional parallel ray tracer in CUDA are depicted in Fig. 2b, together with the GridRT-CUDA. In contrast, the traditional parallel ray-tracer implementation uses a different approach. Parallelism is employed at the ray (task) level only. Thus, blocks of threads are not mapped to voxels of the uniform grid subdivision. Instead, blocks of threads are mapped to groups of primary rays that are going to be traced in parallel, as presented in Section 3. So, each thread is in fact processing a independent primary ray and its corresponding secondary rays and shadow rays that may be spawned by the algorithm. In the end, each thread produces the color of an individual pixel of the final image. From Table 1 and Fig. 2b, it is clear that this version of ray

22

A.S. Nery et al.

(a) GridRT-CUDA kernel execution time in GTX 460 and GTX 465.

(b) Traditional parallel CUDA ray tracing compared to GridRT-CUDA.

Fig. 2. Execution time results and comparisons

tracing scales almost linearly to the number of blocks of threads. The explanation for such acceleration is also in the GPGPU architecture itself: if two or more threads are going to execute through different branches, they are serialized [3]. Hence, we can see from the GridRT-CUDA (Algorithm 3) that there are several possible branches of execution, that can lead to serialization of threads. For that reason, a custom parallel design in FPGA is preferable, because the architecture can be designed according to the characteristics of the application. For instance, although the execution time in [8] is higher, the acceleration is much higher as more processing elements can be fit into the FPGA.

7

Conclusion

In this paper, two different implementations of parallel ray tracing are discussed: the GridRT-CUDA implementation and a traditional CUDA parallel ray tracer. These two implementations are analyzed an compared regarding performance. The GridRT-CUDA implementation achieves acceleration up to 27 blocks of threads in a Nvidia GTX 465 GPU and up to 8 blocks of threads in a Nvidia GTX 460 GPU. From that point, the performance degenerates, especially because of the Streaming Processor model, which is not good for applications that exhibit too many branches of execution, such as in the GridRT architecture. So, several threads were serialized. Also, the performance degenerates because many blocks of threads have to compete for execution on the GPU fewer resources. A more powerful GPU is likely to achieve higher acceleration for further more blocks of threads. Compared to the traditional GPGPU ray tracer, the GridRT-CUDA performance is not good. However, since the GPGPU implementation introduces more hardware overhead compared to a custom hardware design (ASIP-based ASIC implementation), the custom hardware implementation is expected to have lower area and power consumption, as well as better performance.

Massively Parallel Identification of Intersection Points

23

References 1. Akenine-M¨ oller, T., Haines, E., Hoffman, N.: Real-Time Rendering, 3rd edn. A.K. Peters, Ltd., Natick (2008) 2. Carr, N.A., Hall, J.D., Hart, J.C.: The ray engine. In: HWWS 2002: Proceedings of the ACM SIGGRAPH/EUROGRAPHICS Conference on Graphics Hardware, pp. 37–46. Eurographics Association, Aire-la-Ville (2002) 3. Fung, W.W.L., Sham, I., Yuan, G., Aamodt, T.M.: Dynamic warp formation and scheduling for efficient gpu control flow. In: Proceedings of the 40th Annual IEEE/ACM International Symposium on Microarchitecture, MICRO 40, pp. 407– 420. IEEE Computer Society, Washington, DC, USA (2007) 4. Govindaraju, V., Djeu, P., Sankaralingam, K., Vernon, M., Mark, W.R.: Toward a multicore architecture for real-time ray-tracing. In: Proceedings of the 41st Annual IEEE/ACM International Symposium on Microarchitecture, MICRO 41, pp. 176– 187. IEEE Computer Society, Washington, DC, USA (2008) 5. Havran, V., Prikryl, J., Purgathofer, W.: Statistical comparison of ray-shooting efficiency schemes. Technical report, Institute of Computer Graphics and Algorithms, Vienna University of Technology, Favoritenstrasse 9-11/186, A-1040 Vienna, Austria (2000) 6. Kirk, D.B., Hwu, W.-m.W.: Programming Massively Parallel Processors: A Handson Approach. Morgan Kaufmann Publishers Inc., San Francisco (2010) 7. Nery, A.S., Nedjah, N., Fran¸ca, F.M.G.: Two alternative parallel implementations for ray tracing: Openmp and mpi. In: Mecnica Computacional, vol. XXiX, pp. 6295–6302. Asociacin Argentina de Mecnica Computacional (2010) 8. Nery, A.S., Nedjah, N., Fran¸ca, F.M.G., Jozwiak, L.: A parallel architecture for ray-tracing with an embedded intersection algorithm. In: International Symposium on Circuits and Systems, pp. 1491–1494. IEEE Computer Society, Los Alamitos (2011) 9. Owens, J.D., Luebke, D., Govindaraju, N., Harris, M., Kruger, J., Lefohn, A.E., Purcell, T.J.: A survey of general-purpose computation on graphics hardware. Computer Graphics Forum 26(1), 80–113 (2007) 10. Suffern, K.: Ray Tracing from the Ground Up, 1st edn. A.K. Peters, Ltd., Natick (2007) 11. Wald, I., Ize, T., Kensler, A., Knoll, A., Parker, S.G.: Ray tracing animated scenes using coherent grid traversal. In: SIGGRAPH 2006: ACM SIGGRAPH 2006 Papers, pp. 485–493. ACM, New York (2006) 12. Whitted, T.: An improved illumination model for shaded display. Commun. ACM 23(6), 343–349 (1980) 13. Woop, S., Schmittler, J., Slusallek, P.: Rpu: a programmable ray processing unit for realtime ray tracing. In: SIGGRAPH 2005: ACM SIGGRAPH 2005 Papers, pp. 434–444. ACM, New York (2005)

Cascading Multi-way Bounded Wait Timer Management for Moody and Autonomous Systems Asrar Ul Haque1 and Javed I. Khan2 1

College of Computer Science and Information Tech., King Faisal University, Al-Ahsa 31982, Kingdom of Saudi Arabia [email protected] 2 Media Communications and Networking Research Laboratory Department of Math & Computer Science, Kent State University 233 MSB, Kent, OH 44242 [email protected]

Abstract. Timer management is one of the central issues in addressing the ‘moody’ and autonomous characteristics of current Internet. In this paper we formalize the multi-way bounded wait principle for ‘moody’ and autonomous environment. We propose an optimum scheme and compare it with a set of generalized heuristic-based timer management schemes recommended for the harness-a distributed communication al and computational system for moody and autonomous environment. Keywords. Optimum timeout scheme, Timeout heuristics, Grid Computing, P2P search, Web service.

1 Introduction Any distribute system with millions of components must learn to operate with incomplete information. This is becoming the case of various distributed systems operating over the Internet. A classic example is the search for service discovery [1-2]. Such a distributed search is quite different from conventional distributed algorithms. A particular unique characteristic of such a search is that it is never complete. The search propagates via millions of other nodes from a source to the entire network as illustrated in Fig. 1. While it is ideal to expect that answers will arrive from a sweep covering all the nodes in the network, but almost always that is never the case. A search must learn to adapt to work with an imperfect sweep. An interesting question faced by this set of distributed algorithms is how to maximize the quality of the result without waiting inordinate amount of time. The network-based distributed algorithms are run in an environment consisting of various manifestations of this inherent unreliability such as dead-beat node, unreliable or busy peer, missing messages, authentication failure, intentional non-cooperation, selective cooperation etc. We will call it Moody and Autonomous Environment (MAE) environment. Classical communication layer has handled only limited aspects and forms of this unreliability. Various schemes such as error resilience coding and retransmission based transport essentially tell how a communication can best try to Y. Xiang et al. (Eds.): ICA3PP 2011 Workshops, Part II, LNCS 7017, pp. 24–32, 2011. © Springer-Verlag Berlin Heidelberg 2011

Cascading Multi-way Bounded Wait Timer Management

25

Fig. 1. (Top) A client begins searching for items. The search then propagates through thousands of other nodes. Each plays role in forwarding/routing results back to the root of the request. (Bottom) The root peer waits for the results.

create a ‘perfect’ and fault-free notion of services for mostly point-to-point communication. Classical network algorithms running on top of such transport thus assume that the imperfection can be sealed-off at lower layers and it can operate over a virtual perfection. Unfortunately this assumption of virtual perfection is does not always hold in the emerging distributed MAE. All natural communications systems for the MAE use the principle of bounded wait to address the moodiness property. An entity in the MAE while running a distributed algorithm faces the dilemma of how long it should wait for some other entity – if it waits too long it delays the overall computation completion time or even it may miss the timer deadline of its parent. Conversely, if it waits too short a period then it may miss communication from many of its children. Consequently, this deteriorates the overall quality of computation. This paper formalizes the above multi-way bounded wait

26

A. Ul Haque and J.I. Khan

principle for general distributed algorithms for MAE. We suggest an optimum scheme and compare it with a set of generalized heuristic-based timer management. The solution is applicable to any schema of multi-way communication- whether inside a new protocol, at middle-layer, or as a pure application level effort. In this paper we present the proposed timer management scheme within a formal framework of multi-way communication based general distributed computing. We call it harness [3]. The harness is a network computing framework which has a reusable multi-way communication primitive designed to operate in MAE. The harness makes a separation between the communication and the information part of the data exchange. Then a set of six plug-ins allows the computation and communication part to be separately programmed. The messaging pattern and the message content can be programmed independent of each other. Essentially a set of pre-programmed communication patterns can be reused with another set of message synthesis procedures. The harness has been shown to solve various network algorithms. In this section- we briefly introduce the harness framework. Details can be found [3]. The paper is arranged in the following way. First in section 2 we provide a brief overview of various interesting related work. We then formalize the multi-way bounded wait problem and solve it to set the timer optimally in section 3. For comparison, we also provide a set of plausible heuristics. Finally in section 5 we provide a detail performance comparison of the heuristics and the optimum schemes.

2 Related Work As the problem is increasingly becoming real, recently, various timeout schemes have been proposed for a range of large scale network applications. Network Weather Service [4]-a Grid performance data management and forecasting service, is one the first to try dynamic timeout in forecast computing and noted substantial improvement over static scheme. A RTO (retransmission timeout) selection has been proposed [5] for multimedia to achieve the optimal tradeoff between the error probability and the rate cost in order to improve throughput. Timeout strategy has been proposed associating costs for waiting time and retransmission attempts [6] where the timeout value was set to minimize the overall expected cost. The goal of this work was to reduce the number of retransmission attempts. In [7] a scheme was proposed for Deterministic Timeouts for Reliable Multicast (DTRM) to avoid negative feedback explosion [7]. DTRM ensures that retransmission caused by only one NACK from a receiver belonging to a sub-tree arrives early enough so that the timers do not expire in the other receivers in that sub-tree. These recent schemes- have used various heuristics approximations, but are notable because of there pioneering role in the timeout management in multi-way communication in moody environment. The solution we propose assumes that the nodes have prior knowledge of link delay. Indeed, there has been considerable amount of work related to finding link-level delay. Network tomography [8-12] based on statistical estimation theory is an emerging field of characterizing various link level performance metrics of a network. Network tomography closely estimates network internal characteristics including delay distribution, loss, and delay variance by correlating end-to-end measurements for

Cascading Multi-way Bounded Wait Timer Management

27

multicast tree. However, a major limitation of Network tomography is that it focuses on multicast routing only whereas the bulk of the traffic is uni-cast. This has been overcome this by estimating delay distribution by employing uni-cast, end-to-end measurement of back to back packets [13].

3 Multi-way Bounded Wait Principle A node during running a search algorithm might aggregate messages received from a set of downstream children and forwarding to a parent. As messages are received from its children the computation is more accurate since each message contains search results from some other nodes as well. A node faces the dilemma of how long it should wait for the message – if it waits too long it delays the overall computation completion time or even it may miss the timer deadline of its parent. Conversely, if it waits too short a period then it may miss communication from many of its children. Consequently, this deteriorates the overall quality of computation. In this section, the above multi-way bounded wait principle is formalized and an optimum scheme is suggested for setting timers for message generation and aggregation at the intermediate nodes and the terminals. In the following subsections, in order to generalize the problem formulation, we assume the timer value for message aggregation at the root is denoted by D (deadline) and the timer value for message generation of its child is represented by T. Furthermore, we use the notion of profit (denoted by ω ) to signify the total number of nodes pertaining to which search results have been accumulated in a message. 3.1 Formulation of the Problem Let, as shown in Fig. 2, node j has a parent k and a set of children nodes i={ i1, i2,… in}. Let ri x j (t ) and rk(t) be the probability distribution function of round trip time between

k

rkj (t ) j

ri1 j (t ) i1

rin j (t )

ri2 j (t ) i2

ωi j 1

ωi

in 2

j

ωi

Fig. 2. Optimal Timer Setting for Node j

n

j

28

A. Ul Haque and J.I. Khan

the nodes ix and j and, j and k where x={1..n}. Let

ωi

x

j

be the profit carried by a mes-

sage from node ix to j. Given D the timeout value of k, calculate the maximum value of expected profit from j to k and the corresponding timeout value Topt for j. 3.2 Generic Profit Function Formulation T

The question we now pose is how to maximize the profit-P(t). Let C (t )dt be the ∫ 0

D −T

∫ S (t )dt

total profit accumulation at j in time t and

be the probability of successfully

0

reaching the parent, k, in time (D-T) with the accumulated profit. So we show that the basket function is the product of profit accumulation and probability of successful delivery. T

D −T

0

0

P(t ) = ∫ C (t )dt

∫ S (t )dt

(1)

The profit accumulation is summation of the product of profit and delay distribution of each children of j i.e., T

T

0

0 s∈i

∫ C (t )dt = ∫ ∑ ωsj rsj (t )dt D −T

D −T

0

0

(2)

∫ S (t )dt = ∫ r (t )dt

(3)

k

From (1), (2), and (3) we get, T

D

0 s∈i

T

P( D) = ∫ ∑ ω sj rsj (t )dt ∫ rk (t )dt

(4)

Fig. 3 illustrates the formulation of the profit function for node j as in Eq. 1. As the time T is increased the area under ri1 j (t ), ri2 j (t ), and ri n j (t ) also increases indicatD −T

ing higher accumulation of profit C(t). However the area under

∫r

k

(t )dt i.e.

0

D −T

∫ S (t )dt decreases as T increases since D is fixed. Thus possibility of reaching parent 0

Cascading Multi-way Bounded Wait Timer Management

29

T

node k with the accumulated profit C (t )dt diminishes as T is increased. The product ∫ 0

D −T

of

T

∫ S (t )dt and ∫0 C (t )dt is the total profit accumulated at node k in time T. 0

3.3 Solution for Optimal Timer Setting The optimum time, Topt, and the corresponding maximum profit, Pmax, are calculated based on the following assumptions: • The delay distributions of the parent and children of j are of Normal distribution. • The delay distributions of the children of j are independent of each other.

Fig. 3. Formulation of Profit Function

30

A. Ul Haque and J.I. Khan

• Node j generates a message for its parent even if no message is received from its children before timeout occurs at j. Further simplification of Eq. 4 is beyond the scope of this paper. However, it can be noted that P(D), in Eq. 4 is a positive, bounded, and continuous function of T. Furthermore, as T → ±∞, it goes to zero. Therefore, a global maximum of P(D) and the corresponding T, denoted by Topt, must exist, and must satisfy the equation

dP ( D) / dT = 0.

4 Simulation In this section the performance of the optimal scheme is partially presented. To put the performance of the optimum scheme into perspective we have also constructed a number of timer-setting heuristics and picked five best performing ones. We present the optimum scheme in the context of these intuitive solutions. These heuristics are shown in Table 2 and are discussed in [3]. The marginal constant and factors used in simulation for the heuristics were manually selected after extensively looking into performance of individual heuristics in various network scenarios. We assume ϕ , β , α , ρ , ξ , σ , and

λ

to be 40000, 10000, 10, 6.5, 10, 3000, and 1.5 respectively.

4.1 Measuring Parameters One of the important parameters of interest is the completion time (CT) which is the time elapsed between root generating the first request and receiving and processing all the replies. However, completion time alone can not justify usage of a heuristic. To better compare various heuristics, we define capturing efficiency (CE) as the ratio of number of node from which information has been collected at the root divided by total number of responsive nodes. Let N m be overall count of nodes from which information has propagated to the root node,

N t be total number of nodes, and

N NRN number of NRN. Then,

CE =

Nm N t − N NRN

(5)

4.2 Impact of Size of Network Fig. 4 and 5 illustrate impact of size on the heuristics with respect to CT and CE respectively. The CT for optimal scheme for 2500, 5000 and 10000 nodes are 10.11,13.54, and 12.9 seconds respectively. For MCD, CT increases by 2.3s as graph size increases from 2500 to 10000 nodes where as for other heuristics the increase is noticeable. However, CE is not 1 for all three graph sizes for MCD. For 10000 nodes, CE is only 0.72. The optimal scheme has CE=1.0 for al three graph sizes.

Cascading Multi-way Bounded Wait Timer Management Table 1. Various Timeout Heuristics

Ti h = η ih−1 + β where β ≠ f (i, L, RTT ) Ti h = α (η ih−1 )

where α

≠ f (i, L, RTT ) andα > 1

h Ti −1 = RTTi k + γ i where γ i = ρ * ( L − i ) * RTTi −k1 andρ > 1

Ti −h1 = ξ * RTTi k + Ti k where ξ ≠ f (i, L, RTT ) and ξ > 1 Ti −h1 = RTTi k + Ti k + σ

Topt =

where σ

≠ f (i, L, RTT ) and σ > 1

1 (L + μ j − μk ) + π (σ k − σ j ) 2 2 2 NRN Loc=Terminal,NRN=0.2% , α=2.3

100 Time(s)

80

2.5K

60 40

5K 10K

Scheme

Opt

MDT

PDT

PLD

PCD

0

MCD

20

Fig. 4. Impact of Size of Graph on CT NRN Loc=Terminal,NRN=0.2% , α=2.3l 1 2.5K

0.75

Scheme

Opt

0

PDPT

10K PDT

0.25 PLD

5K

PCD

0.5

MCD

Optimal Scheme

Formula

CE

Scheme Heuristic MCD (Fixed Margin Over Cumulative Delay (CRTT) Heuristic PCD (Proportionate Cumulative RTT) Heuristic PLD (Proportionate Level over RTT) Heuristic PDT (Proportionate RTT over Fixed Child Timer) Heuristic MDT (Fixed Margin RTT over Fixed Child Timer)

Fig. 5. Impact of Size of Graph on CE

31

32

A. Ul Haque and J.I. Khan

5 Conclusions Timer management is the natural technique to address the ‘unreliability’ posed by any MAE entity which inherently different from unreliability handled by TCP. We formalized the multi-way bounded wait principle for MAE to respond to this sort ‘unreliability’. We introduced the notion of the lower bound of completion time and the upper bound of capturing efficiency. In this paper we used completion time and capturing efficiency to compare optimal scheme with some heuristics proposed for the harness to show better performance of optimal scheme. We have shown that the optimal scheme outperforms other heuristics in various network conditions. Among the heuristics most promising is MCD. However, a major concern for MCD is that its performance degrades with size of the network whereas the optimal scheme scales well with the size of a network.

References 1. Meshkova, E., Riihij, J., Petrova, M., Petri, M.: A survey on resource discovery mechanisms, peer-to-peer and service discovery frameworks. The International Journal of Computer and Telecommunications Networking Archive 52(11) (August 2008) 2. Ahmed, R., Boutaba, R.: A Survey of Distributed Search Techniques in Large Scale Distributed Systems. Communications Surveys & Tutorials 13(2) (May 2011) 3. Khan, J.I., Haque, A.U.: Computing with data non-determinism: Wait time management for peer-to-peer systems. Journal Computer Communications 31(3) (February 2008) 4. Allen, M.S., Wolski, R., Plank, J.S.: Adaptive. Timeout Discovery using the Network Weather Service. In: Proceedings of HPDC 2011 (July 2002) 5. Zhan, J.C.W., He, Z.: Optimal Retransmission Timeout Selection For Delay-Constrained Multimedia Communications. In: International Conference on Image Processing, ICIP 2004, October 24-27, vol. 3, pp. 2035–2038 (2004), doi:10.1109/ICIP.2004.1421483 6. Libman, L., Orda, A.: Optimal retrial and timeout strategies for accessing network resources. IEEE/ACM Transactions on Networking 10(4), 551–564 (2002) 7. Grossglauser, M.: Optimal deterministic timeouts for reliable scalable multicast. In: IEEE Infocom 1996, pp. 1425–1441 (March 1996) 8. Bu, T., Duffield, N., Presti, F.L., Towsley, D.: Network tomography on general topologies. In: Proc. of ACM SIGMETRICS (2002) 9. Duffield, N.G., Lo Presti, F.: Multicast Inference of Packet Delay Variance at Interior Networks Links. In: Proc. Infocom 2000, Tel Aviv, Israel (March 26-30, 2000) 10. Adams, A., Bu, T., Caceres, R., Duffield, N., Friedman, T., Horowitz, J., Lo Presti, F., Moon, S.B., Paxson, V., Towsley, D.: The use of end-to-end multicast measurements for characterizing internal network behavior. IEEE Communications Magazine (May 2000) 11. Lo Presti, F., Duffield, N.G., Horowitz, J., Towsley, D.: Multicast-Based Inference of Network-Internal Delay Distribution, preprint, AT&T Labs and University of Massachusetts (1999) 12. Bu, T., Duffield, N.G., Lo Presti, F., Towsley, D.: Network tomography on general topologies. ACM SIGMETRICS (June 2002) 13. Coates, M.J., Nowak, R.: Network Delay Distribution Inference from End-to-end Unicast Measurement. In: Proc. of the IEEE International Conference on Acoustics, Speech, and Signal Processing (May 2001) 14. Duffield, N.G., Horowitz, J., Lo Presti, F., Towsley, D.: Network delay tomography from end-to-end unicast measurements. In: Palazzo, S. (ed.) IWDC 2001. LNCS, vol. 2170, pp. 576–595. Springer, Heidelberg (2001)

World-Wide Distributed Multiple Replications in Parallel for Quantitative Sequential Simulation Mofassir Haque1, Krzysztof Pawlikowski1, Don McNickle2, and Gregory Ewing 1 1

University of Canterbury, Department of Computer Science, Christchurch 8140, New Zealand 2 University of Canterbury, Department of Management, Christchurch 8140, New Zealand [email protected] {Krys.Pawlikowski,Don.McNickle,Greg.Ewing}@canterbury.ac.nz

Abstract. With the recent deployment of global experimental networking facilities, dozens of computer networks with large numbers of computers have become available for scientific studies. Multiple Replications in Parallel (MRIP) is a distributed scenario of sequential quantitative stochastic simulation which offers significant speedup of simulation if it is executed on multiple computers of a local area network. We report results of running MRIP simulations on PlanetLab, a global overlay network which can currently access more than a thousand computers in forty different countries round the globe. Our simulations were run using Akaroa2, a universal controller of quantitative discrete event simulation designed for automatic launching of MRIP-based experiments. Our experimental results provide strong evidence that global experimental networks, such as PlanetLab, can efficiently be used for quantitative simulation, without compromising speed and efficiency. Keywords: Multiple Replications in Parallel, Experimental networking facilities, Akaroa2, PlanetLab, Sequential quantitative stochastic simulation, Open queuing network.

1 Introduction Quantitative stochastic simulation of complex scenario can take hours or days to complete. SRIP (Single Replication in Parallel) and MRIP (Multiple Replication in Parallel) are two methods used to reduce simulation time. In SRIP, the simulation program is divided into smaller logical parts and run on different computers. In MRIP, multiple processors run their own replications of sequential simulation, but cooperate with central analyzers (one central analyzer for each performance measure analyzed) that are responsible for analyzing the results and stopping the simulations when the specified level of accuracy is met [1]. The MRIP technique can significantly speed up simulation if replications are launched on a larger homogeneous set of computers [2, 3]. In last few years, a large number of experimental networking facilities have been, or are being developed across the globe: e.g. PlanetLab, GENI, OneLab, G-Lab, Akari, Panlab, etc. [4]. These global networks often consist of thousands of computers. Y. Xiang et al. (Eds.): ICA3PP 2011 Workshops, Part II, LNCS 7017, pp. 33–42, 2011. © Springer-Verlag Berlin Heidelberg 2011

34

M. Haque et al.

Thus they provide a viable alternative for running distributed stochastic simulations in the Multiple Replications in Parallel scenario (MRIP). We selected PlanetLab as the provider of distributed computing resources for investigating various aspects of MRIP simulations, since it is a continuously evolving computing platform with thousands of nodes [5]. These nodes can be easily accessed for running MRIP without investing in infrastructure. However, before using such a globally distributed networking facility for sequential stochastic simulation on multiple computers, factors such as load at selected nodes and potential communication overhead between them have to be carefully considered, as these computers can be shared by a large number of users and some of them are thousands of miles apart. Load generated by these users can vary significantly and quickly. Thus, it can adversely affect performance of computers, and the simulations running on them.

Fig. 1. PlanetLab with deployed nodes around the world [5]

We did extensive experimentation to determine the suitability of PlanetLab nodes for MRIP simulations. Our simulations were run with Akaroa2, a universal controller of quantitative discrete event simulation, designed for automatic launching of MRIPbased experiments. Experiments were designed to measure times needed to produce final simulation results over various sets of PlanetLab computers. Results obtained from the experiments executed over PlanetLab nodes were compared with the results obtained from running MRIP simulations on a local area network at the University of Canterbury. This has allowed us to conclude that a global networking facility such as PlanetLab can be effectively utilized for running MRIP. The rest of the paper is organized as follows. Section 2 spells out the procedure for running Akaroa2 on PlanetLab. Sections 3 explains in detail the experimental set up and evaluation metric. Section 4, presents experimental results, and conclusions are in Section 5.

2 Akaroa2 on PlanetLab In Akaroa2, multiple independent replications of a stochastic simulation are run on different processors, which play the role of independent simulation engines producing

World-Wide Distributed Multiple Replications

35

statistically equivalent output data during one simulation. Multiple simulation engines cooperate with the global analyzer that processes streams of output data coming out from different simulation engines, and stops the simulation once the required accuracy of the results has been achieved. The accuracy is typically measured by the relative statistical error of the results. Two main processes of Akaroa2 are: Akmaster and Akslave. The Akslave process initiates simulation engines on multiple processors, while Akmaster controls sequential collection of output data and their analysis. It collects local estimates from all running Akslaves, calculates final global estimates, displays results, and then terminates the simulation when the stopping criterion is reached [6]. Both steady-state simulations and terminating simulations are supported. In the former case, the procedures for sequential mean and variance analysis are described in [1, 7-8], while the procedure adopted for terminated simulation is presented in [2]. Akaroa2 is widely used for simulations executed on local area networks, as its records of the last 10 years (in July 2011) show over 3100 downloads of the software by users from over 80 countries [9]. In order to run Akaroa2 on PlanetLab, first we need to copy and install Akaroa2 on all the nodes which will be used for running simulation engines. Copying and installing software on hundreds of machines is an intricate task. Either the CoDeploy program [10] provided by PlanetLab or, alternatively, simple shell scripts for automating copying, installation and running of Akaroa-2 on PlanetLab can be used. The shell script we used can be downloaded from the PlanetLab New Zealand web site [11]. For proper execution of MRIP-based simulation, the path variable should be correctly set in the bash profile file of all participating PlanetLab nodes, and simulation program should be copied in the directory specified in the path. The detailed procedure with step by step instructions for running Akaroa-2 on PlanetLab using Linux or Windows operating system can be downloaded from PlanetLab New Zealand web site [11].

3 Experimental Setup To study the feasibility of running Akaroa2 on PlanetLab, we conducted a large number of experiments, considering different strategies for selecting participating nodes of the network. The aim was to measure the times to produce simulation results, from the time instant when the simulation was launched until the time instant when the final results were obtained, to find out how using differently selected sets of PlanetLab nodes can affect users’ quality of experience, in comparison with simulations executed on local computers only. We compared two of many possible strategies for selection of PlanetLab nodes for MRIP simulations. We assumed that the computers are either distributed over a restricted geographical region (so they operate in the same or close time zones), or they are distributed globally (so computers participating in MRIP simulations work in very different time zones). 3.1 Computing Setup CS1 In this computing setting, while operating from New Zealand, we installed Akaroa2 on PlanetLab nodes spread over the European Union. The Akmaster was installed in

36

M. Haque et al.

Italy and simulation engines were located in France, UK, Belgium, Italy, Hungary and Poland. PlanetLab nodes were carefully selected using the CoMon utility [12] to avoid currently heavily loaded nodes. The CoMon utility is provided by PlanetLab for monitoring of resource utilization of all PlanetLab nodes. In CS1 our aim was to assess response times of MRIP-based simulation experiments. The experiments were run on Friday, beginning at 2pm British Standard Time. 3.2 Computing Setup CS2 In this computing environment, simulation engines of Akaroa2 were installed worldwide, so they operated in very different time zones. Again, while operating from New Zealand, we installed the Akmaster in Italy, and the simulation engines were launched in Europe, USA, Canada, New Zealand and Asia; see Figure 2. Nodes were again carefully selected using the CoMon utility, avoiding nodes which were heavy loaded. This setup was used to study and verify effect of communication overhead when simulation engines are thousands of miles apart. The experiments were run on Friday, beginning at 2pm USA Central Standard Time.

Fig. 2. Global distribution of Akaroa2 in CS2

Note that the nodes of PlanetLab used by Akaroa2 represented a homogenous set of computers, as the computers of PlanetLab have to satisfy some minimum common technical requirements. For comparing the quality of users’ experience in such distributed simulation environments, we have also measured the waiting times for simulation results in the traditional local processing environment of Akaroa2, where its simulation engines are located around a local area network. 3.3 Computing Setup CS3 Here, the simulation experiments were run on computers linked by a local area network in a computer laboratory of the Department of Computer Science and Software Engineering, at the University of Canterbury, in Christchurch. Akmaster and Akslave were installed in this controlled local area network environment, the original home location of Akaroa2. The results were used as the reference for comparison with the results obtained from the two other, distributed computing environments.

World-Wide Distributed Multiple Replications

37

The experiments were run on Friday, beginning from 2pm, New Zealand time. The nodes of the local area network, physically located in one laboratory, constitute a homogenous set of computers. Laboratory and PlanetLab nodes are equipped with quad processors and both use the Fedora operating system based on the Linux Kernel. However, the computers available on PlanetLab are of slightly higher technical standards in terms of memory and clock frequency than those available in our CS3 setting. 3.4 Simulation Setting and Evaluation We ran the same sequential stochastic simulation in MRIP scenario in all three computing setups: CS1, CS2 and CS3. For our study, we simulated a simple open queuing network, consisting of a CPU and two disk memories with unlimited buffer capacities, depicted in Figure 3. We estimated steady-state mean response (mean time spent by a customer in this system), assuming that arriving customers form a Poisson process with λ= 0.033 tasks per second. All service times are exponentially distributed with mean service time at the CPU of 6 seconds, mean service time at Disk 1 and mean service time at Disk 2 both of 14 seconds. This makes the servers to the CPU, Disk 1 and Disk 2 loaded at 96%, 92.4% and 92.4%, respectively. Disk 1

Job Sink

Job Source

p1 p3

CPU p2

Disk 2

Fig. 3. Simulated open queuing network

The simulation processes on all computers were to stop when the estimate of steady-state mean response reached a relative statistical error not greater than 5%, at a confidence level of 0.95. This should require about 20 million observations. Running simulation in Multiple Replications in Parallel scenario allowed us to collect this sample of output data faster, as it is produced by multiple simulation engines. To demonstrate that this attractive feature of MRIP remains practical also in case of globally distributed simulation engines, we assessed speedup and relative efficiency of MRIP simulations in setup CS1 and CS2, and compared the results with those from locally distributed simulation engines in CS3. The performance of our MRIP simulations was assessed by measuring response time (RT) of a given simulation setting, defined as the time interval between the time of launching the simulation until

38

M. Haque et al.

the time when the final results are delivered to the user. Then, the speedup of simulation at P > 1 simulation engines can be found as: S P =

Mean_RT(1) Mean_RT(P)

(1)

where Mean_RT (P) is mean response time of P simulation engines running MRIP simulation with P ≥ 1. Alternatively, we looked at the relative speedup of MRIP simulation, defined as SR (P)=

Mean_RT(1)-Mean_RT(P) Mean _RT(1)

*100 %

(2)

for P= 1, 2, 3, …. Note that, due to the truncated Amdahl law for MRIP formulated in [2, 3], there exists a limit on the number of processors which would increase the speedup of MRIP simulation. It is also known that the largest speedup can be obtained in homogeneous computing environments. In the extreme case, if one simulation engine uses a very fast processor and remaining processors are slow, a simulation will not benefit at all from MRIP at all, as the fastest simulation engine can produce the entire sample of required observations needed for stopping the simulation, before any of the remaining slower simulation engines is able to reach its first checkpoint. Another performance measure which we considered is the efficiency of distributed processing during MRIP simulation, or speedup per simulation engine: E (P) =

S(P)

P

(3)

In an ideal situation, the efficiency would be equal to one. However, in practical applications of parallel processing it is usually much smaller. E (P) measures how well the contributing processors are utilized for solving a given problem, despite their mutual communication and synchronization activities.

4 Experimental Results In this section, we present our experimental results obtained under computing setups CS1, CS2 and CS3. We use mean response time as the measure of quality for testing our hypothesis that the MRIP scenario can also be efficiently used in the case of world-wide distributed simulation engines. The mean response times obtained for CS1, CS2 and CS3, measured in seconds, are given in Table 1. Each reported result is an average over 10 independent measurements/simulations. The relative statistical errors of these estimates are not larger than 1% for CS3 and not larger than 6 % for CS1 and CS2, at 0.95 confidence level. Fig. 4 compares mean response times of CS1, CS2 and CS3. The histogram clearly shows that mean response time reduces as the number of nodes increases. The PlanetLab nodes are being shared by a large number of users and are located hundreds of miles apart. Conversely, laboratory nodes are used by only one person and are located close to each other. The mean response times in case of CS3 are therefore smaller than in the case of PlanetLab nodes both in CS1 and CS2. In order to obtain good performance, PlanetLab nodes should be carefully selected, avoiding heavily loaded nodes and busy working hours.

World-Wide Distributed Multiple Replications

39

Table 1. Mean response time for scenario CS1, CS2 and CS3 Number of Nodes

CS1

CS2

CS3

2 4 6 8 10 12 15

88.13 61.94 52.25 45.23 39.8 34.32 27.14

97.53 75.48 64.74 59.34 46.81 43.21 36.35

59.78 47.53 37.08 32.81 29.98 28.62 15.67

CS1

CS2

CS3

Time in Seconds

100 80 60 40 20 0 2

4

6

8

10

12

15

Number of Nodes

Fig. 4. Comparison of mean response times in CS1, CS2 and CS3

Comparison of the mean response times for CS1 and CS2 shows that these mean response times are much shorter if all the PlanetLab nodes are selected from one area (continent), for example within Europe rather than from all over the world. This is primarily because of communication overhead. When controller and simulation engines are located thousands of mile apart, the time used for exchanging data between simulation engines and controller directly effects the mean response time. We also ran the same experiment by selecting PlanetLab nodes from North America only and found results similar to the setup CS2. Speedup for distributed scenario of CS1 and CS2 is calculated using Equation (1) and given in Table 2. Speedup has been calculated using mean response time of two nodes as a reference. In spite of the longer distance between nodes, speedup offered by PlanetLab nodes in the case of CS1 is better than in CS3, because of the slightly better hardware of PlanetLab nodes.

40

M. Haque et al. Table 2. Speedup for distributed scenario of CS1 and CS3 Number of Nodes

CS1

CS3

2 4 6 8 10 12 15

1 1.42 1.69 1.95 2.21 2.57 3.24

1 1.26 1.61 1.82 1.99 2.09 3.17

Efficiency in the case of CS1 and CS3 has been calculated using Equation (3) and is shown in table 3. In this case, there is only small difference between the results. The efficiency decreases as the number of processors increases. This is due to the fact that processor communication is usually slower than computation and exchange of local estimates between Akslaves and Akmaster results in frequent communication. Table 3. Efficiency for scenario CS1 and CS3 Number of Nodes

CS1

CS3

2 4 6 8 10 12 15

0.5 0.31 0.26 0.22 0.19 0.17 0.20

0.5 0.35 0.28 0.24 0.22 0.21 0.21

These results allow us to conclude that it has become practical to use distributed computing resources of global experimental networks for fast quantitative stochastic simulation, paying only a small penalty in the form of a minor worsening of response times, speedup and efficiency of the simulation as comparing with the same simulations run on a local area network. The advantage of using globally distributed computing resources is that they can be substantially larger than the ones available locally. We conducted experiments using two different ways of selection of computers in PlanetLab for simulation engines and compared their performance with performance of simulation run on computers of a local area network. The performance of MRIP in CS1 appears to be better that in CS2. Thus, for best results selection of computers from closer geographical location, avoidance of both heavily loaded nodes and busy hours is recommended.

World-Wide Distributed Multiple Replications

41

5 Conclusions In this paper we have shown that the distributed computing resources of global experimental networks, such as PlanetLab, can be effectively used for running quantitative stochastic simulations in MRIP scenario. Only a small penalty (in the form of a minor worsening of performance) is paid for using globally distributed resources instead of local ones. Launching and on-line control of globally distributed simulations can be done by using for example Akaroa2. It is encouraging news for those who need to run time-consuming quantitative simulations to get accurate final results, but do not have access to sufficiently large number of computers for launching multiple simulation engines. Recently, there has been a surge in development of global and regional experimental networking facilities, see Table 4 [13]. Most of these networks offer free membership and can be effectively used for conducting simulation experiments under control of Akaroa2. Table 4. Selected experimental networking facilities, with size and accessibility Name

Purpose

Size

Access

OneLab Panlab

Multipurpose Multipurpose

Regional Regional

Federica PlanetLab GENI JNB 2 CNGI

Multipurpose Multipurpose Multipurpose Multipurpose Multipurpose

Regional Global Regional Regional Regional

Free membership Planned to be on Payment Free membership Free membership Free membership Free membership Free membership

In future, we plan to investigate the upper bounds for speedup of globally distributed sequential stochastic simulation, such as those in the MRIP scenario. This will require running experiments at full scale, employing hundreds of PlanetLab nodes as simulation engines, with simulations requiring extremely large samples of output data for producing accurate simulation results, in particular if the simulated processes are strongly correlated. Acknowledgments. This work was partially supported by REANNZ (2010/2011 KAREN Capability Build Fund).

References 1. Pawlikowski, K., Yau, V., McNickle, D.: Distributed stochastic discrete-event simulation in parallel time streams. In: 26th Conference on Winter Simulation, pp. 723–730. Society for Computer Simulation International, Orlando (1994) 2. Pawlikowski, K., Ewing, G., McNickle, D.: Performance Evaluation of In-dustrial Processes in Computer Network Environments. In: European Conference on Concurrent Engineering, pp. 129–135. Int. Society for Computer Simulation, Erlangen (1998)

42

M. Haque et al.

3. Pawlikowski, K., McNickle, D.: Speeding Up Stochastic Discrete-Event Simulation. In: European Simulation Symposium, pp. 132–138. ISCS Press, Marseille (2001) 4. Lemke, M.: The Role of Experimentation in Future Internet Research: FIRE and Beyond. In: 6th International Conference on Testbeds and Research Infrastructures for the Development of Networks and Communities, Berlin, Germany (2010) 5. PlanetLab, http://www.planet-lab.org/ 6. Ewing, G., Pawlikowski, K., McNickle, D.: Akaroa-2: Exploiting Network Computing by Distributing Stochastic Simulation. In: 13th European Simulation Multi-Conference, Warsaw, Poland, pp. 175–181 (1999) 7. Ewing, G., Pawlikowski, K.: Spectral Analysis for Confidence Interval Estimation Under Multiple Replications in Parallel. In: 14th European Simulation Symposium, pp. 52–61. ISCS Press, Dresden (2002) 8. Shaw, N., McNickle, D., Pawlikowski, K.: Fast Automated Estimation of Varience in Sequential Discrete Event Stochistic Simulation. In: 25th European Conference on Modelling and Simulation, Krakow, Poland (2011) 9. Akaroa2, http://www.cosc.canterbury.ac.nz/research/RG/net_sim/simulat ion_group/akaroa/about.chtml 10. CoDeploy, http://codeen.cs.princeton.edu/codeploy/ 11. PlanetLab NZ, http://www.planetlabnz.canterbury.ac.nz/ 12. CoMon, http://comon.cs.princeton.edu 13. Haque, M., Pawlikowski, K., Ray, S.: Challenges to Development of Multipurpose Global Federated Testbed for Future Internet Experimentation. In: 9th ACS/IEEE International Conference on Computer Systems and Applications, Sharm El-Sheikh, Egypt (2011)

Comparison of Three Parallel Point-Multiplication Algorithms on Conic Curves Yongnan Li1,2, Limin Xiao1,2, Guangjun Qin1,2, Xiuqiao Li1,2, and Songsong Lei1,2 1

State Key Laboratory of Software Development Environment, Beihang University, Beijing, 100191, China 2 School of Computer Science and Engineering, Beihang University, Beijing, 100191, China {liyongnan.buaa,guangjunster,xiuqiaoli,lss.linux}@gmail.com [email protected]

Abstract. This paper makes a comparison of three parallel point-multiplication algorithms on conic curves over ring Zn. We propose one algorithm for paralleling point-multiplication by utilizing Chinese Remainder Theorem to divide point-multiplication over ring Zn into two different point-multiplications over finite field and to compute them respectively. Time complexity and speedup ratio of this parallel algorithm are computed on the basis of our previous research about the basic parallel algorithms in conic curves cryptosystem. A quantitative performance analysis is made to compare this algorithm with two other algorithms we designed before. The performance comparison demonstrates that the algorithm presented in this paper can reduce time complexity of pointmultiplication on conic curves over ring Zn and it is more efficient than the preceding ones. Keywords: conic curves, ring Zn, finite field Fp, point-addition, point-double, point-multiplication, Chinese Remainder Theorem.

1

Introduction

In recent years, three main classes of public key cryptosystem are considered both secure and efficient: integer factorization system, discrete logarithm system and discrete logarithm system based on mathematical curves. Conic curves cryptosystem belongs to the third one. Professor Cao presented the concept of conic curves cryptography firstly in [1-2]. Then a public-key cryptosystem scheme on conic curves over ring Zn was proposed in [3-5]. Research in [6] introduced the definitions of extended point-addition and point-double on conic curves over ring Zn. In this paper, an efficient technique for parallel computation of the pointmultiplication on conic curves over ring Zn is proposed and our algorithm can reduce time complexity of point-multiplication. The analysis of this parallel methodology is based on our previous work about the basic parallel algorithms used in conic curves cryptosystem. Study in [7] proposed several parallel algorithms for cryptosystem on conic curves over finite field Fp. In [8], original point-addition and point-double were paralleled for cryptosystem on conic curves over ring Zn. Work in [9] introduced Y. Xiang et al. (Eds.): ICA3PP 2011 Workshops, Part II, LNCS 7017, pp. 43–53, 2011. © Springer-Verlag Berlin Heidelberg 2011

44

Y. Li et al.

traditional parallel point-multiplication in conic curves cryptosystem over ring Zn and finite field Fp. Parallel extended basic operations of point-addition and point-double were proposed in [10]. Study in [11] designed two high performance algorithms of point-multiplication for conic curves cryptosystem based on standard NAF algorithm and Chinese Remainder Theorem. The methodology presented in this paper partitions point-multiplication over ring Zn into two point-multiplications over finite field Fp and finite field Fq by calling Chinese Remainder Theorem, which is proposed by famous military strategist “Sun Tzu”. Two point-multiplications are executed respectively and then the temporary results are merged to get the final value. This method is similar with the one we proposed in [11] and the difference is that the preceding research adopted standard NAF algorithm to compute the point-multiplication over finite filed Fp. Time complexity and speedup ratio of this parallel algorithm are computed on the basis of our previous research about the time complexities of the fundamental algorithms on conic curves. And we will evaluate the performance of this parallel algorithm and compare with two old parallel algorithms. The parallel algorithm proposed in this paper not only accelerates the speed of point-multiplication, but also shows higher efficiency than two old parallel algorithms we designed before. The rest of this paper is organized as follows. Next section introduces the definition of point-multiplication on conic curves over ring Zn. Section 3 depicts time complexities of the basic operations in conic curves cryptosystem. Section 4 presents the methodology of paralleling point-multiplication on conic curves over ring Zn. The performance comparison of our techniques is proposed in section 5. The last section concludes the whole paper and points out some future works briefly.

2

Definition of Point-Multiplication

The definitions of point-addition and point-double must be introduced firstly. In conic curves cryptosystem over ring Zn, Cn ( a, b ) means the conic curves. C1, C2 and C3 represent three different fields over ring Zn. For any point P ( x1 , y1 ) ∈ Cn ( a, b) and

Q ( x2 , y2 ) ∈ Cn ( a, b ) , the operator ⊕ is defined as: •

If P ≠ Q , then operation of point-addition is P ⊕ Q .

•

If P = Q , then operation of point-double is 2P .

Operators ⊕ are defined differently in the expressions of point-addition and pointdouble. Point-multiplication signifies summation of many points on conic curves. Parameter k and parameter P represent coefficient and point on conic curves respectively. Point-multiplication kP is defined as: count = k 644 7448 kP = P ⊕ P ⊕ L ⊕ P ,

(1)

In conic curves cryptosystem over ring Zn, we define n=pq (p and q are two different odd prime integers). For more details, please refer to researches in [1-5].

Comparison of Three Parallel Point-Multiplication Algorithms on Conic Curves

3

45

Time Complexities of Basic Operations

As depicted in [7-10], time complexities of several basic operations are listed in Table 1 and Table 2. And we set the runtime of single-precision multiplication as the basic measurement unit. The time complexities of multiple-precision addition and deduction are O(1). Table 1. Time complexities of parallel operations Operation Original point-addition over ring Zn

Time complexity computation communication 644444 47444444 8 64444444 4744444444 8 2 2 2 2 2N p 2N p 3N 3N O( + + 9lN + 3lA + 21) + O( + + 6 N + 6lN + 4 N p + 2lA + 5) sn sp sn sp

Original point-double over ring Zn

communication 6444444computation 4744444448 644444444 7444444448 2 2 3N 2 2 N p 3N 2 2 N p O( + + 9lN + 3lA + 2a + 20.5) + O( + + 6 N + 6lN + 4 N p + 2lA + 5) sn sp sn sp

Extended point-addition over ring Zn

computation communication 64444 4744444 8 64444444 4744444444 8 2 2 2N 2 6N p 2N 2 6N p O( + + 6lN p + 28) + O ( + + 4 N + 12 N p + 4lN p + 10) sn sp sn sp

Extended point-double over ring Zn

computation communication 64444 4744444 8 64444444744444448 2 2 2N 2 6N p 2N 2 6N p O( + + 6lN p + 28) + O( + + 4 N + 12 N p + 4lN p + 4) sn sp sn sp

Pointmultiplication over ring Zn Pointmultiplication over finite field Fp

computation communication 64444444 4744444444 8 6444444444 7444444444 2 2 3N 2 2 N p 3N 2 2 N p O(t ( + + 9lN + 3lA + 24.5) − 3.5) + O(t ( + + 6 N + 6lN + 4 N p + 2lA + 5) + sn sp sn sp

computation communication 6444444 74444448 6444444 4744444448 3n 2 3n2 O(t ( + 10 + 3 ⎢⎡lg X ⎥⎤ + 3 ⎢⎡lg P ⎥⎤ )) + O(t ( + 6n + 2 ⎢⎡lg X ⎥⎤ + 2 ⎢⎡lg P ⎥⎤ ) + 1) s s

Multiplication

computation communication 6 4748 64 748 n2 n2 O( + 2) + O( + 2n) s s

Barrett reduction

computation communication 64 748 64 4744 8 2n 2 2n 2 O( + 8) + O( + 4 n) s s

The meanings of the variables in Table 1 and Table 2 are: • • • •

N: multiple-precision of operand n over ring Zn. Np: multiple-precision of operand p over ring Zn. Nq: multiple-precision of operand q over ring Zn. a: a coefficient of conic curves equation.

46

Y. Li et al.

• • • • • • • • • • •

b: a coefficient of conic curves equation. A: a fixed coefficient over ring Zn. s: process number for computing multiplication. l: word length of the computer. X: numerator of inversion-multiplication. P: denominator of inversion-multiplication. n: multiple-precision of operand over finite field. t: the numbers of bit in coefficient k for computing point-multiplication kP. Sn: the value of s over ring Zn. Sp: the value of s over finite field Fp. Sq: the value of s over finite field Fq. Table 2. Time complexities of sequential operations

Operation

Time complexity

Original point-addition over ring Zn

O(2 N 2 + N p 2 + N q 2 + 10 N + 12lN + 4lA + 1.5b + 19.5)

Original point-double over ring Zn

O(2 N 2 + N p 2 + N q 2 + 10 N + 12 N + 4lA + 1.5b + 2a + 17.5)

Extended point-addition over ring Zn

O( N 2 + 5 N p 2 + 5 N q 2 + 23 N + 8lN + 37)

Extended point-double over ring Zn

O( N 2 + 4 N p 2 + 4 N q 2 + 19 N + 8lN + 27)

Point-multiplication over ring Zn

O((3 ( t − 1) 2)(2 N 2 + N p 2 + N q 2 + 10 N + 12lN +4lA + 1.5b + 17.5) + (2a + 1)(t − 1))

Point-multiplication over finite field Fp

O((3(t − 1) 2)(2n2 + 9n + 5 + 4 ⎡⎢ lg X ⎤⎥ + 4 ⎡⎢lg P ⎤⎥ ))

Multiplication

O ( n 2 + 2n )

Barrett reduction

O(n 2 + 4n + 5)

The relationship of variable N, Np and Nq in Table 1 and Table 2 is: N = N p + N q , N p ≥ N q . The value of coefficient A is

C1 N + C2 N q + C3 N p C1 + C2 + C3

. C1 , C2 and

C3 stand for the numbers of point in the fields of C1, C2 and C3. In this paper, the

symbol “Fq” has the same meaning of finite field Fp and the distinction is that its module is prime integer “q”.

Comparison of Three Parallel Point-Multiplication Algorithms on Conic Curves

4

47

Parallel Point-Multiplication

This section explains the methodology of paralleling point-multiplication for cryptosystem on conic curves over ring Zn. It uses Chinese Remainder Theorem to partition point-multiplication over ring Zn into two point-multiplications over finite field. As Fig.1 shows, there are three steps in the parallel procedure of point-multiplication. Firstly, two reductions are calculated to divide parameter t into tp and tq. Then two point-multiplications are computed over finite field Fp and finite field Fq respectively. Lastly, kP(tp) and kP(tq) are incorporated by calling Chinese Remainder Theorem to get the value of kP(t ) .

Fig. 1. Parallel procedure using Chinese Remainder Theorem

In the first step, two reductions are calculated to map parameter t over ring Zn into tp over finite field Fp and tq over finite field Fq. Then, Tp − left

Tp − right

computation communication 644 7448 644 47444 8 2 = O(2 N p s p + 8) + O(2 N p 2 s p + 4 N p ) ,

(2)

computation communication 644 7448 644 47444 8 = O(2 N q 2 s q + 8) + O (2 N q 2 sq + 4 N q ) .

(3)

We could conclude that the value of Tp − left is bigger than Tp − right because of N p ≥ N q . This procedure costs two communication units (variable t and qq-1), so the parallel runtime of the first step is Tp − first

computation communication 644 7448 6444 474444 8 2 = O(2 N p s p + 8) + O (2 N p 2 s p + 4 N p + 2) .

(4)

48

Y. Li et al.

In the second step, two operations of point-multiplication over finite field Fp and finite field Fq are executed simultaneously. Then the values of the two pointmultiplications and the inversions of the two modules should be multiplied to get the final value of point-multiplication over ring Zn. Parameters pp −1 and qq −1 are two constants in the cryptosystem because p and q are fixed. The multiple-precision of kP ( t p ) pp −1 is 2Np and the multiple-precision of kP ( tq ) qq −1 is 2Nq. Consequently, it costs one point-multiplication and one multiple-precision multiplication over each finite field. We could get the parallel runtime of the second step:

Tp − left

computation 64444444 744444448 2 == O (t (3N p s p + 6lN p + 10) + N p 2 s p + 2)

communication 644444444 47444444444 8 2 + O (t (3N p s p + 6 N p + 4lN p ) + N p 2 s p + 2 N p + 1)

Tp − right

6444444computation 4744444448 = O(t (3 N q 2 sq + 6lN q + 10) + N q 2 sq + 2)

communication 644444444 47444444444 8 2 + O (t (3N q s q + 6 N q + 4lN q ) + N q 2 sq + 2 N q + 1)

,

.

(5)

(6)

Then, Tp − sec ond

computation 64444444 744444448 2 = O(t (3N p s p + 6lN p + 10) + N p 2 s p + 2)

communication 644444444 47444444444 8 2 + O (t (3N p s p + 6 N p + 4lN p ) + N p 2 s p + 2 N p + 1)

.

(7)

In the third step, the values of parameters kP ( t p ) pp −1 and kP ( tq ) qq −1 are merged to get the point-multiplication over ring Zn by computing (8):

(

)

k ( t p ) = kP ( t p ) pp −1 + kP ( tq ) qq −1 mod n .

(8)

Sum of kP ( t p ) pp −1 and kP ( tq ) qq −1 is a 2Np multiple-precision integer. One multiple-precision reduction will be needed because the final result is a 2N multipleprecision integer and 2 N p ≥ N . Therefore, the third step costs one multiple-precision addition and one multiple-precision reduction. The parallel runtime of the third step is

Tp − third

communication 64computation 4744 8 644 474448 2 = O(2 N sn + 8) + O (2 N 2 sn + 4 N + 1) .

(9)

Comparison of Three Parallel Point-Multiplication Algorithms on Conic Curves

49

Consequently, the parallel runtime of point-multiplication is computation 6444444444 474444444444 8 2 Tp1 = O(t (3N p s p + 10 + 6lN p ) + 2 N 2 sn + 3N p 2 s p + 18)

+

communication 6444444 474444444 8 O(t (3 N p 2 s p + 6 N p + 4lN p ) + 2 N 2 sn

.

(10)

+ 3N p 2 s p + 6 N p + 4 N + 4)

The sequential runtime of point-multiplication could be looked up in Table 2: Ts = O((3 ( t − 1) 2)(2 N 2 + N p 2 + N q 2 + 10 N + 12lN + 4lA + 1.5b + 17.5) + (2a + 1)(t − 1))

.

(11)

Then we can get the speedup ratio: O ((3 ( t − 1) 2)(2 N 2 + N p 2 + N q 2 + 10 N + 12lN S=

+ 4lA + 1.5b + 17.5) + (2a + 1)(t − 1)) . computation 6444444444 474444444444 8 O(t (3 N p 2 s p + 10 + 6lN p ) + 2 N 2 sn + 3N p 2 s p + 18) +

5

(12)

communication 6444444 474444444 8 2 O(t (3N p s p + 6 N p + 4lN p ) + 2 N 2 sn

+ 3N p 2 s p + 6 N p + 4 N + 4)

Performance Comparison

This section evaluates the performance of parallel point-multiplication proposed in this paper. And the quantitative performance comparison is made between this parallel point-multiplication and two other parallel point-multiplications we presented before. The parameters over ring Zn are assumed as: N = 2sn = 2 N p = 2 N q = 4s p = 4sq , a = 2 , b = 1 , l = 32 . As demonstrated in [8], A=

C1 N + C2 N q + C3 N p C1 + C2 + C3

.

(13)

If variable n over ring Zn is big enough, the value of C1 is much bigger than C2 and C3 . Then coefficient A will be approximately equal to N. Therefore, Tp1 in (10) is simplified as:

50

Y. Li et al.

computation communication 64444 744448 644 47444 8 Tp1 = O(t (99 N + 10) + 7 N + 18) + O(70tN + 14 N + 4) .

(14)

The time complexity of the parallel point-multiplication proposed in [11] is (15) and it could be simplified as (16). Tp 2

computation 6444444444447444444444448 2 = O ( t + 1 2 ) ( 3N p s p + 6lN p + 10 ) + 2 N 2 sn + 3N p 2 s p + 18

(

)

64444444744444448 ⎛ ( t + 1 2 ) ( 3N p 2 s p + 6 N p + 4lN p ) ⎞ ⎟ +O⎜ ⎜ + 2 N 2 s + 3N 2 s + 4 N + 6 N + 4 ⎟ n p p p ⎝ ⎠ communication

Tp 2

.

(15)

computation communication 644444 47444444 8 64444 4744444 8 = O ( ( t + 0.5 )( 99 N + 10 ) + 7 N + 18) + O ( 70 N ( t + 0.5 ) + 14 N + 4 ) .

(16)

As demonstrated in Table 1 and Table 2, the runtime of traditional parallel pointmultiplication and sequential point-multiplication could be simplified as: computation communication 6444 474444 8 644 47444 8 Tp 3 = O(t (384 N + 24.5) − 3.5) + O (t (272 N + 5) + 1) ,

(17)

Ts = O((3 ( t − 1) 2)(2.5 N 2 + 522 N + 19) + 5(t − 1)) .

(18)

On condition that the communication time unit is 20 percent of computation time unit, the performance evaluation and comparison could be showed in Table 3, Fig. 2, Fig. 3 and Fig. 4. It could be seen that the methodology of paralleling point-multiplication accelerates the speed of point-multiplication and it is more efficient than two other methods. We also make other assumption of relationship between communication time unit and computation time unit. Same conclusion is derived by analyzing the performance comparison on different conditions. Table 3. Performance evaluation N

t

Tp1

Tp2

Tp3

Ts

8

10

9237.2

9694.2

35323.7

58837.5

8

20

18377.2

18834.2

70650.7

124212.5

8

30

27517.2

27974.2

105977.7

189587.5

8

40

36657.2

37114.2

141304.7

254962.5

8

50

45797.2

46254.2

176631.7

320337.5

8

60

54937.2

55394.2

211958.7

385712.5

8

70

64077.2

64534.2

247285.7

451087.5

16

10

18355.6

19264.6

70395.7

121693.5

Comparison of Three Parallel Point-Multiplication Algorithms on Conic Curves Table 3. (Continued) 16

20

36535.6

37444.6

140794.7

256908.5

16

30

54715.6

55624.6

211193.7

392123.5

16

40

72895.6

73804.6

281592.7

527338.5

16

50

91075.6

91984.6

351991.7

662553.5

16

60

109255.6

110164.6

422390.7

797768.5

16

70

127435.6

128344.6

492789.7

932983.5

24

10

27474

28835

105467.7

188869.5

24

20

54694

56055

210938.7

398724.5

24

30

81914

83275

316409.7

608579.5

24

40

109134

110495

421880.7

818434.5

24

50

136354

137715

527351.7

1028289.5

24

60

163574

164935

632822.7

1238144.5

24

70

190794

192155

738293.7

1447999.5

Fig. 2. Performance comparison while N=8

Fig. 3. Performance comparison while N=16

51

52

Y. Li et al.

1800000 1500000 1200000

Tp1 Tp2 Tp3 Ts

900000 600000 300000 0

10

20

30

40

50

60

70

Fig. 4. Performance comparison while N=24

6

Conclusions

In this paper, we presented a methodology of paralleling point-multiplication on conic curves over ring Zn. The method is proposed based on Chinese Remainder Theorem. The performance comparison between the parallel methodology and two other ones demonstrates that the technique introduced in this paper is the most efficient one. Our researches, including study in this paper, are focused on the basic parallel algorithms in conic curves cryptosystem. We plan to design the parallel algorithm of Elgamal cryptograpy on conic curves over ring Zn based on these parallel algorithms in the future. Acknowledgments. This study is sponsored by the fund of the State Key Laboratory of Software Development Environment under Grant No. SKLSDE-2009ZX-01, the Fundamental Research Funds for the Central Universities under Grant No. YWF-1002-058 and the National Natural Science Foundation of China under Grant No. 60973007.

References 1. Cao, Z.: A public key cryptosystem based on a conic over finite fields Fp. In: Advances in Cryptology: Chinacrypt 1998, pp. 45–49. Science Press, Beijing (1998) (in Chinese) 2. Cao, Z.: Conic analog of RSA cryptosystem and some improved RSA cryptosystems. Natural Science Journal of Heilongjiang University 16(4), 5–18 (1999) 3. Chen, Z., Song, X.: A public-key cryptosystem scheme on conic curves over the ring Zn. In: 6th International Conference on Machine Learning and Cybernetics, vol. 4, pp. 2183– 2187. IEEE Press, Hong Kong (2007) 4. Sun, Q., Zhu, W., Wang, B.: The conic curves over Zn and public key cryptosystem protocol. J. Sichuan Univ. (Nat. Sci. Ed.) 42(3), 471–478 (2005) (in Chinese) 5. Wang, B., Zhu, W., Sun, Q.: Public key cryptosystem based on the conic curves over Zn. J. Sichuan Univ. (Engin. Sci. Ed.) 37(5), 112–117 (2005) (in Chinese) 6. Li, Y.: Research of Conic Curve Cryptosystems and the Construction of CC-CSP. Thesis for the degree of master in computer application technology, Northestern University, pp. 25–27 (2008) (in Chinese)

Comparison of Three Parallel Point-Multiplication Algorithms on Conic Curves

53

7. Li, Y., Xiao, L., Hu, Y., Liang, A., Tian, L.: Parallel algorithms for cryptosystem on conic curves over finite field Fp. In: 9th International Conference on Grid and Cloud Computing, pp. 163–167. IEEE Press, Nanjing (2010) 8. Li, Y., Xiao, L., Liang, A., Wang, Z.: Parallel point-addition and point-double for cryptosystem on conic curves over ring Zn. In: 11th International Conference on Parallel and Distributed Computing, Applications and Technologies, pp. 317–322. IEEE Press, Wuhan (2010) 9. Li, Y., Xiao, L.: Parallel point-multiplication for conic curves cryptosystem. In: 3rd International Symposium on Parallel Architectures, Algorithms and Programming, pp. 116– 120. IEEE Press, Dalian (2010) 10. Li, Y., Xiao, L., Chen, S., Tian, H., Ruan, L., Yu, B.: Parallel Extended Basic Operations for Conic Curves Cryptography over Ring Zn. In: 9th IEEE International Symposium on Parallel and Distributed Processing with Applications Workshops, pp. 203–209. IEEE Press, Busan (2011) 11. Li, Y., Xiao, L., Wang, Z., Tian, H.: High Performance Point-Multiplication for Conic Curves Cryptosystem based on Standard NAF Algorithm and Chinese Remainder Theorem. In: 2011 International Conference on Information Science and Applications. IEEE Press, Jeju (2011)

Extending Synchronization Constructs in OpenMP to Exploit Pipeline Parallelism on Heterogeneous Multi-core Shigang Li, Shucai Yao, Haohu He, Lili Sun, Yi Chen, and Yunfeng Peng University of Science and Technology Beijing, 100083 Beijing, China [email protected]

Abstract. The ability of expressing multiple-levels of parallelism is one of the significant features in OpenMP parallel programming model. However, pipeline parallelism is not well supported in OpenMP. This paper proposes extensions to OpenMP directives, aiming at expressing pipeline parallelism effectively. The extended directives are divided into two groups. One can define the precedence at thread level while the other can define the precedence at iteration level. Through these directives, programmers can establish pipeline model more easily and exploit more parallelism to improve performance. To support these directives, a set of runtime interfaces for synchronization are implemented on the Cell heterogeneous multi-core architecture using signal block communications mechanism. Experimental results indicate that good performance can be obtained from the pipeline scheme proposed in this paper compared to the naive parallel applications. Keywords: Pipeline Parallelism, OpenMP, Cell architecture.

1

Introduction

Multi-core architectures are becoming the industry standard in the modern computer industry. There are two main categories: homogeneous multi-core and heterogeneous multi-core. The former one includes only identical cores while the later one integrates a control core and several accelerator cores. IBM/Toshiba/Sony Cell processor [10] is a typical heterogeneous multi-core. It comprises a conventional Power Processor Element (PPE) and eight synergistic processing elements (SPEs). SPEs don’t have hardware caches but each possesses a 256 KB local store. Communications between PPE and SPEs can be implemented through DMA, signal or mailbox. For different memory architectures, there are different programming models, such as message passing for distributed memory and shared memory inter-core communication methods. A well-known programming model for shared-memory parallel programming is OpenMP [1]. In the current definition of the OpenMP, multiple-levels of parallelism [9, 15, 16, 17] can be expressed. However, pipeline parallelism is not well supported in OpenMP. Due to the requirement from both programmers and applications, it’s necessary to extend OpenMP directives to express pipelined executions. In this paper we extend the OpenMP programming model with two groups of synchronization directives (one is Y. Xiang et al. (Eds.): ICA3PP 2011 Workshops, Part II, LNCS 7017, pp. 54–63, 2011. © Springer-Verlag Berlin Heidelberg 2011

Extending Synchronization Constructs in OpenMP to Exploit Pipeline Parallelism

55

based on thread and the other is based on iteration), by which programmers can establish the pipeline model flexibly and simply. Furthermore, runtime interfaces are implemented. To evaluate the performance, we conduct the experiments on the Cell Blade using the NAS IS, EP, LU [14] and SPEC2001 MOLDYN [20] benchmarks. In IS, EP and MOLDYN, from 4.8 to 5.5 speedup factors can be obtained from our pipeline model. In LU, pipeline structure can be established easily using our extended directives rather than using complicated data structures. The remainder of the paper is structured as follows: Section 2 discusses the related work. Section 3 presents the extended directives for pipeline parallelism. In Section 4 we show the runtime support for the pipeline parallelism on Cell Architecture. Experimental results are presented in Section 5 before we conclude in Section 6.

2

Related Work

Pipeline parallelism has been researched on both various programming languages and different architectures. Gonzalez et al. present the research work about extending OpenMP directives to exploit pipelined executions in OpenMP [2]. This pipeline model is work-sharing constructs oriented, while our pipeline model is loop-oriented which makes it possible to partition and pipeline the tasks in the critical region. Michailidis et al. propose a pipeline implementation of LU Factorization in OpenMP using the queue data structure [5]. In this pipeline scheme, no extra directives are extended to support pipeline. In contrast, two functions Put() and Get() are implemented using the original syntax of OpenMP to exchange elements between threads of the pipeline. Baudisch et al. present an automatic synthesis procedure that translates synchronous programs to software pipelines [6]. The compiler [8] generates the dependency graph of the guarded actions. Based on this, the graph is split into horizontal slices [6] that form threads to implement pipeline stages. In addition, threads communicate through FIFO buffers to allow asynchronous execution. Decoupled Software Pipelining (DSWP) [12, 3] can exploit the fine-grained pipeline parallelism inherent in most applications. In order to ensure that critical resources are kept in one thread, DSWP partitions the loop body into a pipeline of threads, rather than partitioning a loop by putting different iterations into different threads. Low-overhead synchronization between the pipeline stages can be implemented with a special synchronization array [3]. Coarse-grained parallelism is suitable for stream programming, because streaming applications are naturally represented by independent filters that communicate over explicit data channels [13]. Thies et al. present exploiting coarse-grained pipeline parallelism in C programs [11] to improve the performance of streaming applications. On Parallel Processor Arrays (PPA), Syrivelis et al. present a runtime support to extract coarse-grained pipeline parallelism out of sequential code [4]. Kurzak et al. present solving system of linear equations on CELL processor [7], and single instruction multiple data parallelism, instruction level parallelism and thread-level parallelism are well exploited on CELL processor. Nevertheless, the pipeline parallelism is less involved in this scheme.

56

S. Li et al.

3

Synchronization Constructs Extensions

In order to exploit pipeline parallelism on both thread and iteration levels, we equip the programmers with two sets of directives correspondingly. One can define the precedences at thread level while the other can define the precedences at iteration level. Synchronizations in the pipeline model are implemented by the extended directives. Each directive is described in detail as follows. 3.1

Synchronization Directives at Thread Level

The OpenMP API uses the fork-join execution model [1]. However, some constructs in the parallel region make the execution or the memory access sequential, such as the critical, atomic and ordered constructs. When the computation amount is great, synchronization constructs badly degrade the performance of the parallel applications. The straightforward method to improve the performance is pipeline and specifying the precedences relationship between different threads. The syntax of the ThrdPipe construct (C or C++ version) is as follows: #pragma omp ThrdPipe [clause[ [,]clause] …] new-line for-loops The clause is one of the following: blck_num (integer-expression) mry_syn (scalar-expression) The ThrdPipe construct specifies that the iterations of the outermost loop will be executed in pipeline by threads in the current context. This construct is one of the extended synchronization constructs which help the programmer to pipeline the loops with data dependency that can’t be parallelized by the loop construct directly. The blck_num clause specifies the number of blocks that the outermost loop is partitioned to. Subsequently, the block size and the theoretical speedup factor can be determined as follows. • Block Size: In our runtime implementation, we simply use a static partition algorithm to partition the outermost loop. For the number of blocks specified p and a loop of n iterations, let the integer q and r that satisfies n=p*q-r, with 0<=r
Extending Synchronization Constructs in OpenMP to Exploit Pipeline Parallelism

57

space equal to the whole shared array. In contrast, we can also choose to allocate address space equal to the block size to reduce the space overhead. In the former choice, the memory access synchronization is not needed, because it’s impossible to overlap the data block. In the latter choice, synchronization must be used to guarantee that there is no data block overlapping. 3.2

Synchronization Directives at Iteration Level

Iteration dependencies are common in the OpenMP applications. The loop construct can certainly partition the loop into several blocks. However, if the blocks be executed in parallel in discrete threads, the terminal result will be invalid, as the iteration dependency makes the computation ordered. If we want to pipeline this program segment, complex data structure will be introduced using the existing OpenMP syntax. In this section, two directives are extended to support pipeline at iteration level. The syntax of the IterWait and GotoIter constructs ( C or C++ version ) is as follows: #pragma omp IterWait( iter_ident, syn_ident ) new-line #pragma omp GotoIter( iter_ident, syn_ident ) new-line Both the IterWait and GotoIter directives are used in the affected loop and their two parameters iter_ident and syn_ident must be specified by the programmers. The IterWait construct used in the beginning of the loop block represents that the thread can’t continue to perform the following code until the iteration specified by the iter_ident parameter has been performed. The GotoIter construct used at the end of the loop block represents the thread continue to perform the following code and informs the iteration specified by the iter_ident parameter that the current iteration has been performed. The parameter iter_ident is an integer expression which contains the loop control variable and an integer constant, combined by the plus or subtract operation. In the expression, the loop control variable represents the current iteration. The whole expression specifies the iteration that should be waited for or should be informed. The parameter syn_ident indicates the synchronizations ID. In our pipeline scheme, multiple synchronizations are supported in order to make the pipeline model more reliable. Nevertheless, meaningful synchronization is formed only by the effective combination of one IterWait directive and its corresponding GotoIter directive. So an exclusive ID is assigned to them by the programmer through the syn_ident parameter. As a result, the IterWait directive and GotoIter directive with the same ID form one synchronization operation.

4

Runtime Support on Cell Architecture

In this section, we present the runtime system to support the extended directives. In order to take full advantage of the Cell architecture, we adopt the signal block transmission mechanism to implement the synchronization among different SPE threads, using the two signal notification channels of each SPE. The runtime library framework is based on our previous research work [18]. In this section, we only show the main runtime interfaces related to extended synchronization constructs.

58

4.1

S. Li et al.

Runtime Interface Supporting Synchronization Based on Threads

Fig. 1 shows the main implementation of one couple of the thread synchronization runtime interfaces, and the two interfaces are explained below. Another couple are implemented in the same way based on the other signal notification channel. void goto_nxt_thread_sig1(int thrdid): This function mainly plays a role in sending signal to other SPEs. The current thread, whose ID is curr_id, sends the signal to the thread whose ID is thrdid. The address of signal notification registers is stored in the array reg_add. void wait_prv_thread_sig1(int thrdid): This function mainly plays a role in waiting signals from other SPEs in block model. The current thread, whose ID is curr_id, reads the signal through the function spu_read_signal1(). void goto_nxt_thread_sig1 (int thrdid){ if(thrdid!=curr_id) /****Send signal_1 ***/ pip_send_signal( reg_add[thrdid], curr_id, 31-thrdid);}

void wait_prv_thread_sig1 (int thrdid){ uint32_t sig = 0; if(thrdid!=curr_id) /***Receive signal_1***/ sig=spu_read_signal1();}

Fig. 1. Thread synchronization runtime interface implementation

4.2

Runtime Interface Supporting Synchronization Based on Iterations

Fig. 2 shows the main part of the implementation of the iteration synchronization runtime interface. Different from the thread synchronization, these interfaces bring fine synchronizations of iteration level. void goto_nxt_loopindex_sig1(int index, int k){ /****Get lower and upper bounds of the loop***/ int lb=curr_lb; int ub=curr_ub; int id=-1; if(lb
Fig. 2. Iteration synchronization runtime interface implementation

void goto_nxt_loopindex_sig1(int index, int k): In this function, we firstly get the lower bound and upper bound in the current thread. Then we will judge whether the iteration need to be synchronized is in the bound. If it’s in the bound,

Extending Synchronization Constructs in OpenMP to Exploit Pipeline Parallelism

59

it will be executed in the current thread serially. Otherwise the function int probe_threadId(int iterID) will be scheduled, which will return the remote thread ID which the iteration locates in. Then it’ll send signal to the remote thread using the thread ID. void wait_prv_loopindex_sig1(int index, int k): In this function, we also judge whether the iteration need to be synchronized is in the bound. If not, it’ll read the signal in the channel by the block model.

5

A Case Study

In this section, we present a case study, pipeline optimization for IS critical region based on thread synchronization in Section 4.1, to introduce how to use the extended directives and the pipeline models. SPEs

#pragma omp critical {for(i=0; i<MAX_KEY; i++) key_buff1[i]+=prv_buff1[i];} Fig. 3a. Original code

SPE5 SPE4 SPE3 SPE2 SPE1 SPE0

t0

#pragma omp ThrdPipe blck_num(16) mry_syn (1) {for(i=0; i<MAX_KEY; i++) key_buff1[i]+=prv_buff1[i];} Fig. 3b. Extended code

t1

t2

Time

for(n=0; n
t3

t4

… …

t5 t6 t7

Block_1

Block_2 Block_3

Synchronization_1

Fig. 3d. Pipeline mode Fig. 3. IS critical region pipeline implementation

Synchronization_2

60

S. Li et al.

One paradigm difficult to parallelize in OpenMP is critical region, which may seriously sacrifice the performance when the computation amount is large. IS contains critical region in which the values of private array are added to the shared array. The original code of IS critical region is shown in Fig. 3a. The critical region retrogresses to serialization because of the read-write dependence about the shared array. In Fig. 3b, we use our extended ThrdPipe construct to optimize the critical region, and its clauses indicate that the shared array key_buff1 is divided into 16 blocks and memory access synchronization is used to reduce the space overhead. Fig. 3c shows the translated code. The constant PARTITION is equal to 16, and because the memory synchronization is chosen, two couples of thread synchronization runtime interfaces are inserted. Fig. 3d presents the pipeline model.

6

Evaluations

The hardware environment of the experiments is a Cell Blade [19], which contains two 3.2GHz Cell processors and 1GB system memory. Each Cell processor contains one PPE and eight SPEs. The benchmarks we use are NAS IS, EP, LU [14] and SPEC2001 MOLDYN [20]. The version of the operating system is Linux Kernel 2.6.25-14. All the applications are compiled by the ppu-gcc and spu-gcc compilers integrated in Cell SDK 3.1, and the optimization level is O5. We first use the ThrdPipe construct to pipeline the critical region in IS, EP and MOLDYN. 8 SPEs in the Cell Blade are used in the pipeline model. Fig. 4, Fig. 5 and Fig. 6 respectively show the speedup factors of IS, EP and MOLDYN critical regions after pipelining in different data sizes and different numbers of blocks. After analyzing the results horizontally and vertically, three aspects conclusions can be got. Firstly, in terms of a specific critical region, when the number of blocks is definite, the speedup factor is proportional to the data size. This is because when the data size increases, the granularity of each block increases. The overhead of data transfer and synchronization operations decreases in the proportion of the total execution time. In Fig. 4, when the IS critical region is divided into 8 blocks, the speedup factor increases from 0.99 to 3.85 as the data size increases from 8 KB to 4 MB. Secondly, when the data size of one critical region is definite, the speedup factor will increase at the beginning and then decline subsequently as the number of blocks increases. This is because the more blocks we divided, the pipeline will exploit more parallelism. Nevertheless, with the increasing number of blocks, the system will introduce more synchronization operations. When the performance improvement can’t compensate the extra overhead, the speedup factor will decrease. In Fig. 5, when the data size in the EP critical region is 512 KB, the speedup factor increases from 1.37 to 2.10 as the number of blocks increases from 2 to 16. However, when the number of blocks further increases to 32, the speedup factor decreases to 2.00. Thirdly, when the data size and the number of blocks are definite, the speedup factor is proportional to the computation complexity in the critical region, because the task granularity is coarser when the computation complexity increases. Then the pipeline will be more effective. From Fig. 5 and Fig. 6, we can see when the data size and the number of blocks are identical, MOLDYN definitely gets a higher speedup factor than EP, as MOLDYN contains more complex computation than EP.

Extending Synchronization Constructs in OpenMP to Exploit Pipeline Parallelism

2 blocks

6

4 blocks

8 blocks

16 blocks

61

32 blocks

5 pu 4 d eep 3 S2 1 0 8 KB

16 KB

256 KB

4 MB

IS Critical Region Data Size

Fig. 4. IS critical region speedup after pipelining

2 blocks

6

4 blocks

8 blocks

16 blocks

32 blocks

5 pu 4 ede 3 pS 2 1 0 16 KB

32 KB

256 KB

512 KB

8 MB

EP Critical Region Data Size

Fig. 5. EP critical region speedup after pipelining

2 blocks

7

4 blocks

8 blocks

16 blocks

32 blocks

6 5 up 4 de e pS 3 2 1 0 16 KB

32 KB

256 KB

512 KB

8 MB

MOL Critical Region Data Size

Fig. 6. MOLDYN critical region speedup after pipelining

Table 1 presents the execution time of blts function of LU, in which the original pipeline is located, before and after using the extended the IterWait and GotoIter directives. The original pipeline is implemented primarily by busy wait and continuously flushing between system memory and local store. However, the flush operation makes no sense when there is no change in the flag[] array. The great amounts of flush operations bring large overhead especially on the Cell architecture. In our pipeline scheme, this problem is solved radically. The pipeline is supported through the signal block communications mechanism at runtime. From the figures in Table 1, we can

62

S. Li et al.

figure out that when the SPEs number varies from 2 to 16, the time overhead of blts function falls by 23.3% averagely, and this illustrates that our pipeline scheme is more efficient that the original one. Table 1. Execution time of LU blts function before and after pipeline optimization Execution time (second)

2 SPEs

4 SPEs

8 SPEs

16 SPEs

Pipeline based on flush

122.9

64.7

34.1

20.6

Pipeline based on signal

97.8

51.0

26.1

14.8

7

Conclusions

In this paper, two sets of synchronization constructs are extended to exploit pipeline parallelism in OpenMP. One can define the precedence at thread level while the other can define the precedence at iteration level. The main contributions of our work are: (i) more parallelism can be exploited using the extended directives; (ii) the extended directives can help the programmers express pipeline more easily; (iii) a high efficient runtime library is implemented using the block communications mechanism to support the extended directives on Cell processor. Experiments results show that our implementation exhibits good performance and scalability. As future work, it’ll be interesting to investigate whether this scheme is suitable for homogeneous CMP architectures. Acknowledgments. The research is partially supported by the Hi-Tech Research and Development Program (863) of China under Grant No. 2006AA01Z105 and No. 2008AA01Z109, Natural Science Foundation of China under Grant No. 60373008, and by the Key Project of Chinese Ministry of Education under Grant No. 106019 and No. 108008.

References 1. OpenMP Application Program Interface, Version 3.0. OpenMP Architecture Review Board (2008) 2. Gonzalez, M., Ayguadé, E., Martorell, X., Labarta, J.: Defining and supporting pipelined executions in OpenMP. In: Eigenmann, R., Voss, M.J. (eds.) WOMPAT 2001. LNCS, vol. 2104, pp. 155–169. Springer, Heidelberg (2001) 3. Rangan, R., Vachharajani, N., Vachharajani, M., August, D.I.: Decoupled software pipelining with the synchronization array. In: Proceedings of the 13th International Conference on Parallel Architectures and Compilation Techniques (PACT), pp. 177–188. IEEE Press, ashington, DC (2004) 4. Syrivelis, D., Lalis, S.: Extracting coarse-grained pipelined parallelism out of sequential applications for parallel processor arrays. In: Berekovic, M., Müller-Schloer, C., Hochberger, C., Wong, S. (eds.) ARCS 2009. LNCS, vol. 5455, pp. 4–15. Springer, Heidelberg (2009)

Extending Synchronization Constructs in OpenMP to Exploit Pipeline Parallelism

63

5. Michailidis, P.D., Margaritis, K.G.: Implementing parallel LU factorization with pipelining on a multicore using OpenMP. In: 13th IEEE International Conference on Computational Science and Engineering, pp. 253–260 (2010) 6. Baudisch, D., Brandt, J., Schneider, K.: Multithreaded code from synchronous programs: Generating software pipelines for OpenMP. In: Methoden und Beschreibungssprachen zur Modellierung und Verifikation (MBMV), Dresden, Germany (2010) 7. Kurzak, J., Dongarra, J.: QR factorization for the CELL processor. Scientific Programming 17, 31–42 (2009) 8. Baudisch, D., Brandt, J., Schneider, K.: Multithreaded code from synchronous programs: Extracting independent threads for OpenMP. In: Design, Automation and Test in Europe (DATE), pp. 949–952. European Design and Automation Association (2010) 9. Teruel, X., Unnikrishnan, P., Martorell, X., et al.: Openmp tasks in ibm XL compilers. In: Proc. of the 2008 Conference of the Center for Advanced Studies on Collaborative Research, pp. 207–221. ACM Press, New York (2008) 10. Gschwind, M.: Chip multiprocessing and the cell broadband engine. In: CF 2006: Proceedings of the 3rd Conference on Computing Frontiers, pp. 1–8 (2006) 11. Thies, W., Chandrasekhar, V., Amarasinghe, S.: A practical approach to exploiting coarse-grained pipeline parallelism in C programs. In: Proceedings of the 40th Annual IEEE/ACM International Symposium on Microarchitecture, pp. 356–369. IEEE Press, Washington, DC (2007) 12. Ottoni, G., Rangan, R., Stoler, A., August, D.I.: Automatic thread extraction with decoupled software pipelining. In: Proceedings of the 38th IEEE/ACM International Symposium on Microarchitecture, pp. 105–118. IEEE Press, Washington, DC (2005) 13. Gordon, M.I., Thies, W., Amarasinghe, S.: Exploiting coarse-grained task, data, and pipeline parallelism in stream programs. In: Proceedings of the 12th International Conference on Architectural Support for Programming Languages and Operating Systems, pp. 151–162. ACM, New York (2006) 14. Jin, H., Frumkin, M., Yan, J.: The OpenMP implementation of NAS parallel benchmarks and its performance. NAS Technical Report NAS-99-011, NASA Ames Research Center, Moffett Field, CA(1999) 15. Ayguade, E., Copty, N., Duran, A., Hoeflinger, J., et al.: A proposal for task parallelism in OpenMP. In: Chapman, B., Zheng, W., Gao, G.R., Sato, M., Ayguadé, E., Wang, D. (eds.) IWOMP 2007. LNCS, vol. 4935, pp. 1–12. Springer, Heidelberg (2008) 16. Ayguade, E., Martorell, X., Labarta, J., Gonzalez, M., Navarro, N.: Exploiting multiple levels of parallelism in OpenMP: a case study. In: 1999 International Conference on Parallel Processing (ICPP), pp. 172–180 (1999) 17. Suess, M., Leopold, C.: Implementing data-parallel patterns for shared memory with OpenMP. In: Proceedings of the International Conference on Parallel Computing (PARCO). IOS Press, Amsterdam (2008) 18. Cao, Q., Hu, C., He, H., Huang, X., Li, S.: Support for OpenMP tasks on cell architecture. In: Hsu, C.-H., Yang, L.T., Park, J.H., Yeo, S.-S. (eds.) ICA3PP 2010. LNCS, vol. 6082, pp. 308–317. Springer, Heidelberg (2010) 19. Altevogt, P., Boettiger, H., Kiss, T., et al: IBM BladeCenter QS21 hardware performance, IBM Technical White Paper WP101245 [R], USA (2008) 20. SPEC: Standard Performance Evaluation Corporation, http://www.spec.org

Generic Parallel Genetic Algorithm Framework for Protein Optimisation Lukas Folkman, Wayne Pullan, and Bela Stantic Institute for Integrated and Intelligent Systems, Griffith University, Australia

Abstract. Proteins are one of the most vital macromolecules on the cellular level. In order to understand the function of a protein, its structure needs to be determined. For this purpose, different computational approaches have been introduced. Genetic algorithms can be used to search the vast space of all possible conformations of a protein in order to find its native structure. A framework for design of such algorithms that is both generic, easy to use and performs fast on distributed systems may help further development of genetic algorithm based approaches. We propose such a framework based on a parallel master-slave model which is implemented in C++ and Message Passing Interface. We evaluated its performance on distributed systems with a different number of processors and achieved a linear acceleration in proportion to the number of processing units. Keywords: Parallel Genetic Algorithm, Protein Optimisation, Protein Structure Prediction.

1

Introduction

Proteins form a group of one of the most important macromolecules in living organisms both quantitatively and functionally. In order to understand the function of a protein, its three-dimensional (3D) structure needs to be determined. Different computational approaches for Protein Structure Prediction (PSP) have been proposed for this purpose. Some of them rely on a template database of known structures and look for sequence similarities. On the other hand, an ab initio (meaning ‘from the origin’) method for PSP is based on the thermodynamic hypothesis which states that a protein’s native conformation is at its free energy minimum. Hence, a conformational search is conducted to find such 3D structure. The generic parallel genetic algorithm framework presented in this paper is aimed for the ab initio method. There are two major issues in addressing PSP using the ab inito approach: a) computationally modelling the problem and designing an effective energy function; b) an efficient conformational search technique to search the free energy landscape to identify conformations which have native features. The energy function is used to lead the conformational search through the vast space of possible conformations. Such an energy function needs to be able Y. Xiang et al. (Eds.): ICA3PP 2011 Workshops, Part II, LNCS 7017, pp. 64–73, 2011. c Springer-Verlag Berlin Heidelberg 2011 

Parallel Framework for Protein Optimisation

65

to distinguish between random and near-native conformations. Energy functions can be classified according to their use of statistical information of known structures into physics-based and knowledge-based energy functions. In the latter one, rather than calculations of physical energy terms describing atomic interactions, energies are derived from the statistical information about the experimentally solved structures in protein databases. It is not feasible to perform a search of all possible conformations in order to find the native one due to the size of the search space. Molecular dynamics simulations together with a physics-based energy function can be used for ab initio prediction. On the other hand, much faster approaches are stochastic and heuristic conformational searches. Simulated annealing using Monte Carlo (MC) [6], Replica Exchange MC [5] and MC with minimisation [8] have proven well in a number of successful PSP algorithms [11,15]. Evolutionary computation methods such as genetic and population-based algorithms lend themselves well for optimisation problems in bioinformatics [10] as well as in PSP [13,7,9,4]. The strength of an effective Genetic Algorithm (GA) lies in its intelligent genetic operators which are applied in turns to different conformations throughout the search. Beside genetic operators, different GA approaches may have a lot in common. Hence, several general GA frameworks are available, for example [2]. In this paper, we would like to propose a framework that is dedicated solely to PSP and requires as little input as possible from the user. Furthermore, several models for a Parallel Genetic Algorithm (PGA) were described in [14]. We utilise one of them in the design of our framework in order to enhance the performance of predictions on distributed systems. The rest of this paper is organised as follows: an overview of the related work in the field is given in Sect. 2. Section 3 describes our generic PGA framework. The experiments and the results obtained with an implementation of our framework are presented in Sect. 4. We conclude the paper and suggest future work in Sect. 5.

2

Related Work and Background

Genetic Algorithms (GAs) are search heuristic methods which can be used for different hard optimisation problems. At the beginning of a GA, a random population of individuals is generated. Each individual encodes a candidate solution of the optimisation problem. It is important for the population to have high entropy, meaning individuals differ significantly and provide samples from different parts of the search space. Throughout the process of a GA, this population evolves using replication, crossovers and mutations. Only the fittest individuals are allowed to stay in the population. By this process of natural selection, the best solutions of the problem are found in the end. It should be noted that the existence of a fitness function is essential for a GA. 2.1

Genetic Algorithms in Protein Structure Prediction

GAs were applied for the first time to the PSP problem in [13]. The authors proposed using different conformations of the target protein for the population of

66

L. Folkman, W. Pullan, and B. Stantic

individuals. The protein’s energy function was used to evaluate the fitness. They designed a crossover as a binary operator that cuts two different conformations at a random position and swaps their two parts producing two new conformations. A mutation was defined as a unary operator which makes a random rotation around one of the residues. The authors described that in the end, only the fittest, meaning lowest in energy, conformations were present in the population. In [7], Conformational Space Annealing (CSA) search is proposed which is based on a distance cut-off GA with minimisation. CSA was applied in [9] for prediction of proteins’ tertiary structures. Furthermore, a feature-based resampling method based on a GA for structure refinement was presented in [4]. 2.2

Parallel Genetic Algorithms

Parallel Genetic Algorithms. (PGAs) for PSP are discussed in [12]. Three models are described – master-slave, fine-grained and coarse-grained. In the master-slave model, there is a master process distributing fitness calculation jobs among its slave processes. The decision making and genetic operators are invoked by the master process. In the other two models, the population is divided into several equal sub-populations. Each process takes care of one such sub-population. The sharing of individuals from these sub-populations among neighbouring processes is allowed in the fine-grained model. For the case of the coarse-grained model, only limited information exchange is possible via the concept of migrations. Although both theoretical and experimental results in [12] suggest that the master-slave model does not perform as well as the other models, it should be noted that the authors used a two-dimensional hydrophobic-polar lattice model to represent a protein’s conformation. In such case, energy calculation is fast and this can be seen as the reason why communication overhead in the master-slave model was so significant. For computationally expensive energy functions, the master-slave architecture may perform reasonably well. A lot of work has been done recently in the field of parallel multiobjective evolution algorithms (PMOEAs) [14,3]. In [14], four PMOEA models (masterslave, diffusion or fine-grained, island or coarse-grained, and hierarchical hybrid ) are described. The first three are identical to those described in the previous paragraph and in [12]. The hierarchical hybrid model is based on the combination of the others. In [3], the authors designed an ab initio PSP PMOEA based on the master-slave model and achieved linear acceleration in proportion to the number of processors.

3

Methods

The overall design of our generic PGA framework and its implementation in the C++ programming language is described in detail in this section. The Boost object-oriented implementation of Message Passing Interface (MPI) was used for communication among processors on distributed systems.

Parallel Framework for Protein Optimisation

3.1

67

Generic Genetic Algorithm Framework

The class diagram of the framework is depicted in Fig. 1a.

(a) framework class diagram

(b) master-slave model activity diagram

Fig. 1. Generic parallel genetic algorithm framework class diagram and parallel masterslave genetic algorithm activity diagram

Pool and Conformations. The interfaces IPool and IConformation together with their implementation classes Pool and Conformation form the core of the whole framework. The Pool object represents the population of individuals and is responsible for deciding whether newly created individuals should be added to the pool or discarded. Furthermore, it is aware of which individuals are being currently processed. The pool contains a set of Conformation objects. The Conformation object represents a specific conformation of the target protein. It keeps a reference to the Protein object (IProtein interface), which describes static features like the name and amino acid sequence of the protein. The Conformation object is used to access and set torsion angles, it takes care of calculating its own free energy as well as derivatives of the energy function for every torsion angle. For this reason, Conformation keeps a set of Derivative objects (IDerivative interface). A Derivative object is basically a couple of the derivative value and the torsion angle identification (ITorsionId interface, TorsionId implementation) for which the value applies. At the beginning of the run of a GA, the population of different conformations needs to be generated. For this purpose, Conformation object can be perturbed. This method alters randomly φ and ψ backbone angles of a number of residues of the conformation by a specific value. This perturbation is accepted only if altered angle values are valid according to the Ramachandran plot. This procedure ensures that all the sub-parts of the conformations can be used reasonably well in the GA search. Energy Function and Conformation Provider. In our framework, we decided to utilise some of Rosetta’s modules [11]. We decided to use Rosetta mainly

68

L. Folkman, W. Pullan, and B. Stantic

for two reasons. First of all, it is an open-source project which makes it possible to extract only the needed functions and optimise them for our purpose. Secondly, Rosetta proved itself as a promising approach in several Critical Assessment of techniques for protein Structure Prediction (CASP) competitions [1]. We use Rosetta’s scoring function to calculate conformation’s energy. Furthermore, we decided to use Rosetta’s Pose object to represent protein’s spatial arrangements. Pose provides means for both full-atom and reduced modelling of a conformation. In the latter case, each side-chain is modelled as a united atom – centre of mass. A framework’s user can choose which model will be used according to their computational resources as full-atom modelling is extremely expensive. The energy calculation is encapsulated in the RosettaProvider singleton object which acts like a mediator for Rosetta’s functionalities and takes care of all the needed inclusions. This way it is easy to use Rosetta’s energy function as the complexity of the robust system is hidden in our provider class. Local Optimisation. In our design, an individual is locally optimised after it has evolved – meaning the crossover or mutation operator has been applied to it. This is done in order to keep conformations at their local minima at all times. This approach was originally proposed for MC simulations in [8] and successfully applied in Rosetta [11]. CSA [7], a GA-based approach, applies local optimisation for each new individual. This helps to rescue conformations that may have had their energy increased after crossover or mutation even though they were moved to another valley in the free energy surface at which bottom structures with lower energy can be found. Another advantage of this approach is redefining the continuous space to be searched into a discrete one composed by local minima of the free energy [8]. The Limited memory Broyden-Fletcher-Goldfarb-Shanno (L-BFGS) method is employed for this purpose here, similarly as in [10]. The LBFGSOptimiser object fulfils this role (IOptimiser interface). Based on derivatives of the energy function for each torsion angle, the local minimum can be found. Genetic Operators. Only two methods – crossover and mutate – are defined in the IGeneticOperator interface. These methods are expected to be implemented by a framework’s user as they form the core functionality of a GA. By having the possibility to implement intelligent genetic operators, one can make their GA more successful than others. An important aspect of our design is the use of interfaces. As stated in the previous subsections, each implementation class implements an interface. The whole framework communication is defined only in terms of these interfaces. This allows for a high level of flexibility. 3.2

Parallel Implementation of the Framework

As described earlier in Sect. 2, there are several design patterns that address parallelism in GAs. We used the master-slave model in our approach – depicted in Fig. 1b. First of all, it is based on the idea that the master process maintains the

Parallel Framework for Protein Optimisation

69

population of individuals and assigns jobs to its slave processes. Hence, there is no need to alter the GA itself. Secondly, in our framework, the work-load for the slave processes should be high enough to keep them occupied and utilise the whole parallel system efficiently. In order to comply with this, we alter the master-slave model described in Sect. 2 slightly. The fitness function is computed by the slave processes. Furthermore, after any change to a conformation, local optimisation of this structure is performed. This is another computation that is passed onto the slave processes as it is computationally expensive. It can even be requested at no extra communication cost as local optimisation and calculation of the energy are tightly coupled so it is naturally performed within the same request. In order to utilise all the slave processes as much as possible, we decided to use a somewhat modified GA. We do not keep the state of a current generation. Individuals enter and leave the population as they are evolving. If we had kept the state of the current generation, the master process would have had to wait for all the slave processes to return the optimised individuals. Just then, it would have made a decision about the individuals in the current generation. Some slave processes would have returned earlier and some later. However, there would have been no work to be given to the early ones leaving these slave processes idle. In our design, after an individual is returned to the master by a slave process, the decision whether it is good enough to stay in the population is made immediately. In case of a negative answer, the individual is discarded. As mentioned before, the genetic operations performed over conformations are computationally inexpensive so they are traditionally performed by the master process. In order to implement this approach, the master process would have had to keep the count of which conformations had already evolved and had been waiting to be sent to the slave processes. Furthermore, some slave processes might have had to wait for the master process to finish genetic operations before they could have returned their results to it. The crossover and mutation always precede the local optimisation. From this point of view, genetic changes, which are fast to perform, could be done by the slave processes within the single request. We follow this idea in order to keep the design as simple as possible and put the utilisation of the slave processes first as they perform the most time-consuming activity.

4

Experimental Results

In order to evaluate benefits, efficiency and usability of our generic PGA framework for protein optimisation, the IGeneticOperator interface needed to be implemented. For this purpose, we used a simple version of crossover and mutation operators from [4]. The crossover operator splits two conformations and swaps their two parts whereas a mutation is performed as a rotation around a single residue. The chance of performing a crossover was equal to the chance of a mutation. The initialisation of the pool was conducted by the perturbation of 3 residues by 10–15 ◦ in each conformation. Our test set consisted of only two proteins – 2ptl (78 residues) and chain A of 3mwx (326 residues) with 3D structures experimentally solved in the past.

70

L. Folkman, W. Pullan, and B. Stantic

2,000 iterations (where one iteration corresponds to the invocation of a genetic operator over one individual) of our GA were executed with a pool size of 100 conformations. This simulation was run as a serial program on a single processor as well as on 2, 3, 4 and 5 processors in parallel using our parallel implementation of the framework. Each run was executed 5 times and the results were averaged. The Rosetta’s decoy structure of 2ptl and the experimentally determined structure of chain A of 3mwx were used as the input structures for the run of our algorithm. Our intention is not to present the ability of the described implementation of our framework to predict the tertiary structure of proteins from their sequences. However, we would like to evaluate the performance of the parallel implementation of the framework. 4.1

Acceleration by Parallelism

The results describing the performance of our framework are presented in Fig. 2a. The x axis is the number of processors and the y axis is the average acceleration. The average running time of the serial algorithm on the 2ptl input was 2,314 s on average whereas it was only 476 s when a master process with 4 slave processes were employed on a parallel system with 5 processors. For the case of the much larger chain A of 3mwx, it was 26,448 s and 5,598 s on average respectively. Figure 2a shows that we achieved linear acceleration in proportion to the number of processors employed. In addition, we were interested in how well the slave processes were utilised and what the communication overhead was. Figure 2b depicts the percentage of time that the master and slave processes spent working. The x axis represents the number of slave processes whereas the percentage of time is shown on the y axis. The slave processes were utilised efficiently and kept working on optimisation of conformations 92.56 % of the total running time on average. However, as the master process was responsible only for assigning jobs to the slave processes and managing the pool, it was idle 86.60 % of the running time on average. Protein optimisation aims at minimising the protein’s free energy. In Table 1, the averaged results from our algorithm are listed when run on a parallel system with

(a) average acceleration

(b) time spent working

Fig. 2. Average acceleration by parallelism and percentage of time spent working

Parallel Framework for Protein Optimisation

71

Table 1. Mean initial and optimised energies and their deviations and the best and mean RMSDs and TM-scores ¯ Protein Init. energy δIE 2ptl 3mwx

248.21 1609.41

142.72 796.59

¯ ¯ ¯ TM-score TM-score Opt. energy δOE RMSD RMSD −44.54 −426.07

9.57 6.69 ˚ A 10.24 ˚ A ˚ 18.12 A ˚ 45.49 11.45 A

0.36 0.69

0.32 0.64

5 processors. It can be seen that the energy was significantly decreased. However, this does not guarantee good prediction accuracy. In Table 1, the best and mean RMSDs1 and TM-scores2 of the final models to the native structures are listed. As the input to our algorithm for chain A of 3mwx was its native structure, the best model was moved away by 11.45 ˚ A. In the case of refinement of Rosetta’s 2ptl decoy structure, the best model had RMSD of 6.69 ˚ A to the native. 4.2

Analysis of the Results

The decoy structure of the small protein 2ptl was used to demonstrate the usability of the PGA implemented in our framework for protein structure refinement and to look at the benefits of running such a refinement on a distributed system. The second protein in the test set was chain A of 3mwx and an experimentally determined structure was used for the input. Having 326 residues, it is quiet a large structure for ab initio prediction. We were interested especially in how the performance of our parallel framework will be sustained in this case. From the results (Fig. 2a), it is apparent that running our PGA on a distributed system with 2 processors does not accelerate the algorithm. This is a clear consequence of not loading the master process with a sufficient amount of work (Fig. 2b). Even though the master process’ work load is very low in the 3, 4 and 5 processors configuration as well, the acceleration becomes more evident and increases linearly in proportion to the number of processors. It should be noted that such acceleration will stop at some stage depending on the size of the target protein. Prediction of a larger protein would be able to load a higher number of processors efficiently.

5

Conclusion and Future Work

In this paper, we described our generic PGA framework for protein optimisation. The framework is designed and implemented in such a way that any of its components can be exchanged with the relevant component implemented by the user. The rest of the framework will continue to function without need for modifications. The basic operation of the framework is to evolve protein’s conformations using crossover and mutation operators, local optimisation of conformations according to their energy using the L-BFGS algorithm and evaluation of their fitness. 1 2

Root Mean Square Deviation of the positions of the backbone Cα atoms. a measure of similarity of topologies of two protein structures [16].

72

L. Folkman, W. Pullan, and B. Stantic

The framework can exploit parallel systems in an efficient manner using the master-slave approach where the master process assigns jobs to the slave processes. The communication overhead on the slave processes is kept as low as possible which results in a fast performance. 92.56 % of the running time on slave processes was utilised for evolving the conformations. Linear acceleration in proportion to the number of processors was achieved in refinement of one small and one large protein chain. The master process was idle 86.60 % of the running time which implied only marginal acceleration on a distributed system with 2 processors. In the future, we need to address utilisation of the master process in a greater depth. Implementation of the island model for PGAs could also provide better work load distribution. Acknowledgements. This research is partly sponsored by NICTA Queensland Research Laboratory.

References 1. Bradley, P., Chivian, D., Meiler, J., Misura, K., Rohl, C., Schief, W., Wedemeyer, W., Schueler-Furman, O., Murphy, P., Schonbrun, J., et al.: Rosetta predictions in CASP5: successes, failures, and prospects for complete automation. Proteins: Structure, Function, and Bioinformatics 53(S6), 457–468 (2003) 2. Cahon, S., Melab, N., Talbi, E.: ParadisEO: A framework for the reusable design of parallel and distributed metaheuristics. Journal of Heuristics 10(3), 357–380 (2004) 3. Calvo, J., Ortega, J.: Parallel protein structure prediction by multiobjective optimization. In: 2009 17th Euromicro International Conference on Parallel, Distributed and Network-based Processing, pp. 268–275 (2009) 4. Higgs, T., Stantic, B., Hoque, M., Sattar, A.: Genetic algorithm feature-based resampling for protein structure prediction. In: 2010 IEEE Congress on Evolutionary Computation (CEC), pp. 1–8. IEEE, Los Alamitos (2010) 5. Kihara, D., Lu, H., Kolinski, A., Skolnick, J.: TOUCHSTONE: an ab initio protein structure prediction method that uses threading-based tertiary restraints. Proceedings of the National Academy of Sciences of USA 98(18), 10125 (2001) 6. Kirkpatrick, S., Gelatt, C., Vecchi, M.: Optimization by simulated annealing. Science 220(4598), 671 (1983) 7. Lee, J., Scheraga, H., Rackovsky, S.: Conformational analysis of the 20-residue membrane-bound portion of melittin by conformational space annealing. Biopolymers 46(2), 103–115 (1998) 8. Li, Z., Scheraga, H.: Monte Carlo-minimization approach to the multiple-minima problem in protein folding. Proceedings of the National Academy of Sciences of USA 84(19), 6611 (1987) 9. Oldziej, S., Czaplewski, C., Liwo, A., Chinchio, M., Nanias, M., Vila, J., Khalili, M., Arnautova, Y., Jagielska, A., Makowski, M., et al.: Physics-based proteinstructure prediction using a hierarchical protocol based on the UNRES force field: assessment in two blind tests. Proceedings of the National Academy of Sciences of USA 102(21), 7547 (2005)

Parallel Framework for Protein Optimisation

73

10. Pullan, W.: An unbiased population-based search for the geometry optimization of Lennard–Jones clusters: 2 ≤ N ≤ 372. Journal of Computational Chemistry 26(9), 899–906 (2005) 11. Rohl, C., Strauss, C., Misura, K., Baker, D.: Protein structure prediction using Rosetta. Methods in Enzymology 383, 66–93 (2004) 12. Santos, E., Lu, L., Santos Jr., E.: Efficiency of Parallel Genetic Algorithms For Protein Folding on 2-D HP Model. In: Proceedings of the Fifth Joint Conference in Information Sciences, Third International Workshop on Frontiers of Evolutionary Algorithms, Atlantic City, NJ, pp. 1094–1097 (2000) 13. Unger, R., Moult, J.: Genetic algorithms for protein folding simulations. Journal of Molecular Biology 231(1), 75–81 (1993) 14. Van Veldhuizen, D., Zydallis, J., Lamont, G.: Considerations in engineering parallel multiobjective evolutionary algorithms. IEEE Transactions on Evolutionary Computation 7(2), 144–173 (2003) 15. Wu, S., Skolnick, J., Zhang, Y.: Ab initio modeling of small proteins by iterative TASSER simulations. BMC Biology 5(1), 1741–7007 (2007) 16. Zhang, Y., Skolnick, J.: Scoring function for automated assessment of protein structure template quality. Proteins: Structure, Function, and Bioinformatics 68(4), 1020 (2007)

A Survey on Privacy Problems and Solutions for VANET Based on Network Model Hun-Jung Lim1 and Tai-Myoung Chung2 2

1 Dept. of Computer Engineering, Sungkyunkwan University School of Information Communication Engineering, Sungkyunkwan University [email protected], [email protected]

Abstract. For a long term of vehicle communication research, now VANET is in a stage of implementation. However, most of VANET researches focus on message transmission. Vehicle is extremely personal device; therefore personal information, so called privacy has to be protected. In this paper, we analyze identity and location privacy threatening factors, problems, and solutions based on network model. To analyze solution’s effectiveness, we define four attack models: External attack, Internal attack, Correlational attack, and Relational attack. According to our research, most of the solutions use pseudonym identity or address changing scheme to protect identity privacy. Also, solutions are weak to or do not consider the relational attack. We analyze this is due the meet the network model’s transparency design goal and protect vehicle’s real identity even revealing the vehicle’s location. The result of this paper could guide a way to design a privacy preserve solution and present a trend of existing solutions. Keywords: VANET, Identity Privacy, Location Privacy.

1

Introduction

VANET is developed to support Car-to-Car (C2C) and Car-to-Infra(C2I) communication. For many years, global researchers and projects have been investigating VANET research issues: routing, security, address allocation, and etc. Based on these researches, some project group built a testbed and implemented programs on the vehicle for communication. The field test results of message exchange and network connectivity are fairly satisfied. For an additional research, they focused on security and privacy issues on VANET. Since the vehicle is extremely personal device, its communication data should be secured and the driver’s privacy should be unrevealed. Generally, privacy means “Right of an individual to decide for himself/herself when and on what terms his or her attributes should be revealed”[1]. Without privacy protection, driver’s attributes such as 5W1H(who, when, where, what, why ,and how) can be revealed and used by adversaries. Privacy in the context of VANET can be categorized into three parts [2]. ─ Data Privacy: Prevent others from obtaining communication data. ─ Identity Privacy: Prevent others from identifying subject of communication. ─ Location Privacy: Prevent others from learning one’s current or past location.

Y. Xiang et al. (Eds.): ICA3PP 2011 Workshops, Part II, LNCS 7017, pp. 74–88, 2011. © Springer-Verlag Berlin Heidelberg 2011

A Survey on Privacy Problems and Solutions for VANET Based on Network Model

75

Usually, data privacy easily achieved through encryption method in an application layer. For that reason, identity privacy and location privacy are usually mentioned as the privacy issues on VANET[3]. In this paper, we will identify the privacy threatening factors and solutions according to TCP/IP network model which will be the main network model in the VANET environment. The contributions of this paper are as follows: ─ By identifying the privacy threatening factors, we guide a way to design a privacy protection solution for follow researchers. ─ By researching the known privacy protection solutions, we describe trend of solutions and their strength and weakness. ─ By describing the treating factors based on network model, we cover all message fields transmitted over the network. This paper is structured as follows. In Section 2 to 5, we describe each layer’s privacy threatening factors, problems, and solutions: network access layer, network layer, transport layer, and application layer. In Section 6, we analyze the solutions based on four attack models and conclude the paper in Section 7.

2

Network Access Layer

At the network access layers, TCP/IP does not define any specific protocol. It supports all of standard and proprietary protocols. This layer puts data in frames and ensures error free transmission. In the scope of VANET environment, IEEE 802.11x technologies can be adapted such as 802.11b, 802.11g, 802.11n and 802.11p. Specially, IEEE 802.11p(WAVE) is designed for vehicular communication. IEEE 802.11x standards define ‘frame’ types to be used in data transmission as well as management and control of wireless links. For the privacy viewpoint, a physical address field, distinguishes a node within a local area, of the frame is a privacy threatening factor. Figure 1.illustrates frame format and privacy treating fields.

Fig. 1. IEEE 802.11 Format

76

H.-J. Lim and T.-M. Chung

Frame is divided into very specific and standardized sections. Each frame has a MAC header, payload and frame check sequence(FCS). MAC header is composed of frame control field, duration ID field, up to four MAC address fields, and sequence control filed. Figure 2 describes the MAC address format.

Fig. 2. MAC Address Format

MAC addresses are most often generated and assigned by the manufacturer of a network interface card (NIC) and are stored in its hardware. This 48-bit address space contains potentially 248 possible MAC addresses. The common MAC address notation is done in hexadecimal numbers of six groups of bytes separated by colons: 01:23:45:67:89:ab. The first 24-bits indicate Organizational Unique Identifier and the following 24-bits are host identifier. Problem By default, the NIC driver reads and uses the stored address, thus the identifiers typically persist as long as the same NIC is used. Also, according to the second least significant bit, which is either 0 or 1, MAC address is universally or locally unique respectively. Therefore every client is uniquely identified by its default MAC address(universally or locally). These two features, permanently and uniquely identify vehicle, cause identity privacy problem. In [4][5], the authors explain that WLAN is weak to privacy due to its follow features. ─ ─ ─ ─ ─ ─

Untrusted network operators Shared channel radio Unsecured frame header Frequent broadcast of MAC address High Density of access points(Low-cost wireless LAN radio) Precise positioning technology

Solution To the best of our knowledge, there is no VANET specific WLAN privacy protection research. However, there are lots of solutions for pervasive computing environment. These solutions could adapt to VANET environment [4][6][5]. Gruteser, M. suggests short-lived interface identifier for the privacy solution[4]. However, the concept of short-lived and periodically update interface identifiers creates new challenges: identifier selection, identifier uniqueness, and integration with port authentication. Huang, L. [5] also insists that the “periodical address updates”

A Survey on Privacy Problems and Solutions for VANET Based on Network Model

77

solution is weak to correlation attacks, which utilize the correlation between the old and new addresses of the same node. The author suggests a concept of a silent period for the additional solution. A silent period is defined as a non-transmission period between the use of new and old identifier. Pang, J evaluates correlation attack[7]. In [6], Jiang, T. summarizes five privacy leakage factors (time, location, sender node identity , receiver node identity, and content) and three privacy protecting methods. -

Anonymize the user or node identity with frequently changing pseudonyms Unlink different pseudonyms of the same user with silent periods between different pseudonyms Increase the entropy of the attackers’ location estimation by reducing the precision of location algorithms.

The other approach is suggested by Greenstein, B. using encryption method called SlyFi[8]. Figure 3 illustrates the privacy solutions for network access layer.

Fig. 3. Network Access Layer Privacy Solutions

3

Network layer

At the network layer, TCP/IP supports the Internetworking Protocol(IP). This layer is responsible for delivering a message from the source host to the destination host based on their addresses. IP has extended IPv4 to IPv6 for the solution of address pool shortage. On February 3, 2011, Internet Assigned Numbers Authority (IANA) announced that the last IP address blocks has allocated. We expect that the IPv4 based network model will turn into the IPv6 rapidly. In this paper, we briefly explain the IPv4 and investigate IPv6 in depth. In the scope of VANET environment, existing IPv4 and IPv6 protocol can be used. Additionally, mobility support IP can be used for vehicle’s mobility. For the privacy viewpoint, a logical address, distinguishes the node within a global area, is the privacy threatening factor. Figure 4 Illustrates IPv4 and IPv6 packet format and privacy threatening fields.

78

H.-J. Lim and T.-M. Chung

Fig. 4. IPv4 and IPv6 Format

3.1

IPv4

IPv4 is the most widely deployed Internet layer protocol. IPv4 is described in IETF publication RFC 791 (September 1981), replacing an earlier definition (RFC 760, January 1980). IPv4 uses 32-bit addresses, which limits the address space to 4.2 billion(232) possible unique addresses. However for some reasons, it is hard to allocate IP address uniquely which is the most important feature of IP address. Figure 5 illustrates the problem and privacy solutions for IPv4.

Fig. 5. IPv4 Problem and Privacy Solutions

To solve the address shortage problem, Dynamic Host Configuration Protocol [9], which generates host addresses based upon availability, and Network Address Translation [10], which maps the private address to the public address, are applied. These two techniques unintentional benefit of privacy protecting a host's address by hiding it within a private address space and by periodically change a host’s address. 3.2

IPv6

IPv6 is a version of the Internet Protocol that is designed to succeed IPv4. Since 1994, IPv6 was developed by the IETF 'IPng'(IP next generation) working group to deal the IPv4 exhaustion problem. IPv6 is described in IETF publication RFC 2460 (December 1998). To overcome the shortage of addresses in IPv4, IPv6 employs 128-bit addresses, which limits the address space to 5 1028 (2128) possible unique addresses. In addition to the larger address space, IPv6 designed for enhanced security and management aspects. For the management aspect, IPv6 supports Stateless auto-configuration which is described in rfc4862. Figure 6 illustrates IPv6 stateless auto-configuration scheme.

A Survey on Privacy Problems and Solutions for VANET Based on Network Model

79

Fig. 6. IPv6 Stateless auto-configuration scheme

Stateless auto-configuration makes an administrator to configure the network of the address while each device automatically configures the interface identifier (IID), of the address. The IID is formed by extending the 48-bit Media Access Control (MAC) address to a 64-bit number, spanning half of the IPv6 address. Problems From a privacy point of view, stateless auto-configuration scheme has two privacy risk: compromising location privacy due to IPv6 network prefix information and compromising identity privacy by IPv6 IID information[10, 11]. The first risk is that the network prefix informs us a network domain location and could guess the node’s location approximately [12]. The second problem is similar to network access layer problem. The component of the IPv6 address is the IID that remains static unless replacing the NIC. As a result, no matter what network the node accesses, IID remains the same. Consequently, simple network tools such as ping and trace-route can be used to track a node's geographic location from any-where in the world [13]. Lindqvist, J. insists that the 64bit-IID is enough to figure out the individuals and it is a major threat for user’s privacy [14]. Also, Trostle, J. mentions that it is possible to track a device by checking the global unicast address with the same Interface identifier and reminded us same Interface identifier’s risk[15]. Solutions To solve the first location problem, Trostle, J suggests encrypting the parts of the prefix such that only appropriate routers in the network can decipher the prefix and obtains the topological information [15]. To solve the second identity problem, IETF suggests privacy extensions [16]. Privacy extension uses a hash value of a nonce with the EUI-64 generated IID or the previous hash result, creating a new IID as the user changes networks. IETF also suggests to use a Cryptographically Generated Address(CGA) as an IID[17]. Both [16] and [17] use a random number; therefore the address is dynamically obscured each time. Also, DHCPv6 can be used for privacy identity protection[18]. In DHCPv6 address, a server assigns a random address to each node, independent from the MAC address. This solution also takes advantage of abundance of IPv6 addresses by allocating one IP address per application. The above suggestions can solve the privacy problems. However, it also requires additional resources such as hash calculation, DHCP server, and amount of addresses. Figure 7 illustrates the privacy solutions for IPv6.

80

H.-J. Lim and T.-M. Chung

Fig. 7. IPv6 Privacy Solutions

3.3

Mobility Support IP

IPv4 and IPv6 usually allocate an address based on network domain and route a message according to the network domain. However, in the VANET environment, a vehicle moves across network domains and periodically causes network handover. When every handover occurs, its IP address has to be changed and for that the connection will be broken. To solve the problem, mobility support IP technologies, called Mobile IP, are required. Mobile IP allows user to move from one network to another while maintaining the connection. Mobile IP for IPv4 is described in IETF RFC 3344 and Mobile IPv6 is described in RFC 3775. In this paper, we focus on the Mobile IPv6 with the same reason as the Internet protocol case. MobileIPv6 is also divided into two categories based on subject of mobility support: Host based and Network based. Host based mobility support : MobileIPv6 MobileIPv6 is a host based mobility support protocol. The core of MobileIPv6 is that the mobile node(MN) uses two kind of address to support a mobility. Home-AgentAddresss(HoA) and Care-of-Address(CoA). First, MN has a fixed IP address called HoA. When it moves to another network, MN generates tentative IP address called CoA. Then, MN sends a Bind Update(BU) message to its correspond node(CN) with new CoA. On receiving the BU message, CN update its HoA-CoA mapping table with new received CoA. Whenever CN sends a message to MN, CN checks the mapping table with a HoA and uses a correlated CoA as a destination address. The advantage of Host based mobility support protocol is that MN handles all of mobility signals even the network fully supports the mobility service. The disadvantage is that the MN requires additional mobility stack and battery consumption. Problem From a privacy point of view, the advantage of host based mobility threats the privacy. The end-point source and destination IP addresses are revealed to others because MN handles entire mobility signals. The privacy problems in the context of Mobile IPv6 are defined in rfc4882 [19]. The primary goal is to prevent adversary on the path between the MN and the CN from detecting roaming due to the disclosure of the HoA and exposure of a CoA to the CN. For example, when a MN roams from its home network to other network, use of a HoA in communication reveals to an adversary that the MN has roamed. Also, when MN roams from its home network other networks, use of CoA in communication with a CN reveals that the MN has roamed.

A Survey on Privacy Problems and Solutions for VANET Based on Network Model

81

Solution IETF suggests an Encrypted Home Address (eHoA) and a Pseudo Home Address (pHoA) to solve the privacy threat [20]. To protect privacy from adversary, the MN uses the eHoA. To protect privacy from CN, the MN uses the pHoA in the extended home address test procedure to obtain a home keygen token; then, it uses the pHoA instead of the real home address in the reverse-tunneled correspondent binding update procedure. Network Based mobility support : Proxy Mobile IPv6 PMIPv6 is network based mobility support protocol that is described in rfc5213. The core of PMIPv6 is that the network handles all of mobility support operations and MN does not participate in mobility. There are two methods to detect MN’s movement. Beacon information in layer2 and network prefix information in layer3 router advertisement message. In MobileIPv6, MN uses layer3 information to detect its movement. When MN detects its movement by receiving a different network prefix in RA message, it generates new CoA for communication. However in PMIPv6, Network uses layer2 information to detect MN’s movement. When network detects MN’s movement by receiving a beacon message, it sent RA message with MN’s previously used network prefix(i.e. Home Network Prefix). Even if MN receives the layer3 RA message, it could not detect its movement because of the same network prefix information. Instead of MN in MobileIP, in PMIPv6, the LMA supports the mobility. Whenever CN sends message to MN, the message is firstly transmitted to LMA. Then, LMA checks MN’s location and forwards the message. Problem and Solution From a privacy point of view, network based mobility support mechanism’s No IP change feature solves the privacy problem that existed in MIPv6. Even the PMIPv6 MN moves to other network, MN does not change its IP address. It maintains the same IP address in every network. i.e., network prefix of the IP address does not guarantee MN’s Location. However, in Identity Privacy viewpoint, MN uses the IID based IP address and inherits IP identity privacy problem described in 3.2. Figure 8 illustrates the privacy solutions for mobility support IP.

Fig. 8. Mobility Support IP Privacy Solutions

82

4

H.-J. Lim and T.-M. Chung

Transport Layer

At the transport layer, TCP/IP supports two protocols: TCP and UDP. Transport layer protocols provide process-to-process communication using port number. In the scope of VANET environment, existing transport protocols can be used. Additionally for a dynamic topology changing VANET environment, some sorts of protocol variants are required. For the privacy viewpoint, port number and sequence number field of TCP can be a privacy threatening factor with a low possibility. Port number is 16bit and it distinguishes a process within a host area. Figure 9 illustrates TCP/UP format and privacy threatening factors.

Fig. 9. TCP and UDP Format

Problem Both TCP and UDP use a random port number between 49,152 and 65,535 to send a message. Even the port number’s attributes are random, short-live, and not unique, it threats privacy by long term profiling. However, the privacy breach possibility is much lower than other layer’s risk. Also, TCP uses a sequence number to identify each byte of data and to support reliable transmission. Though the sequence number is not unique and is changed every time, it could be privacy threatening factor with the same reason as the port number. Solution Despite the port number and sequence number’s privacy threatening factors, its possibility is quite low. Also, transport layer information could be hidden from adversary with an encryption mechanism such as IPSec.

5

Application Layer

At the application layer, TCP/IP supports many protocols and services such as mail, file transfer, World Wide Web, and so on. This layer provides means for the user to access information on the network. Therefore, the application layer supports most kind of services for user’s needs. In the scope of VANET environment, the applications are divided into two categories: Comfort Applications and Safety application. For user’s connivance and application requirement, each application carries driver’s identification or location information. For the privacy viewpoint, user identity(ID) and application contents are privacy threatening factors.

A Survey on Privacy Problems and Solutions for VANET Based on Network Model

83

Problem Application layer ID and location information are closely related to user’s identity. Same as transport layer privacy solution, application layer could hide the information by IPSec or other security mechanism. However, ID could be required because encrypted ID may be revealed or plaintext ID may be required for retrieving the decryption key. With these ID, adversary could link the lower layer information to obtain the location. Solution The IETF Geopriv working group has researched on secure privacy information transmission protocol when service provider needs them[21]. Except Geopriv working group, most of researches focus on primitively block the privacy leak, especially for identity privacy. Common identity privacy protection solutions are divided into three categories. ─ Pseudonymization ─ Random silent period ─ Mix-zone. Beresford, A.R. proposes a concept of mix-zone for privacy protection [22]. Mixzones are anonymzed regions of the network wherein mobile nodes change their identifiers to obfuscate the relation between entering and exiting events. Huang, L. figures out mix-zone problem, weak to correlation attack [23]. The author also suggests a random silent period wherein mobile nodes turn off their transceivers and update their identifiers. In [24], Huang, L. further proposes arranging silent periods into cascades and to enhance anonymity, coordinating silent periods. In [25], Li, M. suggests a usercentric solution letting mobile nodes coordinate their silent periods and decide whether to change pseudonyms. In other hand, Gruteser, M.[4] proposes a network-centric solution in that direction. These the pervasive computing based privacy prevent concepts are extended to VANET environment. In [26], Raya, M. designs to use pseudonyms in VANET communication. In [27] [28], Gerlach, M. suggests using a mix-context instead of pseudonyms for better privacy protection. Buttyán, L. analyzes the performance evaluation for pseudonyms’ safety [29]. The analysis results that the effectiveness is limited as an adversary monitoring 50% of the intersections can successfully track 60% of the vehicles with very high probability because of the non-uniformity of the traffic. Depend on the pseudonyms changing subject, two researches are added networkcentric[30] and user-centric[31]. Similar to pervasive computing environment, pseudonyms changing solution is also weak in VANET environment. So [32],[33] adapted the concept of mix-zone to VANET. [33]’s basic idea is that vehicles do not transmit messages when their speed drops below a threshold, and they change pseudonym during each such silent period. Dahl, M. [34] analyzed the mix-zone in the context of VANET. Meanwhile, Sampigethaya, K.[35],[36] modifies silent period concept with group based message transmit. Figure 10 illustrates the privacy solutions for Application layer.

84

H.-J. Lim and T.-M. Chung

Fig. 10. Application Layer Privacy Solutions

6

Solution Analysis with Attack Models

To analyze privacy protect solution’s effectiveness in VANET, we define four attack models [5, 6, 32]: External attacker, Internal attacker, Correlational attacker, and Relational attacker. External attackers (ExtAtck) install its own radio receivers near the road network and passively eavesdrop vehicle safety messages. These attackers are a kind of sniffers that do not emit any signals, but only listen and localize vehicle s. The external attackers are strongest when they are densely scattered throughout wireless service areas, in which case they are capable of precisely locating a vehicle. Internal attackers (IntAtck) are network providers that must provide wireless services in addition to obtaining vehicle information. Although network providers themselves could be trustworthy, a provider may accidentally leak privacy-sensitive information. For example, wireless social communities (e.g., FON), or WiFi operators (e.g.,Google) provide low cost wireless internet connectivity via WiFi networks in cities. With minor software or hardware modifications, this infrastructure can eavesdrop. Correlational attackers (CorAtck) which utilize the correlation between the old and new identifier of the same vehicle, can defeat current protection methods. This attack assume that with enough temporal and spatial precision, it may be possible for an attacker to correlate two identifier that are sent separately from the same vehicle moving through space. Correlational attack works in the internal and external attack. Relational attackers (RelAtck) use other layer’s information of the same vehicle. Even a layer’s information is strictly protected by any means, these attackers utilize upper or lower layer information by mapping the two layer information coincidentally. For example if a vehicle changes MAC address periodically, with the unchanged IP address the attacker tracks the MAC address’s changes. Relational attacker works in the internal and external attack.

A Survey on Privacy Problems and Solutions for VANET Based on Network Model Solution Pseudonyms id [4] Silent period[5] Pseudonyms identifier [6] Slyfi[8] SLOW[33] SeVeCom[37] DHCP[9][18] NAT[10] Prefix Encryption [15] privacy extensions[16] CGA[17] eHoA&pHoA[20] Pseudonyms id[26] Mix-Context[27] User assigne[31] Network assigned[30] Group based[35][36] Mix-zone[32], [33]

*ɂ

85

ID.Pri

Loc.Pri

ExtAtck

IntAtck

CorAtck

RelAtck

ȿ

¯

ȿ

ȿ

¯

¯

ȿ

¯

ȿ

ȿ

ȿ

¯

ȿ

ȿ

ȿ

ȿ

ȿ

¯

ȿ

¯

ȿ

¯

˰

೗

ȿ

¯

ȿ

ȿ

ȿ

¯

ȿ

ȿ

ȿ

¯

ȿ

ȿ

ȿ

೗

ȿ

¯

¯

¯

ȿ

ȿ

ȿ

¯

ȿ

¯

¯

ȿ

ȿ

¯

˰

¯

ȿ

¯

ȿ

ȿ

˰

¯

ȿ

¯

ȿ

ȿ

˰

¯

ȿ

ȿ

ȿ

೗

˰

¯

ȿ

¯

ȿ

ȿ

¯

¯

ȿ

¯

ȿ

ȿ

˰

¯

ȿ

¯

ȿ

ȿ

¯

¯

ȿ

¯

ȿ

¯

ȿ

¯

ȿ

೗

ȿ

ȿ

ȿ

¯

ȿ

¯

ȿ

ȿ

ȿ

¯

6DWLVI\LQJ೚ &RQGLWLRQDO6DWLVI\LQJ¯ 1RW6DWLVI\LQJ˰GdGu–›GyŒ“ˆ›Œ‹

According to our research, most of solutions are focus on the identity privacy. For the external attack, all of the solutions protect the privacy treats. The internal attack is also successfully protects privacy treats except the network based solutions and key exchange based solutions. In the case of correlation attack, the analysis is limitedly progressed only for the periodic identity change solutions. The early solutions are weak to correlation attack. However with the silent period and mix-zone concept, correlation attacks are dramatically reduced. Though the achieved solution development, most of all solution is dangerous to relational attack. We attention that the VANET privacy is focus on identity protect solutions and weak to relational attack. The reason of having the former feature is that the owner's location is more important than the vehicle's location. i.e., the real risk factor is ‘who’ is in somewhere, not someone is in ‘specific place’. The latter feature is due to the

86

H.-J. Lim and T.-M. Chung

network model’s transparency design goal. This design goal is advantage to support variable protocols for each layer without dependency problem. However, disadvantage to VANET privacy protection.

7

Conclusion

In our paper, we investigate privacy treating factors and its known solutions based on network model in VANET. Most of the solutions use pseudonym identity or address changing scheme to protect identity privacy. Additionally, known solutions are weak to or do not consider the relational attack. We interpret this is due the meet the network model’s transparency design goal and protect vehicle’s real identity even revealing the vehicle’s location. However, these approaches are not the ultimately protect privacy problem. Also if the attacker is interested in the location of the vehicle, it is potential treat to VANET environment. For example, a thief may analyze the incoming packets to police station and knows the patrol car’s location. Therefore in our future works, we plan to design a multi-layered privacy protection mechanism for VANET. Acknowledgments. This work (Grants No. 00044301) was supported by Business for Cooperative R&D between Industry, Academy, and Research Institute funded Korea Small and Medium Business Administration in 2010.

References 1. Kent, S.T., Millett, L.I.: IDs–not that easy: Questions about nationwide identity systems. Natl. Academy Pr., Washington DC (2002) 2. Beresford, A.R., Stajano, F.: Location Privacy in Pervasive Computing. IEEE Pervasive Computing 2, 46–55 (2005) 3. Fuentes, J.M., González-Tablas, A.I., Ribagorda, A.: Overview of Security Issues in Vehicular Ad-Hoc Networks (2010) 4. Gruteser, M., Grunwald, D.: Enhancing Location Privacy in Wireless LAN through Disposable Interface Identifiers: A Quantitative Analysis. Mobile Networks and Applications 10, 315–325 (2005) 5. Huang, L., Matsuura, K., Yamane, H., Sezaki, K.: Enhancing wireless location privacy using silent period. In: IEEE Wireless Communications and Networking Conference, 2005, vol. 2, pp. 1187–1192. IEEE, Los Alamitos (2005) 6. Jiang, T., Wang, H.J., Hu, Y.C.: Preserving location privacy in wireless LANs. In: Proceedings of the 5th International Conference on Mobile Systems, Applications and Services, pp. 246–257. ACM, New York (2007) 7. Pang, J., Greenstein, B., Gummadi, R., Seshan, S., Wetherall, D.: 802.11 user fingerprinting. In: Proceedings of the 13th Annual ACM International Conference on Mobile Computing and Networking, pp. 99–110. ACM, New York (2007) 8. Greenstein, B., McCoy, D., Pang, J., Kohno, T., Seshan, S., Wetherall, D.: Improving wireless privacy with an identifier-free link layer protocol. In: Proceeding of the 6th International Conference on Mobile Systems, Applications, and Services, pp. 17–20. Citeseer (June 2008) 9. Droms, R.: Dynamic host configuration protocol (1997)

A Survey on Privacy Problems and Solutions for VANET Based on Network Model

87

10. Srisuresh, P., Holdrege, M.: RFC 2663. IP Network Address Translator (NAT) Terminology and Considerations (1999) 11. Thomson, S., Narten, T., Jinmei, T.: RFC4862: IPv6 Stateless Address Autoconfiguration. Standards Track, http://www.ietf.org/rfc/rfc4862.txt AUTOCONFStating the Problem, 31 12. Haddad, W., Nordmark, E., Dupontand, F., Bagnulo, M., Park, S., Patil, B.: Privacy for Mobile and Multi-homed Nodes: MoMiPriv Problem Statement (2005) 13. Groat, S., Dunlop, M., Marchany, R., Tront, J.: The privacy implications of stateless IPv6 addressing. In: Proceedings of the Sixth Annual Workshop on Cyber Security and Information Intelligence Research, pp. 1–4. ACM, New York (2010) 14. Lindqvist, J.: IPv6 is Bad for Your Privacy (2007) 15. Trostle, J., Matsuoka, H., Tariq, M.M.B., Kempf, J., Kawahara, T., Jain, R.: Cryptographically protected prefixes for location privacy in ipv6. In: Martin, D., Serjantov, A. (eds.) PET 2004. LNCS, vol. 3424, pp. 142–166. Springer, Heidelberg (2005) 16. Narten, T., Draves, R., Krishnan, S.: RFC 4941-Privacy Extensions for Stateless Address Autoconfiguration in IPv6. IETF (September 2007) 17. Nikander, P., Arkko, J., Kempf, J., Zill, B.: SEcure Neighbor Discovery (SEND) 18. Droms, R., Bound, J., Volz, B., Lemon, T., Perkins, C., Carney, M.: Dynamic host configuration protocol for IPv6 (DHCPv6) (2003) 19. Koodli, R.: RFC 4882: IP Address Location Privacy and Mobile IPv6: Problem Statement (2007) 20. Qiu, Y.: RFC 5726 : Mobile IPv6 Location Privacy Solutions (2010) 21. Geographic Location/Privacy (geopriv), http://datatracker.ietf.org/wg/geopriv/ 22. Beresford, A.R., Stajano, F.: Mix zones: User privacy in location-aware services. In: Proceedings of the Second IEEE Annual Conference on Pervasive Computing and Communications Workshops, 2004, pp. 127–131. IEEE, Los Alamitos (2004) 23. Huang, L., Yamane, H., Matsuura, K., Sezaki, K.: Towards modeling wireless location privacy. In: Danezis, G., Martin, D. (eds.) PET 2005. LNCS, vol. 3856, pp. 59–77. Springer, Heidelberg (2006) 24. Huang, L., Yamane, H., Matsuura, K., Sezaki, K.: Silent Cascade: Enhancing Location Privacy without Communication Qos Degradation. In: Clark, J.A., Paige, R.F., Polack, F.A.C., Brooke, P.J. (eds.) SPC 2006. LNCS, vol. 3934, pp. 165–180. Springer, Heidelberg (2006) 25. Li, M., Sampigethaya, K., Huang, L., Poovendran, R.: Swing & swap: user-centric approaches towards maximizing location privacy. In: Proceedings of the 5th ACM Workshop on Privacy in Electronic Society, pp. 19–28. ACM, New York (2006) 26. Raya, M., Hubaux, J.P.: The security of vehicular ad hoc networks. In: Proceedings of the 3rd ACM Workshop on Security of Ad Hoc and Sensor Networks, pp. 11–21. ACM, New York (2005) 27. Gerlach, M., Guttler, F.: Privacy in VANETs using changing pseudonyms-ideal and real. In: IEEE 65th Vehicular Technology Conference, VTC 2007-Spring, pp. 2521–2525. IEEE, Los Alamitos (2007) 28. Gerlach, M.: Assessing and Improving Privacy in VANETs. ESCAR, Embedded Security in Cars (2006) 29. Buttyán, L., Holczer, T., Vajda, I.: On the Effectiveness of Changing Pseudonyms to Provide Location Privacy in VANETs. In: Stajano, F., Meadows, C., Capkun, S., Moore, T. (eds.) ESAS 2007. LNCS, vol. 4572, pp. 129–141. Springer, Heidelberg (2007)

88

H.-J. Lim and T.-M. Chung

30. Dötzer, F.: Privacy issues in vehicular ad hoc networks. In: Danezis, G., Martin, D. (eds.) PET 2005. LNCS, vol. 3856, pp. 197–209. Springer, Heidelberg (2006) 31. Golle, P., Greene, D., Staddon, J.: Detecting and correcting malicious data in VANETs. In: Proceedings of the 1st ACM International Workshop on Vehicular Ad Hoc Networks, pp. 29–37. ACM, New York (2004) 32. Freudiger, J., Raya, M., Felegyhazi, M., Papadimitratos, P., Hubaux, J.P.: Mix-zones for location privacy in vehicular networks. In: Proceedings of the 1st International Workshop on Wireless Networking for Intelligent Transportation Systems (WiN-ITS 2007) (2007) 33. Buttyán, L., Holczer, T., Weimerskirch, A., Whyte, W.: Slow: A practical pseudonym changing scheme for location privacy in vanets. In: 2009 IEEE Vehicular Networking Conference (VNC), pp. 1–8. IEEE, Los Alamitos (2010) 34. Dahl, M., Delaune, S., Steel, G.: Formal Analysis of Privacy for Vehicular Mix-Zones. In: Gritzalis, D., Preneel, B., Theoharidou, M. (eds.) ESORICS 2010. LNCS, vol. 6345, pp. 55–70. Springer, Heidelberg (2010) 35. Poovendran, R., Sampigethaya, K., Huang, L., Li, M., Matsuura, K., Sezaki, K.: CARAVAN: Providing Location Privacy for VANET (2005) 36. Sampigethaya, K., Li, M., Huang, L., Poovendran, R.: Amoeba: Robust Location Privacy Scheme for Vanet. IEEE Journal on Selected Areas in Communications 25, 1569–1589 (2007) 37. Papadimitratos, P., Buttyan, L., Hubaux, J.P., Kargl, F., Kung, A., Raya, M.: Architecture for secure and private vehicular communications. In: 7th International Conference on ITS Telecommunications, ITST 2007, pp. 1–6. IEEE, Los Alamitos (2007)

Scheduling Tasks and Communications on a Hierarchical System with Message Contention Jean-Yves Colin and Moustafa Nakechbandi LITIS, Université du Havre, IUT, 76610 Le Havre, France {jean-yves.colin,moustafa.nakechbandi}@univ-lehavre.fr

Abstract . A Directed Acyclic Graph (DAG) of tasks with small communication delays has to be scheduled on the identical parallel processors of clusters connected by a hierarchical network. The number or processors and of clusters is not limited. Message contention has to be avoided. Task duplication is allowed. In this paper, we present a new polynomial algorithm that computes the earliest start dates of all tasks and spreads these tasks to use few processors per cluster, for a DAG with small communication delays. It also avoids message contention, and always delivers messages on time. Keywords: Scheduling, DAG, Hierarchical Communications, Message contention, Task Duplication, CPM/PERT.

1

Introduction

The efficient use of distributed memory multiprocessors and grids is a very difficult problem. An application is made of different parts, with specific processing times and communication delays, that need to be scheduled carefully. Examples of applications include numerical analysis applications, logistics systems based on heterogeneous distributed computing systems, high performance Data Mining systems, and Automated Document Factories in banking environments. In the classical scheduling problem with communication delays, a positive processing time is associated to each task of a Directed Acyclic Graph (DAG) and a positive communication delay is associated to each precedence constraint between the tasks of this DAG. The tasks then have to be scheduled on the processors of the distributed memory multiprocessor or grid. This problem is known to be NP-hard in the general case even if the number of available processors is not limited [8]. Many studies are currently available on several aspects of this classical scheduling problem [3] [4] [5] [9] [13] [17] [18]. Task duplication, for example, is used in several studies to lower the communication overheads by executing identical copies of some of the tasks on different processors [1] [4] [5] [12]. 

This work is partially funded by the GRR "Transport Logistique et Technologie de l’Information" of the Université du Havre, France.

Y. Xiang et al. (Eds.): ICA3PP 2011 Workshops, Part II, LNCS 7017, pp. 89–98, 2011. c Springer-Verlag Berlin Heidelberg 2011 

90

J.-Y. Colin and M. Nakechbandi

Hierarchical communications are taken into account in some studies too. The processors are typically grouped into clusters, with communication between processors of the same cluster being faster than communications between processors of different clusters [1] [7]. These problems are increasingly recognized to be unrealistic however, because they do not consider message contention [2] [11] [14] [21]. In [15] for example, the authors show the NP-Completeness of the two processor scheduling problem with tasks of execution time 1 or 2 units, unit interprocessor communication latency and message contention. In [6], a CPM/PERT-like polynomial scheduling algorithm for DAG with small communication delays and task duplication is proposed. It is optimal and always avoids message contention, if resources are not limited. It does not consider hierarchical communications, however. More recent studies use heuristics to avoid message contention, and present extensive experimental evaluations to evaluate performance improvements [19] [20]. In this paper, we present a new polynomial algorithm for DAG with small communication delays. The distributed architecture is made of clusters, has a two level communication network and has communication channels that can transmit at most one message at any time. The algorithm computes, if resources are not limited, the earliest start dates of all tasks and spreads these tasks to use few processors per cluster. It also schedules the communications so that message contention is avoided, and always delivers messages on time.

2 2.1

The 2lVds Problem The 2lVds Model

A 2-levels Virtual Distributed System architecture (2lVds) is a distributed memory multi-processor architecture (or grid) with a not limited number of homogeneous processors. The processors are grouped into clusters. Both the number of clusters and the number of processors in each cluster are not limited. Each processor belongs to one and only one cluster (Fig. 1). There is a complete communication network between all the processors. Each direct connection between any two processors is made of two unidirectional channels, one in each direction. All communications channels inside all clusters are cluster

cluster

slow communication channels

fast communication channels

cluster

processor

Fig. 1. A 2lVds architecture

Scheduling Tasks and Communications on a Hierarchical System

91

identical and all communications channels between processors of different clusters are identical too, but slower than the intra-cluster channels. Each unidirectional channel may carry at most one message at any time. An application is represented by a DAG G = (V, E) (or precedence graph) where V designates the set of tasks, and E the set of precedence constraints. Formally, a 2lVds scheduling problem may then be specified by the four parameters V, E, p, c, in which V = {1, 2, ...n} is the set of n tasks, E is the set of arcs (i, j) with (i, j) ∈ E representing a precedence constraint from task i ∈ V to task j ∈ V , p is the set of processing times with pi ∈ p being the processing time of task i ∈ V on any processor π of the 2lVds architecture, and c is the set of communications delays. To each arc (i, j) ∈ E are associated two values ci,j (1) ∈ c and ci,j (2) ∈ c. ci,j (1) is the positive communication delay of a message from task i to task j, if i and j are executed on different processors inside the same cluster (intra-cluster communication delay). ci,j (2) is the positive communication delay of a message from task i to task j, if i and j are executed in different clusters (inter-cluster communication delay), with ci,j (1) ≤ ci,j (2). If two communicating tasks i and j are executed on the same processor, there is no need for any communication or its duration is considered negligible, so the communication delay is then 0. A task is indivisible, starts when all the data it needs from its predecessors are available, and sends all the data needed by its successors at the end of its execution. All the immediate successors of a task use the same result from this task. This assumption implies that a task needs to send one message only to a given processor, even if several of its successors are to be processed on it, because one message is enough for all. If it does not hold, the task may usually be divided into sub-tasks such that the assumption is satisfied. Fig. 2 presents an example of such a DAG. The value above each node is its processing time, and the two values above each arc are its two communication delays. Task duplication is allowed. That is, several instances (or copies) of the same task may be executed on different processors. We will denote ik the k th copy of task i. Because we must take into account the messages in a schedule, we will denote m(ik , jl ) a message sent from a copy ik of task i to a copy jl of task j. 2 2 2 2 1 1,2 2 1,2 3 1,2 4 1,2 1,2 1,2 2 2 2 2 1,2 1,2 1,2 5 6 7 8 1,2 1,2 1,2 9 2 2 2 9 1,2 10 1,2 11 1,2 12 1,2 1,2 1,2 2 2 2 2 13 1,2 14 1,2 15 1,2 16 Fig. 2. Example of a DAG with two communication delays

92

J.-Y. Colin and M. Nakechbandi

A schedule S of a 2lVds scheduling problem is then a 5-tuple (F, tc , π, M, t ), where m

F (i) is the positive number of copies of task i ∈ V , tc (ik ) is the starting time of copy ik of task i, 0 < k ≤ F (i), π(ik ) is the processor assigned to copy ik of task i, 0 < k ≤ F (i), M (i, j) is the set of all messages sent by copies of task i to copies of task j, tm (m(ik , jl )) is the starting time of message m(ik , jl ) ∈ M (i, j). First, to be feasible, a schedule S must satisfy the following conditions: – at least one copy of each task is processed, i.e. ∀i ∈ V , F (i) > 0, – at any time, a processor executes at most one copy, – for each (i, j) ∈ E, for any copy jl of j, there is one copy ik of i that is on the same processor or that sends its message on time to jl , i.e. if π(jl ) = π(ik ) then tc (jl ) ≥ tc (ik ) + pi else if π(jl ) and π(ik ) are in the same cluster then tc (jl ) ≥ tc (ik ) + pi + ci,j (1) else tc (jl ) ≥ tc (ik ) + pi + ci,j (2) end if If, in a schedule S, ik and jl satisfy the above condition, we will say that the Generalized Precedence Constraint is true for the two copies (in short, that GP C(ik , jl ) is true). Second, a feasible schedule S must additionally satisfy the condition that there is no message contention, i.e. in all channels used to transmit at least two messages m(ik , jl ) and m(rt , sq ) from a processor πik to a processor πjl , with message m(ik , jl ) finishing before message m(rt , sq ), we have if if π(jl ) and π(ik ) are in the same cluster then tm (m(rt , sq )) ≥ tm (m(ik , jl )) + ci,j (1) else tm (m(rt , sq )) ≥ tm (m(ik , jl )) + ci,j (2) end if Now, let C(ik ) be the completion time of a copy ik of a task i, i.e. C(ik ) = tc (ik ) + pi . The maximum completion time, or makespan, Cmax of a solution S is the largest completion time of all copies of all tasks in this solution: Cmax =

max

i∈V,k≤F (i)

{tc (ik ) + pi } .

(1)

As usual for this kind of problem, we want to minimize Cmax , that is, find a ∗ . feasible solution S ∗ with the smallest makespan Cmax One can note that, if ci,j (1) = ci,j (2), this scheduling problem is actually equivalent to the classical DAG scheduling problem with communication delays

Scheduling Tasks and Communications on a Hierarchical System

93

which, in the general case, is a NP-hard problem, even if the number of processors is not limited [16]. For this reason, we will only consider a DAG satisfying the conditions in the following two equations. They guarantee that the DAG has small communication delays. We will denote PRED (i) (respectively SUCC (i)) the set of immediate predecessors (resp. successors) of task i in G. ∀i ∈ V,

min

g∈P RED(i)

pg ≥

max

h∈P RED(i)−{g}

ch,i (1) .

(2)

Equation (2) means that processing times are locally superior or equal to the communication delays inside the clusters. It ensures that the earliest start date of any copy of each task may be computed in polynomial time. ∀i ∈ V,

min

k∈SUCC(i)

pk ≥

max

j∈SUCC(i)−{k}

ci,j (2) .

(3)

Equation (3) is very similar to (2). It means too that the processing times are locally superior or equal to the communication delays between the clusters. However, (2) deals with the predecessors of a task and with the intra-clusters communication delays, while (3) deals with the successors, and with the interclusters communication delays. Also (2) is true in most cases if (3) is true. One can note that there is already a trivial solution to the 2lVds problem: use one cluster only, and schedule all tasks on the processors of this cluster using the algorithm in [6]. This trivial solution, however, is not helpful at all, because real architectures have a limited number of processors in each cluster. For this reason, we propose the following new algorithm 2lVdsOpt. It schedules the tasks and communications in a 2lVds problem in polynomial time and spreads the tasks on as many clusters as possible to use less processors per cluster. 2.2

The 2lVdsOpt Algorithm

This algorithm has four steps. The first step 2lVdsLwb() computes the earliest start date of all copies of each task of the DAG. The second step 2lVdsCs() computes the critical sequences of the DAG according to the earliest start dates calculated during the first step. The third step 2lVdsCc() computes the graph of the critical sequences of the DAG, and its connected components according to the communication delays ci,j (1). The last step 2lVdsBuild() computes the solution, scheduling the tasks and communication on the 2lVds architecture. Computing the Earliest Start Dates. The first step of 2lVdsOpt computes the earliest start date bi of all copies of each task i of the DAG. This is done in procedure 2lVdsLwb() (cf. Algorithm 1). Table 1 presents the earliest start dates of each task of the DAG of Fig. 2 computed by procedure 2lVdsLwb(). Computing the Critical Sequences. The second step of 2lVdsOpt computes the critical sequences resulting from the earliest start dates calculated during step 1. Let B be the set of the earliest start dates bi of all tasks of V .

94

J.-Y. Colin and M. Nakechbandi

Algorithm 1. procedure 2lVdsLwb(V , E, p, c) for all tasks i ∈ V such that P RED(i) = ∅ do let bi = 0 {assign 0 to i as its earliest start date bi } end for while there is a task i which has not been assigned an earliest starting date bi and whose predecessors h ∈ P RED(i) all have an earliest starting date bh assigned to them do let c = maxh∈P RED(i) bh + ph + ch,i (1) find g ∈ P RED(i) such that bh + ph + ch,i (1) = c   let bi = max bg + pg , maxh∈P RED(i)−{g} bh + ph + ch,i (1) end while Table 1. Earliest start dates bi of the tasks i of the DAG of Fig. 2 computed by procedure 2lVdsLwb() (cf. Algorithm 1) task i:

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

bi :

0

2

4

6

4

7

9

11

0

9

11

13

11

14

16

18

Let GC be the critical subgraph of G according to the earliest start dates in B. (i, j) is an arc of GC if (i, j) ∈ E and bj < bi + pi + ci,j (1). That is, an arc (i, j) in GC means that these two tasks must have copies on the same processor, because there is not enough delay to transmit the result of any copy ik to a copy jl from one processor to another processor of the same cluster. GC is always a forest [5]. A critical sequence sc of the DAG is a proper path of GC. The computation is done in procedure 2lVdsCs() (cf. Algorithm 2). Computing the Graph of the Critical Sequences. The third step of 2lVdsOpt builds the undirected graph GSC of the critical sequences scs and computes its connected components [10]. Let CC be the set of all computed critical sequences scs . Algorithm 2. procedure 2lVdsCs(V , E, p, c, B) GC = ∅ for all arcs (i, j) ∈ E do if bj < bi + pi + ci,j (1) then GC = GC ∪ {(i, j)} end if end for s=0 for all tasks i ∈ V do if task i is a leaf of the critical subgraph GC then let critical sequence scs be the path from the root of the tree in GC that includes task i, to task i s=s+1 end if end for

Scheduling Tasks and Communications on a Hierarchical System

95

GSC has one node ss for each critical sequence scs of CC computed during the previous step. Also, there is one edge (ss , st ) or (st , ss ) in GSC if ∃(i, j) ∈ E, with i ∈ scs , and i ∈ / sct , and j ∈ sct , such that bj < bi + pi + ci,j (2). This edge means that there is not enough time to transmit one message between at least one task i of scs to another task j of sct between two clusters. So scs and sct must be processed in the same cluster. The computation is done in procedure 2lVdsCc() (cf. Algorithm 3). Algorithm 3. procedure 2lVdsCc(V , E, p, c, B, CC) GSC = ∅ for all critical sequences scs ∈ CC do let ss be the new node related to scs end for for all nodes ss do GSC = GSC ∪ {ss } for all nodes st ∈ GSC − {ss } do if there is no edge between ss and st in GSC and there is at least one arc (i, j) / sct and j ∈ sct , such that bi < bi + pi + ci,j (2) then of E with i ∈ scs and i ∈ add one edge between ss and st to GSC end if end for end for compute the connected components gs of GSC

Fig. 3 shows the six critical sequences sc1 to sc6 found for the DAG of Fig. 2 using the computed earliest start dates in Table 1. It also shows the graph of the critical sequences and its two connected components. Computing the Solution. The last step of 2lVdsOpt builds a solution with minimal makespan using all the data computed in the preceding phases. One cluster is allocated to each connected component, and one processor of this cluster is allocated to each critical sequence of this connected component. One copy of each task of each critical sequence is executed at its earliest start date. All messages are sent as soon as the sending copy of the task finishes its execution. The computation is done in procedure 2lVdsBuild() (cf. Algorithm 4). Fig. 4 shows the Gantt chart of the final schedule found for the DAG of Fig. 2. Two clusters, each with three processors, are used. Tasks 1, 2, 9 and 10 have two copies each in this schedule. 2.3

Analysis of the Algorithm

Let n be the number of tasks and m be the number of arcs. The complexity of procedure 2lVdsLwb() is O(max(m, n)), and the complexity of procedure 2lVdsCs() is O(m). The complexity of building the graph of the critical sequences in 2lVdsCc() is O(n) [5], and of computing its connected components is O(n). Thus the complexity of 2lVdsCc() is O(n) too.

96

J.-Y. Colin and M. Nakechbandi

1

sc2

2

5

6

9

10

13

sc5

14

sc1

3 7

sc4

sc3

11 15

s1

4 s2

s3

8 s4

12 sc6

s5 16

s6

Fig. 3. The six critical sequences sc1 to sc6 in the critical graph GC of the DAG in Fig. 2 (left), and the graph GSC of these critical sequences with the two resulting connected components (right)

Algorithm 4. procedure 2lVdsBuild(V , E, p, c, B, CC, GSC) for all connected components gc ∈ GSC do allocate a new cluster Πc to gc for all node ss ∈ gc do let scs be the critical sequence related to node ss allocate a new processor πs in cluster Πc to this critical sequence scs for all task i ∈ scs do F (i) = F (i) + 1, tc (iF (i) ) = bi , π(iF (i) ) = πs end for end for end for for all copy jl of task j do let π(jl ) be the processor that executes jl for all task i ∈ P RED(j) do if there is no copy of task i on π(jl ) and π(jl ) does not already receive one message from any copy of i on time for copy jl then remove any message m from any copy of i to processor π(jl ) find one copy ik that can send its message on time to jl send one message m(ik , jl ) from copy ik at date bi + pi to processor π(jl ) end if end for end for

π1 11 21 31 41 cluster Π1 π2 12 22 51 π3 61 π4 91 cluster 92 Π 2 π5 π6 0 2 4 6 8

71 81 101 111 121 102 131 141 151 161 time 10 12 14 16 18 20

Fig. 4. Gantt chart of the solution of the DAG of Fig. 2, using two clusters Π1 and Π2 , with three processors per cluster (π1 , π2 and π3 in Π1 , and π4 , π5 and π6 in Π2 )

Scheduling Tasks and Communications on a Hierarchical System

97

Using a graph-level approach, one can show that the complexity of the first part of 2lVdsBuild() is O(n2 ). Because the second part of 2lVdsBuild() tries, in the worst case, to find one suitable copy of each predecessor for each copy of each task, it is possible to establish that the complexity of this second part is O(m2 n2 ). The complexity of procedure 2lVdsBuild() is then O(m2 n2 ). So the complexity of the overall algorithm is O(m2 n2 ). Also, we have the following theorems. Theorem 1. The solution built by 2lVdsOpt has minimal makespan. Theorem 2. At least one copy of each task is executed. Theorem 3. The GPC are true for all copies of all tasks. Theorem 4. In the solution computed, each copy of each task receives at least one message on time from at least one copy of each of its predecessor, if a message is needed. Theorem 5. There is no message contention on any unidirectional channel.

3

Conclusion

A Directed Acyclic Graph of tasks with small communication delays had to be scheduled on the identical parallel processors of several clusters connected by a hierarchical network. The number of processors and of clusters was not limited. Message contention had to be avoided. Task duplication was allowed. We presented a new polynomial algorithm that computes the earliest start dates of tasks and spreads these tasks to use few processors per cluster. It also schedules the communications so that there is no message contention and messages are always delivered on time.

References 1. Bampis, E., Giroudeau, R., König, J.-C.: Using Duplication for Multiprocessor Scheduling Problem with Hierarchical Communications. Parallel Processing Letters 10(1), 133–140 (2000) 2. Beaumont, O., Boudet, V., Robert, Y.: A Realistic Model and an Efficient Heuristic for Scheduling with Heterogeneous Processors. In: 11th Heterogeneous Computing Workshop (HCW 2002). IEEE Computer Society Press, Los Alamitos (2002) 3. Bittencourt, L.F., Sakellariou, R., Madeira, E.R.M.: DAG Scheduling Using a Lookahead Variant of the Heterogeneous Earliest Finish Time Algorithm. In: 18th Euromicro International Conference on Parallel, Distributed and Network-Based Computing (PDP 2010), Pisa, Italy (2010) 4. Bozdag, D., Ozguner, F., Catalyurek, U.V.: Compaction of Schedules and a Two Stage Approach for Duplication-Based DAG Scheduling. IEEE Transactions on Parallel and Distributed Systems 20(6), 857–871 (2009) 5. Colin, J.-Y., Chrétienne, P.: Scheduling with Small Communication Delays and Task Duplication. Operations Research 39(4), 680–684 (1991)

98

J.-Y. Colin and M. Nakechbandi

6. Colin, J.-Y., Colin, P.: Scheduling Tasks and Communications on a Virtual Distributed System. European Journal of Operational Research 94(2) (1996) 7. Colin, J.-Y., Nakechbandi, M.: Scheduling Tasks with Communication Delays on 2Levels Virtual Distributed Systems. In: Proceedings of the 7th Euromicro Workshop on Parallel and Distributed Processing (PDP 1999), Funchal, Portugal (1999) 8. Garey, M., Johnson, D.: Computers and Intractability, a Guide to the Theory of NP-Completeness. Freeman, New York (1979) 9. Giroudeau, R., König, J.-C.: Scheduling with Communication Delay. In: Multiprocessor Scheduling: Theory and Applications, pp. 1–26. ARS Publishing (2007) 10. Hopcroft, J., Tarjan, R.: Efficient Algorithms for Graph Manipulation. Communications of the ACM 16, 372–378 (1973) 11. Kalinowski, T., Kort, I., Trystram, D.: List Scheduling of General Task Graphs under LogP. Parallel Computing 26, 1109–1128 (2000) 12. Kruatrachue, B., Lewis, T.G.: Grain Size Determination for Parallel Processing. IEEE Software 5(1), 23–32 (1988) 13. Kwok, Y.-K., Ahmad, I.: Static Scheduling Algorithms for Allocating Directed Task Graphs to Multi-Processors. ACM Computing Surveys (CSUR) 31(4), 406– 471 (1999) 14. Marchal, L., Rehn, V., Robert, Y., Vivien, F.: Scheduling Algorithms for Data Redistribution and Load-Balancing on Master-Slave Platforms. Parallel Processing Letters 17(1), 61–77 (2007) 15. Norman, M.G., Pelagatti, S., Thanisch, P.: On the Complexity of Scheduling with Communication Delay and Contention. Parallel Processing Letters 5(3), 331–341 (1995) 16. Papadimitriou, C.B., Yannakakis, M.: Toward an Architecture Independent Analysis of Parallel Algorithms. In: Proceedings of the 20th Annual ACM Symposium Theory of Computing, Santa Clara, California, USA (1988) 17. Rayward-Smith, V.J.: Scheduling with Unit Interprocessor Communication Delays. Discrete Math. 18, 55–71 (1987) 18. Sarkar, V.: Partitioning and Scheduling Parallel Programs for Execution on Multiprocessors. MIT Press, Cambridge (1989) 19. Sinnen, O., Sousa, L.: Communication Contention in Task Scheduling. IEEE Transactions on Parallel and Distributed Systems 16(6), 503–515 (2005) 20. Sinnen, O., To, A., Kaur, M.: Contention-Aware Scheduling with Task Duplication. Journal of Parallel and Distributed Computing 71(1), 77–86 (2011) 21. Tam, A., Wang, C.L.: Contention-Aware Communication Schedule for High Speed Communication. Cluster Computing 6(4), 339–353 (2003)

Spiking Neural P System Simulations on a High Performance GPU Platform Francis George Cabarle1 , Henry Adorna1 , Miguel A. Mart´ınez–del–Amor2, and Mario J. P´erez–Jim´enez2 1

2

Algorithms & Complexity Lab, Department of Computer Science, University of the Philippines Diliman Diliman 1101 Quezon City, Philippines [email protected], [email protected] Research Group on Natural Computing, Department of Computer Science and Artificial Intelligence, University of Seville, Avda. Reina Mercedes s/n, 41012 Sevilla, Spain {mdelamor,marper}@us.es

Abstract. In this paper we present our results in adapting a Spiking Neural P system (SNP system) simulator to a high performance graphics processing unit (GPU) platform. In particular, we extend our simulations to larger and more complex SNP systems using an NVIDIA Tesla C1060 GPU. The C1060 is manufactured for high performance computing and massively parallel computations, matching the maximally parallel nature of SNP systems. Using our GPU accelerated simulations we present speedups of around 200× for some SNP systems, compared to CPU only simulations. Keywords: Membrane computing, Spiking Neural P systems, GPU computing, CUDA, parallel computing.

1

Introduction

P systems are by nature distributed, parallel, and non-deterministic computing models defined within Membrane computing, which is a research area initiated by Gheorghe P˘ aun in 1998 [16]. The objective, as with other disciplines of Natural computing (e.g. DNA/molecular computing, quantum computing, etc.), is to obtain inspiration from the way nature computes to provide efficient solutions to the limitations of conventional models of computation e.g. a Turing machine. Membrane computing can be thought of as an extension of DNA or molecular computing, zooming out from the individual molecules of the DNA and including other parts and sections of living cells in the computation, introducing the concept of distributed computing as well [16]. P systems are abstractions of the compartmentalized structure and parallel processing of biochemical information in biological cells. There are several P sytem variants defined in literature, each one based on the abstraction of different aspects (or ingredients) from cells, and that many of them have been proven Y. Xiang et al. (Eds.): ICA3PP 2011 Workshops, Part II, LNCS 7017, pp. 99–108, 2011. c Springer-Verlag Berlin Heidelberg 2011 

100

F.G. Cabarle et al.

to be computationally complete [5]. There are three general classifications of P systems considering the level of abstraction: cell-like (a rooted tree where the skin or outermost cell membrane is the root), tissue-like (a graph connecting the cell membranes) and neural-like (a directed graph, inspired by neurons interconnected by their axons and synapses). The last type refer to Spiking Neural P systems (in short, SNP systems), where the time difference (when neurons fire and/or spike) plays an essential role in the computations [11]. An interesting result of P systems is that they are able to solve computationally hard problems (e.g. NP-complete problems) usually in polynomial, often linear time, but usually requiring exponential space as trade off [16]. Due to the nature of P systems, they are yet to be fully implemented in vivo, in vitro, or even in silico. Thus, practical computations of P systems are driven by silicon-based simulators. There are several simulators for P systems implemented over different software and hardware technologies [7]. In practice, P system simulations are limited by the physical laws of silicon architectures, which are often inefficient or not suitable when dealing with P system features, such as massive parallelism. However, in order to improve the efficiency of the simulators, it is necessary to exploit current technologies, leading to solutions in the area of High Performance Computing (HPC), such as accelerators or manycore processors. In this respect, Graphics Processing Units (GPUs) have been consolidated as accelerators thanks to their throughput-oriented and highlyparallel architecture [9]. Several simulators for P systems have been developed over highly parallel platforms, including reconfigurable hardware as in FPGAs [14], CPU-based clusters [6], as well as in NVIDIA corporation’s Compute Unified Device Architecture (CUDA) enabled GPUs [4,3]. These efforts show that parallel devices are very suitable in accelerating the simulation of P systems, at least for transition and active membrane P systems [3,4]. Efficiently simulating a Spiking Neural P (SNP) system, the P system variant of interest in this work, would thus require new efforts in parallel computing. Since SNP systems have already been represented as matrices due to their graph-like properties [18], simulating them in parallel devices such as GPUs is the next natural step. Matrix algorithms are well known in parallel computing literature, including GPUs [8], due to the highly parallelizable nature of linear algebra computations mapping directly to the data-parallel GPU architecture. An SNP system simulator using CUDA was presented in [1] and [2]. These previous works however were executed in GPUs of workstations only, hence we intend to do better. We adapt and analyse the performance of this simulator on a high-end GPU NVIDIA Tesla C1060, designed ground-up for parallel computing and HPC, by simulating SNP systems of different sizes. A final simulator for SNP systems using CUDA would allow the designers to check their models, and perform other complex computations such as computing backwards. This paper is organized as follows: Section 2 and 3 provide backgrounds for CUDA and SNP systems, respectively. The design of the simulator and simulation results are given in Section 4 and 5, respectively.

Spiking Neural P System Simulations on a High Performance GPU Platform

2

101

GPU Computing and NVIDIA CUDA

As many-core based platforms, GPUs are massively data-parallel processors which have high chip scalability in terms of processing units (cores, threads), and high bandwidth with internal GPU memories. The architectural difference between CPUs and GPUs is the reason why the latter offer larger performance increase over CPU only implementation of parallel code working on large amounts of input data [12]. The main advantages of using GPUs are their low-cost, lowmaintenance and low power consumption relative to conventional parallel clusters and setups, while providing comparable or improved computational power [10]. For example, the latest GPUs of NVIDIA with 512 cores are readily available at consumer electronics stores for around $500. GPUs can be programmed using a framework introduced by NVIDIA in 2007 called CUDA [12]. CUDA is a programming model and hardware architecture for general purpose computations in NVIDIA’s GPUs [12]. The programmer can use CUDA for free of charge (including the compiler, driver, SDK, libraries, etc), and is easy to learn because it’s an extension of the C language. CUDA implements a heterogeneous computing architecture, where two different parts are often considered: the host (CPU side) and the device (GPU side). The host part of the code is responsible for controlling the program execution flow, transferring data to and from the device memory, and executing specific codes, called kernel functions, on the device. The device acts as a parallel coprocessor to the host. The host outsources the parallel part of the program as well as the data to the device, since it is more suited to parallel computations than the host. The kernel code is executed in the device by a set of threads. They are organized into a three-level hierarchy, from highest to lowest: a grid of thread blocks, blocks of threads, and threads which can share data through shared memory and can perform simple barrier synchronization [12,15].Using kernel functions, the programmer can specify the GPU resources: up to 65,535 blocks and up to 512 threads per block.

3

Spiking Neural P Systems

Now we first formally define SNP systems as computing models. An SNP system without delay, of degree m ≥ 1, is of the form Π = (O, σ1 , . . . , σm , syn, in, out), where: [1.] O = {a} is the alphabet made up of only one object a, called spike; [2.] σ1 , . . . , σm are m number of neurons of the form σi = (ni , Ri ), 1 ≤ i ≤ m, where: (a) ni ≥ 0 gives the initial number of spikes (a) contained in neuron σi ; (b) Ri is a finite set of rules of the following forms:

102

F.G. Cabarle et al.

(b-1) E/ac → ap , are Spiking rules, where E is a regular expression over a, c ≥ 1, and p ≥ 1 number of spikes are produced (with the restriction c ≥ p), transmitted to each adjacent neuron with σi as the originating neuron, and ac ∈ L(E); ak → ap , is a special case of (b-1) where L(E) = {ac }, k = c, p = 1; (b-2) as → λ, are Forgetting rules, for s ≥ 1, such that for each rule E/ac → ap of type (b-1) from Ri , as ∈ / L(E); [3.] syn = {(i, j) | 1 ≤ i, j ≤ m, i = j } are the synapses i.e. connection between neurons; [4.] in, out ∈ {1, 2, . . . , m} are the input and output neurons, respectively. The system works as follows: At any given time, a σi (neuron) should use exactly one rule only, if and only if the condition ac ∈ L(E) is met. This condition means as long as the multiplicity of spikes is in the language generated by the regular expression E, a rule (or several of them) is (are) applicable. The rule to be used or applied is chosen non-deterministically. If a spiking rule is used, after rule application c spikes are consumed in the σi , producing p number of spikes to all other σj such that (i, j) ∈ syn. If a Forgetting rule is applied, s number of a are removed from σi and no a or spike is produced. A global clock is followed by the system. Parallelism is at the system level, although each neuron works sequentially.

Fig. 1. Π1 generates the set N - {1}. Π1 outputs are the time differences between the first spike of σ3 and its succeeding spikes. A total ordering of the neurons is seen (σ1 to σ3 ) including a total ordering of the rules (1 to 5).

We designate the SNP system shown in Figure 1 as Π1 [18].For our simulations we use 2 additional systems: Figure 8 in [11] and Figure 14 in [11] which we designate as Π2 and Π3 respectively. Next we present the matrix representation of an SNP system and its computations. This representation makes use of the following vectors and matrix definitions: Configuration vector. Ck is the vector containing all spikes in every σ on the kth computation step/time. C0 is the initial Ck of the system. Spiking vector. Sk shows, at a given Ck , if a rule is applicable (having value 1 ) or not (having value 0 instead). Spiking transition matrix. MSN P is a matrix comprised of aij elements where aij is given as: −c if rule ri is in σj and is applied consuming c spikes; p if rule

Spiking Neural P System Simulations on a High Performance GPU Platform

103

ri is in σs (s = j and (s, j) ∈ syn and is applied producing p spikes in total; 0 if rule ri is in σs (s = j and (s, j) ∈ / syn. The spiking transition matrix MΠ1 is shown in equation (1). ⎛ ⎞ −1 1 1 ⎜ −2 1 1 ⎟ ⎜ ⎟ ⎟ MΠ1 = ⎜ (1) ⎜ 1 −1 1 ⎟ ⎝ 0 0 −1 ⎠ 0 0 −2 Equation (2) provides the configuration vector at the (k + 1)th step: Ck+1 = Ck + Sk · MΠ

(2)

For Π1 C0 =< 2, 1, 1 >. and we have the S0 =< 1, 0, 1, 1, 0 > given its C0 . Note that a second alternative S0 =< 0, 1, 1, 1, 0 >, is possible if we use rule (2) over rule (1) instead (but not both at the same time). V alidity in this case means that only one among several applicable rules is used and thus represented in the Sk . The C0 , S0 for Π2 and Π3 can be similarly shown.

4

Parallel SNP System Simulation on GPU

We designate the improved SNP system simulator in this paper as snpgpu-sim4 which is an update to snpgpu-sim3 produced in [2]. Among the improvements of snpgpu-sim4 over snpgpu-sim3 include the use of multiple thread-blocks to accomodate matrices more than 512 elements, and a more streamlined part of the simulation code for handling the relationships between Ri , Ck , and Sk . This section will further expound on these, among other things. The simulator takes in 3 inputs: M f , C0f , and Rf which are the file counterparts of M , C0 , and Ri , respectively. Skf is the file counterpart of Sk , which is produced by the simulator itself once it is run. PyCUDA was used in addition to conventional Python and CUDA C languages. PyCUDA is a Python wrapper for NVIDIA CUDA C and C++, enabling programmers to create GPU software using Python, and has been used for high performance computing [13]. The inputs are text files with delimiters, between rule to another rule in a σ and between σs themselves. The elements of M are entered in row-major order format into the file, and are mapped onto each thread of a thread block, within the block grid as shown in Figure 3. Figure 2 shows an instance of host-device interaction. The host functions sequentially and calls the kernel function/s. The device is split up into a grid of thread blocks, each with their own threads, and operate on the data in a single program, multiple data (SPMD) programming style [12]. The simulation algorithm is shown in Algorithm 1, which also indicates where a specific part of the simulation runs on (either host or device parts). Part I loads the 3 initial inputs and the succeeding inputs from their file counterparts, checking for formatting and pre-processing them for Part II. Part II, from Part I’s outputs and from

104

F.G. Cabarle et al.

Fig. 2. Diagram showing a single run of the simulation flow. The host runs sequentially while the device is made up of a grid of thread blocks, each with their own threads operating in paralel. Require: Input files: Ck, M, r. I. (HOST) Load input files.M f , Rf are loaded once only. C0f is also loaded once, then Ckf s afterwards. II. (HOST) Determine if a rule in Rf is applicable based on the number of spikes present in each σ seen in Ckf . Then generate all valid and possible spiking vectors in a list of lists Skf . III. (DEVICE) Run kernel function on all valid and possible Skf s from the current Ckf . Produce the next file configuration counterparts of Ck + 1s and their corresponding Skf s. IV. (HOST+DEVICE) Repeat steps I to IV, till at least one of the two Stopping criteria is encountered. Algorithm 1. Overview of SNP system simulation algorithm

Ckf and Rf , produces all the valid and possible Skf s. Part II produces all valid and possible Skf files as follows: For each ni of σi , the {1,0} strings are produced on a per neuron level. For example, for Π1 we have n1 = 2 for σ1 . Now we have σ1 strings ‘10’ (choose to use R1 instead of R2 ) and ‘01’ (choose to use R2 over R1 ). We only have one string for σ2 , the string ‘1’, since σ2 has only one rule and it is readily applicable. Neuron σ3 produces only one string also, ‘10’, since only one rule is applicable given its n3 = 1. Only R4 is used in σ3 and not R5 . Once all the neuron level {1,0} strings are produced, the strings are exhaustively paired up with the other strings in the other σs from left to right as the ordering is important. The output therefore of Part II in this example given Ck =< 2, 1, 1 > are (1,0,1,1,0) and (0,1,1,1,0). Elements of the input files are treated as strings up to this point, because of the the concatenation and regular expression checking processes, among others. Part III now treats the input elements as integral values. Equation 2 is performed in parallel such that each thread is either adding or multiplying a (matrix or vector) element. Once the Ck+1 are produced by the device, the results are

Spiking Neural P System Simulations on a High Performance GPU Platform

105

moved back to the host. Part IV then checks whether to proceed or to stop based on 2 stopping criteria for the simulation: (I) if a zero vector (vector of zeros) is encountered, (II) if the succeeding Ck s have all been produced in previous computations. Both (I) and (II) make sure that the simulation halts and does not enter an infinite loop.

Fig. 3. Different representations of a given matrix X: (a) original matrix form (b) linear array in row-major order form, (c) using CUDA thread blocks in a single thread block grid. The linear array shows how the array’s elements are laid out: a (4 × 4) grid made up of (2 × 2) thread blocks. Each thread in a thread block computes a unique element of the array in parallel, and all of them execute the same kernel function.

5

Simulation Results and Observations

The simulations in this paper were executed using an Intel Xeon E5504 quad core CPU running at 2 GHz per core (there are two of these CPUs so there are effectively 8 cores). Each core has a 4MB cache. The GPU is an NVIDIA Tesla C1060 high performance GPU with 240 streaming-processor (SP) cores organized as 30 streaming multiprocessor (SMs) and has 4GB of memory for storing data used by the kernel functions. A 64-bit Ubuntu 10.04 Linux operating system was used to host the simulations. A sequential i.e. CPU only version of snpgpusim4 was created and compared to snpgpu-sim4. We designate this CPU only simulator as snpcpu-sim4. snpcpu-sim4 is identical to snpgpu-sim4 except for the computation of equation (2). Figure 4 shows the running times of the simulators with Π1 as the SNP system. The run times per SNP system are shown using three different time measurements: the real time, user time, and sys time taken using the Ubuntu Linux command time, based on the Unix command of the same name. The real time is the time that has elapsed during the run of the program (a ‘wall clock’ time measurement). The user time is the time spent by the program running in the CPU while in user mode. The sys time is the total CPU time used by the OS on behalf of the program that is being measured, and while the process is in kernel mode. A program or process in kernel mode means the process can use system calls or services such as allocating memory for itself, including hardware access (a more privileged

106

F.G. Cabarle et al.

execution mode) while being in user mode means the program is usually restricted to its initial resources only (less privileged execution mode) [17]. In Figure 4 we see the large improvement of snpgpu-sim4 over snpcpu-sim4, as expected. As expected also, snpcpu-sim4 used up more time from the CPU as seen in the real and sys times. It’s worth mentioning that snpgpu-sim4 used a bit more of the CPU in the user times (though still significantly less than snpcpu-sim4 ) because snpgpu-sim4 still needed some work from the CPU to process the inputs. Another noteworthy point is that with all three runtime figures (Figure 4 to 6) the user run time is significantly far less compared to the other two time measurements because it only measures the time used by the program alone in the CPU, and no other programs are involved in the time measurement. Table 1summarizes averages of the kernel function runtimes and the CPU counter parts of the kernel functions as well as the average speedups. The maximum size, in terms of the number of neurons (Cknum ) and rules (Rnum ) of a system, that the current setup can simulate is given by Cknum = 4GBytes/(16Bytes + 4Bytes × Rnum ).

Fig. 4. Runtime graph of snpgpu-sim4 versus snpcpu-sim4 for Π1 showing (a) real, (b) user, and (c) sys times usage

Fig. 5. Runtime graph of snpgpu-sim4 versus snpcpu-sim4 for Π2 showing (a) real, (b) user, and (c) sys times usage

Spiking Neural P System Simulations on a High Performance GPU Platform

107

Fig. 6. Runtime graph of snpgpu-sim4 versus snpcpu-sim4 for Π3 showing (a) real, (b) user, and (c) sys times usage Table 1. Summary of averages: kernel and CPU times, and speedup. All time measurements are in seconds, except for KRTA which is in microseconds. RTSA is Real Time Speedup Average, UTSA is User Time Speedup Average, STSA is System Time Speedup Average. KRTA is the Kernel Runtime Average, the amount of time the kernel function spent running inside the GPU/device. CRTA is the CPU Runtime Average, the amount of CPU time used by the CPU only (i.e. sequential) counterpart of the kernel function. RTSA UTSA STSA Π1 156.1439811343 3.5999180999 178.3754195194 Π2 3.2014649226 0.9619771863 4.3513513514 Π3 67.0445847755 8.4018691589 192.8963174046

KRTA 107.33688871 μs 216.442000587 μs 153.418998544 μs

CRTA 3.8535563 3.938559 3.9137748

Acknowledgments. Francis Cabarle is supported by the DOST-ERDT program. Henry Adorna is funded by the DOST-ERDT research grant and the Alexan professorial chair of the UP Diliman Department of Computer Science. M.A. Mart´ınez–del–Amor and M.J. P´erez-Jim´enez are supported by “Proyecto de Excelencia con Investigador de Reconocida Val´ıa” of the “Junta de Andaluc´ıa” under grant P08-TIC04200, and by the project TIN2009–13192 of the “Ministerio de Educaci´ on y Ciencia” of Spain, both co-financed by FEDER funds.

References 1. Cabarle, F., Adorna, H., Mart´ınez-del-Amor, M.A.: An Improved GPU Simulator For Spiking Neural P Systems. Accepted in the IEEE Sixth International Conference on Bio-Inspired Computing: Theories and Applications, Penang, Malaysia (September 2011) 2. Cabarle, F., Adorna, H., Mart´ınez-del-Amor, M.A.: A Spiking Neural P system simulator based on CUDA. Accepted in the Twelfth International Conference on Membrane Computing, Paris, France (August 2011)

108

F.G. Cabarle et al.

3. Cecilia, J.M., Garc´ıa, J.M., Guerrero, G.D., Mart´ınez-del-Amor, M.A., P´erez-Hurtado, I., P´erez-Jim´enez, M.J.: Simulating a P system based efficient solution to SAT by using GPUs. Journal of Logic and Algebraic Programming 79(6), 317–325 (2010) 4. Cecilia, J.M., Garc´ıa, J.M., Guerrero, G.D., Mart´ınez-del-Amor, M.A., P´erez-Hurtado, I., P´erez-Jim´enez, M.J.: Simulation of P systems with active membranes on CUDA. Briefings in Bioinformatics 11(3), 313–322 (2010) 5. Chen, H., Ionescu, M., Ishdorj, T.-O., P˘ aun, A., P˘ aun, G., P´erez-Jim´enez, M.: Spiking neural P systems with extended rules: universality and languages. Natural Computing: an International Journal 7(2), 147–166 (2008) 6. Ciobanu, G., Wenyuan, G.: P Systems Running on a Cluster of Computers. In: Mart´ın-Vide, C., Mauri, G., P˘ aun, G., Rozenberg, G., Salomaa, A. (eds.) WMC 2003. LNCS, vol. 2933, pp. 123–139. Springer, Heidelberg (2004) 7. D´ıaz, D., Graciani, C., Guti´errez, M.A., P´erez-Hurtado, I., P´erez-Jim´enez, M.J.: Software for P systems. In: P˘ aun, G., Rozenberg, G., Salomaa, A. (eds.) The Oxford Handbook of Membrane Computing, ch. 17, pp. 437–454. Oxford University Press, Oxford (2009) 8. Fatahalian, K., Sugerman, J., Hanrahan, P.: Understanding the efficiency of GPU algorithms for matrix-matrix multiplication. In: Proceedings of the ACM SIGGRAPH/EUROGRAPHICS Conference on Graphics Hardware (HWWS 2004), pp. 133–137. ACM, NY (2004) 9. Garland, M., Kirk, D.B.: Understanding throughput-oriented architectures. Communications of the ACM 53(11), 58–66 (2010) 10. Harris, M.: Mapping computational concepts to GPUs. In: ACM SIGGRAPH 2005 Courses, NY, USA (2005) 11. Ionescu, M., P˘ aun, G., Yokomori, T.: Spiking Neural P Systems. Journal Fundamenta Informaticae 71(2,3), 279–308 (2006) 12. Kirk, D., Hwu, W.: Programming Massively Parallel Processors: A Hands On Approach, 1st edn. Morgan Kaufmann, MA (2010) 13. Kl¨ ockner, A., Pinto, N., Lee, Y., Catanzaro, B., Ivanov, P., Fasih, A.: PyCUDA: GPU Run-Time Code Generation for High-Performance Computing. Scientific Computing Group, Brown University, RI, USA (2009) 14. Nguyen, V., Kearney, D., Gioiosa, G.: A Region-Oriented Hardware Implementation for Membrane Computing Applications and Its Integration into Reconfig-P. In: P˘ aun, G., P´erez-Jim´enez, M.J., Riscos-N´ un ˜ez, A., Rozenberg, G., Salomaa, A. (eds.) WMC 2009. LNCS, vol. 5957, pp. 385–409. Springer, Heidelberg (2010) 15. NVIDIA corporation, NVIDIA CUDA C programming guide, version 3.0. NVIDIA, CA, USA (2010) 16. P˘ aun, G., Ciobanu, G., P´erez-Jim´enez, M. (eds.): Applications of Membrane Computing. Natural Computing Series. Springer, Heidelberg (2006) 17. Stallings, W.: Operating systems: internals and design principles, 6th edn. Pearson/Prentice Hall, NJ, USA (2009) 18. Zeng, X., Adorna, H., Mart´ınez-del-Amor, M.A., Pan, L., P´erez-Jim´enez, M.: Matrix Representation of Spiking Neural P Systems. In: Gheorghe, M., Hinze, T., P˘ aun, G., Rozenberg, G., Salomaa, A. (eds.) CMC 2010. LNCS, vol. 6501, pp. 377–391. Springer, Heidelberg (2010)

SpotMPI: A Framework for Auction-Based HPC Computing Using Amazon Spot Instances Moussa Taifi, Justin Y. Shi, and Abdallah Khreishah Temple University, Computer Science Department, Philadelphia, PA, USA {moussa.taifi,shi,akhreish}@temple.edu

Abstract. The economy of scale offers cloud computing virtually unlimited cost effective processing potentials. Theoretically, prices under fair market conditions should reflect the most reasonable costs of computations. The fairness is ensured by the mutual agreements between the sellers and the buyers. Resource use efficiency is automatically optimized in the process. While there is no lack of incentives for the cloud provider to offer auction-based computing platform, using these volatile platform for practical computing is a challenge for existing programming paradigms. This paper reports a methodology and a toolkit designed to tame the challenges for MPI applications. Unlike existing MPI fault tolerance tools, we emphasize on dynamically adjusted optimal checkpoint-restart (CPR) intervals. We introduce a formal model, then a HPC application toolkit, named SpotMPI, to facilitate the practical execution of real MPI applications on volatile auction-based cloud platforms. Our models capture the intrinsic dependencies between critical time consuming elements by leveraging instrumented performance parameters and publicly available resource bidding histories. We study algorithms with different computing v.s. communication complexities. Our results show non-trivial insights into the optimal bidding and application scaling strategies.

1

Introduction

The economy of scale affords cloud computing extreme cost effectiveness potentials. While it is in general difficult to assess the real cost of a computation task, the auction-based provisioning scheme offers a fair pricing structure. Theoretically, prices under fair market conditions reflect the most reasonable costs of computations. The fairness is ensured by the mutual agreements between the sellers and the buyers. From the consumer’s perspective, high performance computing (HPC) applications are the biggest potential beneficiaries since their infrastructure costs are the most expensive. From the seller’s perspective, HPC applications represent the most reliable income stream since they are the most resource intensive users. Theoretically, resource usage efficiency is automatically maximized under the auction-based provisioning schemes. Y. Xiang et al. (Eds.): ICA3PP 2011 Workshops, Part II, LNCS 7017, pp. 109–120, 2011. c Springer-Verlag Berlin Heidelberg 2011 

110

M. Taifi, J.Y. Shi, and A. Khreishah

Traditional HPC applications are typically optimized for hardware features to obtain processing efficiency. Since transient component errors can halt the entire application, it has become increasingly important to create autonomic applications that can automate checkpoint and re-starting with little loss of useful work. Although the existing HPC applications are not suitable for volatile computing environments, with an automated checkpoint-restart (CPR) HPC toolkit, it is plausible that practical HPC applications could gain additional cost advantages using auction-based resources by dynamically minimized CPR overheads. We first establish models for estimating running times of a HPC application using auction-based cloud resources. The proposed models take into account the time complexities of the HPC application, the overheads of checkpoint-restart, and the publicly available resource bidding history. We seek to unravel the interdependencies between the applications’ computing/communication complexities, the number of required processors, bidding prices and the eventual processing costs. We then introduce the SpotMPI toolkit and show how it automate MPI application processing using volatile resources and the guidance of the formal models. Applying our models to recent bidding histories of Amazon EC2 HPC resources, we report preliminary results for two HPC application types with different computing and communication complexities.

2 2.1

Background Auction-Based Computing: Spot Instances

Amazon is one of the first cloud computing vendors that provides at least two types of cloud instances: on-demand instance and spot instance. An on-demand instance has a fixed price. Once ordered, it provides services according to Amazon’s Service Level Agreement (SLA). Spot instance is a type of resource whose availability is controlled by the current bidding price and the auctions market. There are three special features of Amazon’s spot instance pricing policy: – A successful bid does not guarantee the exclusive resource access for the entire requested duration. The Amazon engine can terminate access at any time if a higher bid is received. – Amazon does not charge a partial hour (job terminated before reaching the hour boundary) if the termination was caused by out-bidding. Otherwise, the partial hour is charged in full if the user terminates the job. – Amazon will only charge the user the highest market price that is less than the user’s successful bid. We have chosen two types of Amazon EC2 HPC resources for this study. The cc1.4xlarge and the cg1.4xlarge are cluster HPC instances that provide cluster level performance(23 GB of memory, 8 cores, 10 Gigabit Ethernet). The main difference is the presence of GPUs (2 x NVIDIA Tesla “Fermi” M2050)in the cg1.4xlarge which provide more power for compute intensive applications.

SpotMPI: Auction-Based HPC Computing

111

Figures 1 records a sample market price history for the cc1.4xlarge instance type from May 5 to May 11, 2011. This instance type shows typical user behavior for more legacy HPC applications. The cg1.4xlarge instance type illustrates ressources for HPC applications that can benefit from GPU processing. Since many legacy HPC applications are not suitable for GPU processing, cg1.4xlarge history is less interesting.

Fig. 1. Market Prices of cc1.4 Instance in May 2011

2.2

HPC in the Cloud

Although HPC applications are the biggest potential beneficiaries of cloud computing, except for a few simple applications, there are still many practical concerns. – Most mathematical libraries rely on optimized numerical codes that exploit common hardware features for extreme efficiencies. Some of the hardware features are not mapped in virtual machines. Hardware cache is one of the examples. Consequently, HPC applications suffer more performance drawbacks in addition to the normal virtualization overhead. – Many HPC applications have high inter-processor communication demand. Current virtualized networks have difficulty meeting these high demands. – All existing HPC applications handle only two communication states: success and failure. While success is a reliable state, failure is not. Existing applications treat timeout identical to failure. Consequently, any transient component failure can halt the entire application. Using volatile spot instances for these applications is a serious challenge. Initially, the low end cloud services provide little guarantee on the deliverable performance for HPC applications. Recently, high end cloud resources have been developed specifically for HPC applications. These improvements have demonstrated hopeful features ([28], [25] and [17]). These improvements show the diminishing overhead of virtual machine monitors such as XEN [6]. Due to the severity of declining MTBFs, fault tolerance for MPI applications has also progressed. These developments inspired the design and development of SpotMPI. 2.3

Checkpoint-Restart (CPR) MPI Applications

Much research has been done to provide fault tolerance for MPI applications. FTMPI [12] uses interactive process fault tolerance. Starfish [3] supports a number

112

M. Taifi, J.Y. Shi, and A. Khreishah

of CPR protocols and LA-MPI [13] provides message level failure tolerance. Egida [21] experimented with message logging grammar specification as means for fault tolerance. Cocheck [24] extends the Condor [18] scheduler to provide a set of fault tolerance methods that can be used by MPI applications. We choose the OpenMPI’s coordinated CPR because of its cluster wide checkpoint advantage since more fine grained strategies will not work in this highly volatile environment ( [16], [20] and [30]). The challenges for using the volatile spot instances are not much different than regular clusters with crash failures. In fact, for performance analysis, the spot instance out-of-bid failures can be modeled as random crash failures, since the Amazon cloud engine terminates out-bid instances without prior notice[2]. These out-of-bid failures force applications to adopt technologies to prevent excessive work loss due to frequent interruptions. Map-reduce applications ([11] can easily adopt single task CPR using spot instances [8]). A map-reduce application does not require inter-task communications. Parallel processing can be controlled external to the individual tasks. Therefore, spot instances can be used as “accelerators” via a simple job monitor that tracks and restarts dead jobs automatically [9]. Another noticeable effort using simulation to study the spot instances includes [5] and [26]. By simulating the behavior of a single instance under different bids, these work outlined the inherent tradeoff between completion time and budget. In [5] a decision model is proposed that describes a simulator that is able to determine under a set of conditions the total expected time of a single application. Yet another study [26] discussed a set of checkpoint strategies that can maximize the use of the spot instances while minimizing the costs. To the best of authors’ knowledge there have been no direct evaluation of practical MPI applications on spot instances. The volatile auction-based computing platform challenges the established HPC programming practices.

3

Evaluating MPI Applications Using Auction-Based Platform

For HPC applications using large number of processors, the CPR overhead is the biggest cost factor. Without CPR optimization, it is unlikely to gain practical acceptance for MPI application to run on the volatile auction-based platforms. We report a theoretical model based on application resource time complexities [22] and optimal CPR models ([10], [27]). In addition, we describe a toolkit named SpotMPI that can support autonomic MPI application using spot instance clusters. This toolkit can monitor spot instances and bidding prices, automate checkpointing at bidding price (and history) adjusted optimal intervals and automatically restart application after out-of-bid failures.

4

Theoretical Model

Auction prices vary dynamically depending on the supply and demand in the Amazon market place. There are no guidelines from Amazon as how the prices are set.

SpotMPI: Auction-Based HPC Computing

113

Table 1. Definition of Symbols and Variables for Modeling the Runtime Symbol t0 α K0 K1 K2 T E ET noobbidi tobserved P W u N ns Tpar

Description Interval of application-wide checkpoint. Expected rate of out-of-bid failures. Time needed to create and store a checkpoint. Time needed to read and recover a checkpoint. Average out-of-bid downtime. Estimated time needed to run the application with no checkpoints and no failures Expected running time between checkpoints Expected total running time Total number of out of bid failures corresponding to bidi over tobserved Total observed time Number of processing units Instrumented processor capacity in number of computational steps per second Instrumented network capacity in bytes per second Problem size Number of iterations Parallel processing time

Unlike other project [29] that use autoregressive models to maximize the profit for fictitious cloud provider, we focus on the intrinsic characteristics of users application and bidding history. We are interested in the inherent dependencies between these characteristics and their impact on the optimal CPR intervals – the largest cost factor for MPI applications to run on a volatile platform. 4.1

Bid-Aware Optimal CPR Interval

We assume that the time between consecutive out-of-bid failures is exponentially distributed with rate α. This allows the out-of-bid failures to be modeled the same way as component failures but at different rates. Thus we can extend the previous works on optimal CPR interval for distibuted memory applications. In this paper, we refer to the original CPR interval work by [27], which was extended by [10] and was later adapted to MPI by [19]. We start our discussion using the same symbols. Like in [19], we obtain the expected application running time with checkpoints and failures. Important assumptions are that out-of-bid failure occurs only once per checkpoint interval and all failures are independent: ET =

T t2 (K0 + t0 + α(t0 K1 + 0 )) t0 2

This leads to the optimal CPR interval by ([19], [27] and [10]):  2K0 t0 = (1) α A crucial difference between stable clusters and spot instances clusters is that an out-of-bid failure will force an application downtime that is absent for component

114

M. Taifi, J.Y. Shi, and A. Khreishah

failures. This means that the restart time K1 cannot happen until the average downtime per out-of-bid failure K2 has elapsed. The expected running time using spot instances becomes: ET =

T t2 (K0 + t0 + α(t0 (K1 + K2 ) + 0 )) t0 2

K2 can be obtained using the price history and the current bid. Given the pricing history and a current bid, we can also calculate the new density α of out-of-bid failures. Thus the optimal bid-aware CPR interval can be calculated as:  2K0 (2) t0 = noobbid i

tobserved

5

SpotMPI Toolkit

SpotMPI is a HPC toolkit constructed using OpenMPI coordinated CPR library [15], the Starcluster [1], and the BLCR project [14]. The OpemMPI and BLCR libraries facilitate the execution of automatic CRP at optimal intervals. Starcluster facilitates the creation and management of HPC clusters using Amazon EC2 resources. The latest Starcluster also supports spot instances and allows Python plugins during the launch of the cluster.As shown in Figure 2, the SpotMPI toolkit integrates multiple complementary tools for the auction-based HPC computing. 6WDUFOXVWHU 2UFKHVWUDWLRQ 6HUYLFH

+3& $SSOLFDWLRQV

6SRW03, )UDPHZRUN

$:6 $PD]RQ 6SRW ,QVWDQFHV

2SHQ03, &KHFNSRLQW 5HVWDUW 6HUYLFH

Fig. 2. SpotMPI environment

SpotMPI has four components: cluster monitor, CPR calculator, checkpoint executor, and restarter. These modules are initiated by the Starcluster script at spot instance cluster creation time. The cluster monitor pulls the status and

SpotMPI: Auction-Based HPC Computing

115

bidding prices of all instances continuously. The interactive bidding price and dynamic price history are used by the CPR calculator to generate the next optimal CPR interval. A composite timing model (next section) is responsible for estimating the total processing times. The checkpoint executor saves the state of the MPI application in the users EBS volume at dynamically adjusted intervals. Any out-bid failure will cause the application to halt. Upon a winning bid, the application will automatically restart from the last checkpoint using the Open MPI restart library. &KHFNSRLQW LQWHUYDO IRUFDVWHU

&KHFNSRLQW 5HVWDUW 6HUYLFH

$PD]RQ(&2Q 'HPDQGLQVWDQFHVHJ IRUPRQLWRULQJ

0RQLWRULQJ 6HUYLFH

&OXVWHU 2UFKHVWUDWLRQ HJ6WDUFOXVWHU

$PD]RQ(%6VWRUDJH

$PD]RQ(&6SRWLQVWDQFHVHJ &OXVWHULQVWDQFHVFF[ODUJH DQGFJ[ODUJH

8VHU(%69ROXPH HJ'DWDDQG03, ([HFXWDEOHV

Fig. 3. SpotMPI architecture design

6

Computational Results

Steady State Timing Model. To evaluate T , we need an estimate of the failure free processing time. We use the steady state timing model [22] to determine required running time based on major component usage complexities. Table 1 shows the symbols used in timing models. The general problem of assessing the processing time of a parallel application is difficult. There are too many hard to quantify factors. However, a steady state timing model can capture the intrinsic dependencies between major time consuming elements, such as computing, communication and input/output, by using instrumented capabilities like W and u. The idea is to eliminate the nonessential constant factors. Thus contrasting timing models can reveal non-trivial parallel processing insights [7]. In this paper, we chose to study two typical algorithm classes (Table 2) for the use of spot instance computing. Timing models in general can be applied to all deterministic algorithm classes [22]. Evaluation of Bid-aware Optimal CPR Interval. We validate the bidaware CPR interval using non-optimal intervals. In Figures 4 the behavior of the speed up is visualized under different CPR intervals.

116

M. Taifi, J.Y. Shi, and A. Khreishah Table 2. Algorithm Classes A1 and A2

Compute and Communication Timing model Sample Application Complexities 2 A1 : (O(n2 ), O(n)) Tpar = PNW + 16N Molecular force simulation u 3 2 A2 : (O(n3 ), O(n2 )) Tpar = PNW + 16N Linear solvers u

This figure shows the clear advantages of bid-aware optimal CPR intervals that have avoided longer completion times and higher total costs. We also notice that as the bid increases the advantage of optimal CPR interval decreases. This is because at higher bids, frequent checkpointing is not needed as much. Total Expected Speedup vs Bid prices vs Checkpoints intervals 60 0.01*optimal C/R interval 0.2*optimal C/R interval 1*optimal C/R interval 20*optimal C/R interval 50*optimal C/R interval

50

Speed up

40

30

20

10

0 0.52

0.53

0.54

0.55

0.56 Bid prices

0.57

0.58

0.59

0.6

Fig. 4. A1 Speedup Using 100 Spot Instances and Different CPR Intervals

Bidding Price and Application Processing Time. We are also interested in understanding, given the price history, how a new bid would affect the total processing time. Once we do this, we can then derive a number of other important metrics, such as speedup, efficiency, total cost, speedup per dollar, and efficiency per dollar deploying different numbers of processing units. In the following calculations, we assume: – The application uses the bid-aware optimal CPR intervals. – The HPC application is run under the optimal granularity (synchronization overhead is zero). – The Amazon resources deliver the advertised capabilities. We can then plug the steady state timing models in Table 2 directly into equation 2 4.1: ( PNW + 16N t2 u )ns (3) (K0 + t0 + α(t0 (K1 + K2 ) + 0 )) ET (A1 ) = t0 2 3

ET (A2 ) =

2

( PNW + 16N t2 u )ns (K0 + t0 + α(t0 (K1 + K2 ) + 0 )) t0 2

(4)

SpotMPI: Auction-Based HPC Computing

117

Table 3. Critical Parameters Variable P W u N ns

Range 200 to 1, 000 instances 1.5×109 measured algorithmic step per second using cc1.4xlarge Network speed: 250MB per second measured Problem size: 104 to 105 Number of iterations: 103 to 106

Equations 3 and 4 capture the intrinsic dependencies between critical factors, such as bidding price, price history, the number of spot instances (P ) and overall processing time. To minimize errors, we conducted program instrumentations to get the ranges of W and u. Table 3 shows the value ranges in our calculations. We report the results in Figures 5 and 6. Total Expected Efficiency vs Bid prices vs Number of Instances

Total Expected Speedup vs Bid prices vs Number of Instances

400

#PU=200 #PU=400 #PU=600 #PU=800 #PU=1000

0.8

200

0.6 0.4

0.54

0.56 Bid prices

0.58

0.2 0.52

0.6

2000

#PU=200 #PU=400 #PU=600 #PU=800 #PU=1000

1000

0 0.52

0.54

0.56 Bid prices

0.58

0.6

0.54

0.56 Bid prices

0.58

0.6

30

0.8 #PU=200 #PU=400 #PU=600 #PU=800 #PU=1000

0.6 0.4 0.2 0 0.52

0.54

0.56 Bid prices

0.58

0.6

#PU=200 #PU=400 #PU=600 #PU=800 #PU=1000

20

10

0 0.52

Total Speed up per dollar vs Bid prices vs Number of Instances

Total Expected Maximum Cost vs Bid prices vs Number of Instances 3000

Total Speed up per dollar

Total Expected Maximum Cost

0 0.52

Total Efficiency per dollar

#PU=200 #PU=400 #PU=600 #PU=800 #PU=1000

Total Expected time vs Bid prices vs Number of Instances Total Expected time in hours

1

Efficiency

Speed up

600

1.5

0.54

0.56 Bid prices

0.58

0.6

Total Efficiency per dollar −3 xvs10Bid prices vs Number of Instances #PU=200 #PU=400 #PU=600 #PU=800 #PU=1000

1

0.5

0 0.52

0.54

0.56 Bid prices

0.58

0.6

Fig. 5. A1 using 200 to 1,000 spot instances with n=100,000 to 100,0000 iterations

First, we observe that HPC applications can indeed gain practical feasibility using spot instances under optimized CPR intervals. As indicated by the Amdahl’s law [4], the diminish of return is also clearly visible when the number of spot instances increases for the same algorithms. For A1 (with linear communication complexity), speed up and efficiency drop significantly when spot instances are greater than 200. For A2 (with O(n2 ) communication complexity), speedup and efficiency drop much earlier. We also notice that for (A2 ), the bidding prices have much bigger impact on speedup than A1 . The added dimension of bidding price reveals cost effectiveness of different configurations. Although higher bids can deliver better performance, the cost effectiveness actually decreases (see Speedup Per Dollar charts). Therefore, the

118

M. Taifi, J.Y. Shi, and A. Khreishah

users should use these figures to optimize budget, processing deadline or anything in between. The non-trivial insight is the high price sensitivity for algorithms with high communication complexities. The cost effectiveness is also difficult to visualize without the proposed tools. These results provide the ground for selecting the best number of processors (spot instances) and the most promissing bidding price for a given objective. Total Expected Speedup vs Bid prices vs Number of Instances

Total Expected Efficiency vs Bid prices vs Number of Instances

100

Efficiency

Speed up

60 40

0.2 0.1

20 0.52

0.54

0.56 Bid prices

0.58

0 0.52

0.6

Total Expected Maximum Cost vs Bid prices vs Number of Instances 1500 #PU=200 #PU=400 #PU=600 #PU=800 #PU=1000

1000

500

0 0.52

0.54

0.56 Bid prices

0.58

0.6

0.54

0.56 Bid prices

0.58

0.6

7

0.8 #PU=200 #PU=400 #PU=600 #PU=800 #PU=1000

0.6 0.4 0.2 0 0.52

0.54

0.56 Bid prices

0.58

0.6

#PU=200 #PU=400 #PU=600 #PU=800 #PU=1000

6 5 4 3 2 0.52

Total Speed up per dollar vs Bid prices vs Number of Instances Total Speed up per dollar

Total Expected Maximum Cost

#PU=200 #PU=400 #PU=600 #PU=800 #PU=1000

0.3

Total Efficiency per dollar

#PU=200 #PU=400 #PU=600 #PU=800 #PU=1000

80

Total Expected time vs Bid prices vs Number of Instances Total Expected time in hours

0.4

3

0.54

0.56 Bid prices

0.58

0.6

Total Efficiency per dollar −3 xvs10Bid prices vs Number of Instances #PU=200 #PU=400 #PU=600 #PU=800 #PU=1000

2

1

0 0.52

0.54

0.56 Bid prices

0.58

0.6

Fig. 6. A2 Using 200 to 1,000 Spot Instances For N=10,000 and 1,000 Iterations

7

Conclusion

Finding the optimal bidding strategy for any application is a difficult problem. For specific applications, our proposed approach gives reasonable predictions that can guide the choice of a promising bidding strategy based on the intrinsic dependencies of critical factors. The timing model along the bid-aware CPR model provide an effective tool to determine the optimal bid as well as the optimal number of processing units needed for completing a specific application. This research paves the ground for more specialized pricing models for cloud providers by giving more insights about the return of investment. For example, since the speedup gain slows down when the number of processors reaches a level, it makes sense to give lower prices as “volume discounts” that is sensitive to the communication complexities. The new pricing models may change users’ behavior that in tern would also affect the providers that eventually would reach an equilibrium. Meanwhile, the resource utilization is maximized. Other innovative ideas are also possible. For example, self-healing applications [23] could enjoy much better cost advantages by setting bidding ranges to organize defensive rings to protect the users core interests while maintaining the lowermost cost structures.

SpotMPI: Auction-Based HPC Computing

119

Spot instances give the provider much freedom in dispatching resources for meeting dynamic users needs. This ultimate freedom allows for the ultimate computational efficiency and fair revenue/cost generation. It also challenges the HPC community for highly efficient and more flexible programming means that can automatically exploit cheaper resources on the fly. Acknowledgment. The authors would like to thank Professor Slobodan Vucetic and his Ph.D. student Vladmir Coric for the initial discussions of Amazon bidding histories. This research is supported in part by the National Science Foundation grant CNS 0958854 and educational resource grants from Amazon.com.

References 1. Starcluster (2010), http://web.mit.edu/stardev/cluster/ 2. Amazon hpc cluster instances (2011), http://aws.amazon.com/ec2/hpc-applications/ 3. Agbaria, A.M., Friedman, R.: Starfish: fault-tolerant dynamic mpi programs on clusters of workstations. In: Proceedings of the Eighth International Symposium on High Performance Distributed Computing, 1999, pp. 167–176 (1999) 4. Amdahl, G.M.: Validity of the single processor approach to achieving large scale computing capabilities. In: Proceedings of the Spring Joint Computer Conference, April 18-20, pp. 483–485. ACM, New York (1967) 5. Andrzejak, A., Kondo, D., Yi, S.: Decision model for cloud computing under sla constraints. In: Proc. IEEE Int Modeling, Analysis & Simulation of Computer and Telecommunication Systems (MASCOTS) Symp., pp. 257–266 (2010) 6. Barham, P., Dragovic, B., Fraser, K., Hand, S., Harris, T., Ho, A., Neugebauer, R., Pratt, I., Warfield, A.: Xen and the art of virtualization. In: Proceedings of the Nineteenth ACM Symposium on Operating Systems Principles, pp. 164–177. ACM, New York (2003) 7. Blathras, K., Szyld, D.B., Shi, Y.: Timing models and local stopping criteria for asynchronous iterative algorithms. Journal of Parallel and Distributed Computing 58(3), 446–465 (1999) 8. Borthakur, D.: The hadoop distributed file system: Architecture and design (2007), http://developer.yahoo.com/hadoop/tutorial/ 9. Chohan, N., Castillo, C., Spreitzer, M., Steinder, M., Tantawi, A., Krintz, C.: See spot run: using spot instances for mapreduce workflows. In: Proceedings of the 2nd USENIX Conference on Hot Topics in Cloud Computing, p. 7. USENIX Association (2010) 10. Daly, J.T.: A higher order estimate of the optimum checkpoint interval for restart dumps. Future Generation Computer Systems 22(3), 303–312 (2006) 11. Dean, J., Ghemawat, S.: Mapreduce: Simplified data processing on large clusters. Communications of the ACM 51(1), 107–113 (2008) 12. Fagg, G., Dongarra, J.: Ft-mpi: Fault tolerant mpi, supporting dynamic applications in a dynamic world. In: Dongarra, J., Kacsuk, P., Podhorszki, N. (eds.) PVM/MPI 2000. LNCS, vol. 1908, pp. 346–353. Springer, Heidelberg (2000) 13. Graham, R.L., Choi, S.E., Daniel, D.J., Desai, N.N., Minnich, R.G., Rasmussen, C.E., Risinger, L.D., Sukalski, M.W.: A network-failure-tolerant message-passing system for terascale clusters. International Journal of Parallel Programming 31(4), 285–303 (2003)

120

M. Taifi, J.Y. Shi, and A. Khreishah

14. Hargrove, P.H., Duell, J.C.: Berkeley lab checkpoint/restart (blcr) for linux clusters. In: Journal of Physics: Conference Series, vol. 46, p. 494. IOP Publishing (2006) 15. Hursey, J.: Coordinated Checkpoint/Restart Process Fault Tolerance for MPI Applications on HPC Systems. PhD thesis, Indiana University, Bloomington, IN, USA (July 2010) 16. Hursey, J., Squyres, J.M., Mattox, T.I., Lumsdaine, A.: The design and implementation of checkpoint/restart process fault tolerance for open mpi. In: Proc. IEEE Int. Parallel and Distributed Processing Symp. IPDPS 2007, pp. 1–8 (2007) 17. Iosup, A., Ostermann, S., Yigitbasi, N., Prodan, R., Fahringer, T., Epema, D.: Performance analysis of cloud computing services for many-tasks scientific computing. IEEE Transactions on Parallel and Distributed Systems 22(6), 931–945 (2011) 18. Litzkow, M., Tannenbaum, T., Basney, J., Livny, M.: Checkpoint and migration of unix processes in the condor distributed processing system. Technical report, Technical Report (1997) 19. Lusk, E.: Fault tolerance in mpi programs. Special issue of the Journal High Performance Computing Applications, IJHPCA (2002) 20. Moody, A., Bronevetsky, G., Mohror, K., de Supinski, B.R.: Design, modeling, and evaluation of a scalable multi-level checkpointing system. In: Proc. Int. High Performance Computing, Networking, Storage and Analysis (SC) Conf. for, pp. 1–11 (2010) 21. Rao, S., Alvisi, L., Vin, H.M.: Egida: An extensible toolkit for low-overhead faulttolerance. In: Twenty-Ninth Annual International Symposium on Fault-Tolerant Computing, 1999. Digest of Papers, pp. 48–55. IEEE, Los Alamitos (1999) 22. Shi, J.Y.: Program scalability analysis. In: International Conference on Distributed and Parallel Processing. Geogetown University, Washington D.C (1997) 23. Shi, J.Y., Taifi, M., Khreishah, A., Wu, J.: Sustainable gpu computing at scale. In: 14th IEEE International Conference in Computational Science and Engneering 2011 (2011) 24. Stellner, G.: Cocheck: Checkpointing and process migration for mpi. In: Proceedings of the 10th International Parallel Processing Symposium, IPPS 1996, pp. 526–531. IEEE Computer Society, Washington, DC, USA (1996) 25. Vecchiola, C., Pandey, S., Buyya, R.: High-performance cloud computing: A view of scientific applications. In: Proc. 10th Int. Pervasive Systems, Algorithms, and Networks (ISPAN) Symp., pp. 4–16 (2009) 26. Yi, S., Kondo, D., Andrzejak, A.: Reducing costs of spot instances via checkpointing in the amazon elastic compute cloud. In: 2010 IEEE 3rd International Conference on Cloud Computing, pp. 236–243. IEEE, Los Alamitos (2010) 27. Young, J.W.: A first order approximation to the optimum checkpoint interval. Communications of the ACM 17(9), 530–531 (1974) 28. Youseff, L., Wolski, R., Gorda, B., Krintz, C.: Evaluating the performance impact of xen on mpi and process execution for hpc systems. In: Proceedings of the 2nd International Workshop on Virtualization Technology in Distributed computing, p. 1. IEEE Computer Society, Los Alamitos (2006) 29. Zhang, Q., Grses, E., Boutaba, R., Xiao, J.: Dynamic resource allocation for spot markets in clouds. In: Proceedings of the 11th USENIX Conference on Hot Topics in Management of Internet, Cloud, and Enterprise Networks and Services (2011) 30. Zheng, G., Shi, L., Kal´e, L.V.: Ftc-charm++: An in-memory checkpoint-based fault tolerant runtime for charm++ and mpi. In: 2004 IEEE International Conference on Cluster Computing, pp. 93–103. IEEE, Los Alamitos (2004)

Investigating the Scalability of OpenFOAM for the Solution of Transport Equations and Large Eddy Simulations Orlando Rivera1 , Karl F¨urlinger2, and Dieter Kranzlm¨uller1,2 1

Leibniz Supercomputing Centre (LRZ), Munich, Germany {Orlando.Rivera,Dieter.Kranzlmueller}@lrz.de 2 MNM-Team Ludwig-Maximilians-Universit¨at (LMU), Munich, Germany [email protected]

Abstract. OpenFOAM is a mainstream open-source framework for flexible simulation in several areas of CFD and engineering whose syntax is a high level representation of the mathematical notation of physical models. We use the backward-facing step geometry with Large Eddy Simulations (LES) and semiimplicit methods to investigate the scalability and important MPI characteristics of OpenFOAM. We find that the master-slave strategy introduces an unexpected bottleneck in the communication of scalar values when more than a hundred MPI tasks are employed. An extensive analysis reveals that this anomaly is present only in a few MPI tasks but results in a severe overall performance reduction. The analysis work in this paper is performed with the tool IPM, a portable profiling and workload characterization tool for MPI programs.

1 Introduction OpenFOAM (Open Field Operation and Manipulation) is an extensive framework for the solution of Partial Differential Equation (PDE) using the Finite Volume Method (FVM). It is one of the most popular open source tools used in continuum mechanics and Computational Fluid Dynamics (CFD). Written in C++, it makes use of advanced features in OOP (Object Oriented Programming) and modern programming techniques to mimic the mathematical notation of tensor algebra and PDE solutions [8]. OpenFOAM is not a monolithic program. Instead, it consists of many libraries grouped by functionality, on top of which solvers are built. A solver is a problemspecific glue-like program, which is linked with appropriate libraries for a specific problem. Some libraries are common to all solvers: basic mesh manipulation, parallelization, finite volume method, etc. Others libraries are used only if needed, e.g., for turbulence models, for compressible or incompressible flows, for dynamic mesh handling, and so on. The parallelization of OpenFOAM is performed using MPI (Message Passing Interface). Applications and solvers in OpenFOAM are the same for serial or parallel execution; a master-slave model is used for parallel runs. Non-blocking and blocking send/receive functions and reduction operations are at the core of each solver. Y. Xiang et al. (Eds.): ICA3PP 2011 Workshops, Part II, LNCS 7017, pp. 121–130, 2011. c Springer-Verlag Berlin Heidelberg 2011 

122

O. Rivera, K. F¨urlinger, and D. Kranzlm¨uller

In this paper we analyze the scalability and performance characteristics of OpenFOAM for a Large Eddy Simulation test case. We perform a strong scaling study up to 256 MPI tasks and a weak scaling study up to 512 MPI tasks. We use performance tools, in particular the Integrated Performance Monitor (IPM), in order to gain an understanding of the main performance influencing factors and we identify an imbalanced time in point to point operations as a cause for limited scalability. The rest of this paper is organized as follows: In Sect. 2 we give an overview of related performance studies that have been conducted using OpenFOAM. In Sect. 3 we describe our experimental setup and in Sect. 4 we describe the results of our performance study. We conclude and discuss directions for future work in Sect. 5.

2 Related Work The parallel behavior of OpenFOAM is not very well understood when executed on massively parallel systems. Scalability and efficiency of OpenFOAM is an area of open debate. It has been reported that OpenFOAM scales well for small core counts, up to 64 MPI tasks using the lid-driven cavity test by Calegari et al. [12] and in a study for the HPC advisory council [9]. Pringle reported his efforts to port and test OpenFOAM on a Cray XT5 system (the HeCTOR system at EPCC) with the laminar lid-driven cavity test as well [4]. He was successful to run up to 2048 MPI tasks. Results on scalability were measured using meshes with 1003 and 2003 cells (one and eight million, respectively). In this setup OpenFOAM experiences a scalability limit at around 512 MPI tasks. The CSC - IT Center for Science in Finland has also reported benchmarking results running the lid-driven cavity test with up to 22 million cells in two systems, in one of then reaching super linear scalability with 1024 cores [1].

3 Test Cases and Experimental Setup An important type of simulation in turbulent regimes is the Large Eddy Simulation (LES), which is becoming very popular because it resolves the most relevant turbulent features within the fluid with high degree of accuracy. LES is based on the concept that smaller scale turbulence is isotropic, while larger turbulent and energetic eddies are the result of the current geometrical configuration. Larger scales are simulated, and small scales are filtered out and the effects are modeled; this process is known as the closure problem [10]. The building blocks of many LES solvers are the transport equations which are the main mechanism to solve many problems in Fluid Dynamics. For a first set of experiments we solved a simplified form of the transport equation for a scalar quantity: dθ + a∇θ = 0. dt

(1)

For a second set we performed a full large eddy simulation (LES). The chosen geometry for our tests was the backward-facing step. This is a sufficiently simple test for the numerical and physical properties and there are enough experimental and numerical results for proper validation [7]. Most importantly, it allows us to discover some features and characteristics of the underlying methodology such as

Investigating the Scalability of OpenFOAM

123

those regarding the numerical stability, domain decomposition strategies, MPI execution characteristics, and the performance of iterative solvers and their scalability. The test case is described in detail in [6] and depicted in Fig. 1. The backwardfacing step has a Reynolds number Re of 4800 with respect to the step height. The dimensionless inlet bulk velocity is 4.8, normed with respect to the geometrical units. The domain is 25 times larger than the step height in the x direction and three times the step height in the z direction.

Fig. 1. The backward facing step geometry. The step size is H, the total length of the domain is 25H, height and depth are 3H. Periodic boundary conditions on the near and far walls are used.

The solver used in our LES experiments is called pisoFoam for LES, OpenFOAM version 1.7.x, and it uses the Pressure Implicit solution by Split Operator method (PISO) [5]. The turbulence modeling is the k-equation eddy-viscosity model with the cube root of cell volume as LES delta. At the sides of the domain periodic boundary conditions are used, while top and bottom were set as non-slip walls. Finally, inlet boundary conditions with an artificial noise of 2% of the velocity value and the outlet with pressure-driven type were specified. Pressure equations were solved using the geometric algebraic multigrid (GAMG) and the Bi-Conjugate Gradient (BiCG) iterative solvers with 3 pressure corrections. Other fields were solved with the BiCG solver. The tests were run over 100 time steps with output at every 50 time steps. In all cases a CFL condition number less than 1.0 was specified. Three meshes with different resolutions were used. The first mesh has 250, 96, and 64 cells in the x, y and z direction, respectively. It contains 2.15 million hexahedrical cells with an expansion ration of 2 in the perpendicular direction resulting in better resolution near the top and bottom walls; cells close to the wall are half the size of cells along the center line. Also, for capturing better flux features, cells were stretched in the x direction close to the step by 50%. The maximal aspect ratio is 5.62 resulting in a mesh of good quality. Runs on 16, 32, 64, 128, and 256 cores were performed on this mesh for a strong scaling study. The second and third meshes contains 4 and 8 times more cells than the first mesh, i.e., 8.62 and 17.20 million cells, respectively. 256 MPI tasks were used on the 8.62 million cell mesh, while 512 MPI tasks on the finer mesh. The number of cells in x, y, and z direction, as well as the expansion ratios, were adjusted to conform to meshes of equivalent characteristics of the first mesh resulting in maximum aspect ratios of 3.90 for the 8.62 million cell mesh, and 4.01 for the 17.20 million cell mesh. The second and third mesh was used in weak scaling studies.

124

O. Rivera, K. F¨urlinger, and D. Kranzlm¨uller

To understand the scalability characteristics of OpenFOAM, we employed IPM (Integrated Performance Monitor)1, which is a portable profiling and workload characterization tool for MPI applications. IPM drastically reduces the overhead caused by application instrumentation and displays very detailed information at the same time [2]. Studying an application with IPM reveals important parameters beyond mere studies of total wallclock scalability. It identifies bottlenecks, detects hot spots and collects statistics that help to optimize representative sections of the code. Despite some challenges posed by the advanced software engineering techniques used by OpenFOAM, IPM was able to instrument OpenFOAM and derive useful data. All our experiments were conducted using a massively parallel general-purpose computer, an SGI Altix 4700, with 9728 Intel Itanium 2 cores, a peak performance of 62.3 TFlops and 19 partitions connected by a high performance NUMA link interconnect [11] installed and operated by the Leibniz Supercomputing Centre (LRZ). All runs, when possible, were restrained to a single shared-memory partition. In case a partition was not large enough, two partitions with the same number of cores were specified. The system is operated by the Leibniz Supercomputing Centre (LRZ).

4 Experimental Results In this section we describe the results of our experiments. We start with the strong scaling study from 16 to 256 tasks (512 tasks for the transport equation) followed by the weak scaling study up to 512 tasks. 4.1 Strong Scaling Study The solution of the simplified transport equation is done by means of the preconditioned conjugate gradient with a diagonal incomplete Cholesky decomposition. We used 16, 32, 64, 128, 256, and 512 MPI tasks for the strong scaling test on the 2.15 million cell mesh. Fig. 2 (left) shows the scalability results of this experiment. Evidently, this setup scales up to 128 tasks, after which a slowdown of the execution occurs. Looking at the contribution from various MPI calls, a large contribution of the MPI Recv routine becomes evident. Fig. 2 (right) shows the time spent in various MPI routines for each rank for the 128 task run. Evidently MPI Recv is the most dominant contributor and the time in MPI Recv is highly variable across ranks. MPI Recv is used by pseudo reduction operations in OpenFOAM and our results suggest that paying more attention to how synchronization values are distributed by OpenFOAM’s internal solvers is warranted. In the rest of this section we focus on the LES test case. For our strong scaling LES study we used 16, 32, 64, 128, and 256 MPI tasks on the 2.15 million cell mesh. All runs had the same setup and the domain decomposition was done with the Metis partitioner in order to have approximately the same number of cells per sub-domain and to minimize the maximum connectivity of the sub-domains [3]. The resulting subdomains are well-balanced and vary at most 3% in their number of cells. Table 1 shows the average number of cells per task for each run. 1

http://www.ipm2.org

Investigating the Scalability of OpenFOAM

MPI_Recv

200

Wall time (sec)

MPI_Waitall

MPI_Allreduce

125

Total MPI

3.5

180 3.0

160 MPI time (sec.)

140 120 100 80 60

2.5 2.0 1.5 1.0

40

0.5

20 0

0.0

16

32

64

MPI Task

128

256

512

0

127 MPI ranks

Fig. 2. Scaling study (left), and distribution of time in various MPI routines (right) for the transport equations test case

In Table 1 we also summarize the wallclock execution time from 16 to 256 MPI tasks as measured by IPM. This table also shows the percentage of execution time spent in MPI and I/O routines. For both MPI and I/O the minimum, maximum, and average percentages over all MPI tasks are listed. Note that minimum and maximum values for MPI and I/O routines do not generally refer to the same MPI rank. Table 1. Wallclock execution time, minimum, maximum, and average fraction of time spent in MPI and I/O routines, respectively, for 16, 32, 64, 128, and 256 MPI tasks MPI Tasks 16 32 64 128 256

#cells Wall Time MPI I/O (avg) (sec) (% min) (% max) (% avg) (% min) (% max) (% avg) 134 400 574.23 6.40 14.43 9.96 0.08 0.51 0.13 67 200 303.82 14.24 26.02 14.24 0.11 0.74 0.14 33 600 197.47 25.98 40.71 31.91 0.23 0.31 0.71 16 800 213.07 27.81 65.07 33.42 0.37 0.91 0.55 8 400 270.00 25.65 69.57 31.70 0.63 1.20 0.90

From Table 1 we see that the pisoFoam solver has an acceptable scalability up to 64 MPI tasks. With 128 and 256 MPI tasks the execution actually slows down. This could be due to several factors, such as a data set that is too small, a suboptimal use of MPI communication facilities, or a slow interconnection network. If we plot the minimum, the maximum and the average percentages of the MPI wall time, we see that for 16, 32, and 64 MPI tasks the average is in the middle of the maximum and minimum. However, for 128 and 256 the average value follows the minimum, while the maximum is much larger (Fig. 3, left). This is a clear indication that one or a few tasks experience significant overheads. IPM does not only measure the overall time spent in MPI, it also gives statistics for individual MPI function calls and data transfer sizes. A comparison of the number of calls (Fig. 3, right) with the aggregate time spent in each function (not shown in Fig. 3) reveals that there is no correlation between these numbers. OpenFOAM’s main communication mechanism used to interchange field values among sub domains is by means of MPI Isend/MPI Irecv/MPI Waitall. As shown in

126

O. Rivera, K. F¨urlinger, and D. Kranzlm¨uller

Fig. 3. Time in MPI as a fraction of wallclock time, plotting minimum, maximum, and average among ranks (left). Number of calls by MPI routine (log scale, right).

Fig. 3 (right), the number of calls of MPI Irecv/MPI Isend is one or two orders of magnitude larger compared to any other MPI function. The number of MPI Waitall calls remains almost constant, or even decreases slightly with the number of tasks. Calls to MPI Allreduce increase but not at the same rate as sends and receives. From Fig. 4 (left) we see a reduction in the relative time used by MPI Isend/MPI Irecv and MPI Allwait when 256 MPI tasks are used, compared to the 64 MPI task run, but the number of calls increases (Fig. 3, right). Smaller communication areas, locality communication patterns and the low latency interconnect explains this behavior. In Fig. 4 (right) we can see that Metis has decomposed the domain with some sub domains divided in more than two parts. Hence, increasing the total number of neighboring sub domains, and, subsequently, the calls of MPI Isend/MPI Irecv can result in load imbalance. As shown in Fig. 3 (right) and Fig. 4 (left), the time in MPI Allreduce is comparatively high considering that only a single scalar value is aggregated. How many times this function is called depends on the type of iterative solver and the number of

Fig. 4. Time spent in various MPI routines as the core count increases (left). Example for the graph-based Metis domain decomposition (right). Rank 0 is shown in dark grey, rank 5 in light gray.

Investigating the Scalability of OpenFOAM

127

iterations this solver requires to converge. For example, the 16 MPI task run, and using the BiCG solver for the pressure, results in 5179 calls for the first time step. By specifying the GAMG solver, the number of calls is 7839. There is, however, no advantage in using the BiCG over the GAMG since the number of iterations needed by BiCG is several hundreds (500–700) compared to the 20–40 used by GAMG solver, and the speed up is three to ten times faster by using the GAMG solver. Hence, trying to reduce the number of MPI Allreduce calls by changing the pressure-correction solver would be a mistake. The same can also be said for MPI Recv, which is not used to interchange values at the inter domain boundary, but to transport single scalar values. Both MPI Allreduce and MPI Recv are being called very infrequently but constitute an important fraction of the MPI time, for 256 MPI tasks more than 9% of time is spent in these routines. MPI Recv is a limiting factor not only because its contribution grows with increasing number of tasks, but also because there is a clustering of a few MPI tasks responsible for this increment. If we recall Fig. 3 (left) for 128 and 256 MPI tasks, the maximum value of the MPI time (as percentage of the wall time) was far away from the average. This is because few ranks were slowing down the whole execution time. Since there is a synchronization at the end of each iteration, equation solution and time step, it is enough that one MPI rank, for whatever reason, slows down to have a very negative impact in the whole application. Fig. 5 can help to understand this behavior. Here the absolute times for MPI Recv, MPI Allreduce and the sum of all MPI routines (ordered by rank for 128 and 256 MPI tasks) are plotted. MPI Allreduce uniformly constitutes about 25% and 32% of the MPI time for 128 and 256 MPI tasks.

Fig. 5. Total MPI time, time in MPI Recv and MPI Allreduce per rank. 128 MPI tasks (top) and 256 MPI tasks (bottom).

128

O. Rivera, K. F¨urlinger, and D. Kranzlm¨uller

MPI Recv behaves differently. In the the 256 task case for example, MPI ranks 0 to 7 have MPI overheads that double the average, primarily caused by the peak in MPI Recv time. For these ranks (0-7) between 61% to 69.57% of the wall time (270 sec.) are spent solely in MPI routines. For 128 tasks the situation is similar, ranks 12 to 15 present the MPI Recv peak (4 ranks), while the rest have a smother distribution. As mentioned before, MPI Recv/MPI Send pairs are mainly used to communicate scalar values, and as such interconnect bandwidth is not an issue here. MPI Recv is also not called more frequently than other functions, in fact compared to MPI Isend/MPI Irecv they are called too few times to argue that latency is a main factor. Whether 128 MPI tasks is an important limit and/or under which conditions is a question that cannot be answered at this point without more sophisticated tracing tools and a deeper understanding of how and when MPI Recv is used by OpenFOAM. 4.2 Weak Scaling Several questions remained unanswered in the previous section, mainly because the mesh with 2 millions of cells was not large enough for runs with more than 256 MPI tasks. Two meshes were prepared, one of 8.62 millions of cells to be run with 256 cores and the second of 17.20 millions of cells to be used with 512 cores. Table 2. Wall time, ranks with the minimum and maximum MPI and I/O time and average (percentage) for 256 and 512 tasks. Both the GAMG and BiCG solver were used for comparison for the 512 runs. MPI Tasks 256 512 512

#cells Solver Wall Time MPI I/O (avg) (sec) (% Min) (% Max) (% Avg) (% Min) (% Max) (% Avg) 33 688 GAMG 486.64 31.2 54.39 36.77 0.39 1.29 0.74 33 600 GAMG 600.95 41.56 68.77 47.29 0.32 0.56 0.88 33 600 BiCG 1893.24 20.01 27.16 23.04 0.11 0.18 0.28

The domain decomposition was done with Metis using the same parameters, as described in the strong scaling section, in order to have approximately 33 000 cells per sub domain. This number proved to be sufficiently large for a reasonable speedup for 64 MPI tasks with the smaller mesh. In Table 2 we list the percentage of time in MPI and I/O for the strong scaling study. Both the GAMG and BiCG solver were used for the 512 runs for comparison. For increased scalability one can specify the BiCG instead of the GAMG, and thus reducing the MPI overhead. However, the GAMG solver, with all its MPI issues, is 3 times faster than the BiCG solver (performance is actually the proper value to be measured and not scalability). This analysis gave us an indication that one should isolate and pay more attention to the GAMG and its specific MPI functionality. MPI Recv and MPI Allreduce are important overheads for GAMG (with 256 and 512 MPI tasks). For the BiCG solver, MPI Allreduce represents an important overhead, 11% which is half of the MPI time. This result agrees with the ones reported in [9], [12], and [4].

Investigating the Scalability of OpenFOAM

129

In previous work [13] we have shown that MPI Recv time is concentrated in a few ranks which slow down the simulation. However, in order to generalize these findings it will be necessary to run tests with more complicated geometries and domains, higher counts of cells per rank with larger domains, and new strategies for domain decomposition.

5 Summary and Outlook Understanding the parallel behavior of OpenFOAM is a complex task due to the internal structure and flexibility that OpenFOAM offers. However, some patterns and specific characteristics have been discovered. We have seen from the results presented that some configurations are more suitable for scaling and under which circumstances bottlenecks may occur. One of the choices for this problem and type of mesh, is to use the GAMG solver. This solver converges rapidly to a solution, faster than BiCG, but its MPI footprint increases and becomes more problematic with a larger number of MPI tasks. We have learned that the scalability of the GAMG solver is limited up to a certain point (for our study 64 MPI tasks). An inflexion point can be seen, however, where the good scalability properties of the BiCG could be used on massively parallel systems. Moreover, scalability alone is not a relevant parameter but overall performance. At this point is difficult to generalize, much more data and test cases with different solvers need to be investigated. Compressible, density based or large-mach-number solvers might have a different MPI signature. In our experiments the major performance issues did not occur for MPI routines with the largest visit counts or data transfers. Instead MPI routines that appeared harmless at first, such as MPI Recv and MPI Allreduce, were identified as problematic. We have also recognized that these problems were concentrated in only a few MPI tasks and that further research is needed to consolidate these observations. In terms of performance, I/O is not an important issue in our experiments. That does not mean there is no opportunity to improve it. In OpenFOAM each MPI instance reads its input and writes its output. If OpenFOAM breaks the barrier of thousands of cores, we will have consequently thousand of files, which is not suitable for any available file system. Lastly, we have to emphasize that all these findings would not have been possible at all, if we did not have profiling and tracing tools. Without these tools we could be optimizing or paying attention to wrong places. IPM was flexible enough to produce the required information (were other tools have failed) at the same time it was concise and informative (where some other tools could have produced huge amounts of redundant information). We hope these tools keep pace with the development in software and hardware, and specifically with the expected scale of next-generation machines. Acknowledgements. This project was in part financially supported the EU reintegration grant MADAME under agreement no. PIRG07-GA2010-268351.

130

O. Rivera, K. F¨urlinger, and D. Kranzlm¨uller

References 1. CSC - IT Center for Science. OpenFOAM - CSC, http://www.csc.fi/english/research/sciences/CFD/CFDsoftware/ openfoam/ofpage 2. F¨urlinger, K., Wright, N.J., Skinner, D.: Effective performance measurement at petascale using IPM. In: Proceedings of The Sixteenth IEEE International Conference on Parallel and Distributed Systems (ICPADS 2010), Shanghai, China (December 2010) 3. Karypis, G., Kumar, V.: MeTis: Unstructured Graph Partitioning and Sparse Matrix Ordering System, Version 4.0 (2009), http://www.cs.umn.edu/˜metis 4. Pringle, G.: Porting OpenFOAM to HECToR A dCSE Project (2010), http://www.hector.ac.uk/cse/distributedcse/reports/openfoam/ openfoam/index.html 5. Jasak, H.: Error Analysis and Estimation for the Finite Volume Method with Applications to Fluid Flow. PhD thesis, Department of Mechanical Enginering, Imperial College of Science, Technology and Medicine (1996) 6. Kobayashi, H., Wu, X.: Application of a local subgrid model based on coherent structures to complex geometries. Center for turbulent research Stanford University and NASA. Annual research brief, pp. 69–77 (2006) 7. Le, H., Moin, P., Kim, J.: Direct numerical simulation of turbulent flow over a backwardfacing step. Journal of Fluid Mechanics 330(1), 349–374 (1997) 8. Weller, H.G., Tabor, G., Jasak, H., Fureby, C.: A tensorial approach to computational continuum mechanics using object orientated techniques. Computers in Physics 12(6), 620–631 (1998) 9. HPC Advisory Council. OpenFOAM Performance Benchmark and Profiling (2010), http://www.hpcadvisorycouncil.com/pdf/OpenFOAM_Analysis_and _Profiling_Intel.pdf 10. Berselli, L.C., Iliescu, T., Layton, W.J.: Mathematics of Large Eddy Simulations of Turbulent Flows, 1st edn., pp. 18–25. Springer, Heidelberg (2005) 11. Leibniz-Rechenzentrum (LRZ). Hardware Description of HLRB II (2009), http://www.lrz.de/services/compute/hlrb/hardware/ 12. Calegari, P., Gardner, K., Loewe, B.: Performance Study of OpenFOAM v1.6 on a Bullx HPC Cluster with a Panasas Parallel File System. In: Open Source CFD Conference, Barcelona, Spain (November 2009) 13. Rivera, O., F¨urlinger, K.: Parallel aspects of openfoam with large eddy simulations. In: Proceedings of the 2011 International Conference on High Performance Computing and Communications (HPCC 2011), Banff, Canada (September 2011)

Shibboleth and Community Authorization Services: Enabling Role-Based Grid Access Fan Gao and Jefferson Tan Faculty of Information Technology, Monash University, VIC 3800 Australia [email protected], [email protected]

Abstract. Classical authentication and authorization in grid environments can become a user management issue due to the flat nature of credentials based on X.509 certificates. While such credentials are able to identify user affiliations, such systems typically leave out a crucial aspect in user management and resource allocation: privilege levels. Shibboleth-based authentication mechanisms facilitate the secure communication of such user attributes within a trust federation. This paper describes a role-based access control framework that exploits Shibboleth attribute handling and CAS (Community Authorization Services) within a Grid environment. Users are able obtain appropriate access levels to resources outside of their domain on the basis of their native privileges and resource policies. This paper describes our framework and discusses issues of security and manageability. Keywords: grids, resource allocation, user management, single sign-on.

1 Introduction Grids are used to “solve unique research problems and to collaborate between different researchers across the globe” [1]. They provide substantial support for research in a variety of applications, but a number of challenges remain, including security. We focus on two types of security services within grids: access control and communication security. Currently, Public Key Infrastructure (PKI) [2] provides the primary authentication mechanism, as in the case of the Grid Security Infrastructure (GSI) [3], where PKI-based X.509 identity certificates are used for authentication. Apart from the ability to employ asymmetric encryption, there is the advantage of using proxy certificates to support Single Sign-On (SSO) [4], allowing for security delegation via the more limited proxy certificates. The notion of SSO is not unique to X.509 certificates, as it is also used in Shibboleth SSO [5], which works within the context of a trust federation. A participating entity entrusts user authentication to another participating entity from which a service is requested. Due to the design of secure transactions to facilitate release of user data to support such trust relationships, we have found an apt application for Shibboleth SSO: role-based access control, translating to role-based authorization. This paper focuses on authentication, authorization and resource access control with Shibboleth and Community Authorization Service (CAS). The goal is to provide access control in large-scale grids where service providers need not rely solely Y. Xiang et al. (Eds.): ICA3PP 2011 Workshops, Part II, LNCS 7017, pp. 131–140, 2011. © Springer-Verlag Berlin Heidelberg 2011

132

F. Gao and J. Tan

on their own basis for the privileges that they grant to remote users. Our scheme supports the determination and application of a remote user’s appropriate authorization level based on local policies. In Section 2, related works are reviewed in light of our work. Section 3 illustrates the overview of the proposed framework’s design. Section 4 discusses the details of this framework, and we evaluate the framework design in Section 5. Finally, the conclusions are presented in Section 6.

2 Background and Related Works The Grid Security Infrastructure (GSI) [3] relies on X.509 and SSL/TLS mechanisms through which a trusted third party can sign, and thus vouch for, either user or service credentials consisting of a public and private key pair and a certificate. This trusted third party is the Certificate Authority (CA), which bears the mutual trust of all participating entities. The four steps in GSI authentication to access the Grid [1] are: 1. 2.

3.

4.

Obtain the CA certificate that contains its public key. The secret key pair is generated at the local host with a certificate request. The latter is signed using the user’s private key, which is used by the CA to verify the certificate’s authenticity. The CA verifies the user‘s information, typically with the assistance of an authorized registration authority operator (RAO). The CA then signs the request using its private key, and releases the signed certificate to the user. Store the CA’s certificate and the signed certificate, which are all used for subsequent authentication transactions.

These certificates (and associated keys) can form the basis access control. From among the three access control models, Mandatory Access Control (MAC), Discretionary Access Control (DAC), and Role-Based Access Control (RBAC), the MAC model is often used in secure grids [6]. Security labels describe some aspect of an entity that can be used to describe access conditions with another entity. Two types of security labels are security classification on an object and security clearance on a user. In the DAC model, users control access to files and other resources that they create, use, or own, and can pass access rights on as preferred. However, the RBAC model seems more appropriate in incorporating the user’s role in deciding to allow access to resources, and roles reflect a user’s privileges. RBAC is considered to be scalable and flexible ([1][7]). It also provides a mechanism to articulate policies instead of being locked into a particular security policy. Shibboleth-based grid resource access control [5] integrates Shibboleth and authorization based on the X.812 Access Control Framework standard [8]. Shibboleth pushes trust out towards local security infrastructures where users are accountable. Each participating organization relies on its own Identity Provider (IdP) to vouch for its users when accessing remote services. An optional Where Are You From (WAYF) service allows users to identify their home organizations prior to being redirected to their own IdP. A Service Provider (SP) integrates pre-existing or new web services, enforcing IdP-based authentication before making access control decisions.

Shibboleth and Community Authorization Services: Enabling Role-Based Grid Access

133

Consequently, access to resources can be decided on the basis of local security policies of the resource provider. Cacheable Decentralized Groups Access Control [9], based on decentralized group membership, relies on a global namespace, and end users could create and manage the decentralized groups by access control lists (ACLs) on shared resources. In Trust Degree Based Access Control, [9] a centralized security server is used to manage and compute a trust value for every access request. The authorization model, based on the eXtensible Access Control Markup Language (XACML) resulted in the Community Authorization Service (CAS) [10]. It promises to improve access control in terms of performance and ease of maintenance [9]. Some issues are crucial in the context of this study. Users are typically not experts in grid computing, and there should be a way to simplify authentication and authorization procedures. Moreover, as grids become increasingly complex, their management would dramatically increase in difficulty as well. Confidentiality is the centre of attention for Resource Providers (RPs), with data resources being commonly confidential in different communities and domains. Scalability affects the performance of GSI. Normally, additional mechanisms increase overheads, as would the storage requirement for user information such as attributes, keys, roles, etc. There are human overheads as well, such as in obtaining a signed certificate from the CA via a registration authority operator. Authentication management such as key production, cancellation, and certificate storage is also becoming a challenge in large-scale environments. There is also the problem of trust. If trust relationships are broken, the whole system could be affected. There are also interoperability issues among several security mechanisms that are individually effective but may not have been designed to fit together. These mechanisms only focus on particular issues or provide only part of the solution. There is no adherence to one complete, end-to-end standard among them.

3 RBAC with Shibboleth and CAS Our work emphasizes that utilizing a user’s role attribute, which Shibboleth-style authentication can supply at sign-on, enhances access control. As per Shibboleth SSO, users authenticate themselves through the IdP of their home organization, which releases one or more role attributes. The same sign-on allows them to obtain credentials from an on-demand CA, which can incorporate the role data into a newly signed certificate. This is used to obtain role-based capabilities from the CAS, which gives the users access to authorized resources at the appropriate level of access. As Figure 1 shows, the proposed RBAC framework includes six major components. The Authentication System is based on Shibboleth. The Grid Service Web Portal is the gateway to the target Grid resource. The Credential Proxy Server generates proxy certificates as needed by an authenticated user. The Certificate Pool System is a novel method of restricting access by using a limited pool of certificates that can be active at any given time. The Authorization Server enforces policies that take user credentials, including role attributes, to make authorization decisions. Grid Resources include the Globus gatekeeper service for launching jobs or the GridFTP service for file transfers. The simple workflow of our framework is as follows:

134

F. Gao and J. Tan

Fig. 1. The proposed RBAC framework with Shibboleth and CAS

1. 2. 3. 4. 5. 6.

7. 8. 9.

The user sends an access request to the Grid Service Web Portal via browser. The portal sends an authentication and attribute request to the user. The Authentication System authenticates the user and releases his role attribute. The user forwards authentication data and the role attribute to the Grid Service Web Portal, which forwards attributes to the Credential Proxy Server. The authenticated user sends an access request to the Credential Proxy Server. The Credential Proxy Server obtains a local certificate for the user from the Certificate Pool System, and generates a proxy credential based on the user’s role attribute, reflecting the appropriate privilege level for the user. The Proxy Server sends the proxy credential to the Authorization Server. The Authorization Server provides/activates the appropriate access capabilities for the user based on the role-based proxy credential. The user may now access Grid resources at the appropriate level of access.

The Community Authorization Service. (CAS), once a component of the Globus Toolkit (GT) for fine-grained access control, is part of our proposed framework. The CAS Server verifies the user’s credentials and generates proxy credentials for the user with specific capabilities based on the user’s role, as well as the security policies stored in the CAS Database. The latter also stores information about users, groups, actions, resources and policies. CAS acts as a broker between user and resource provider. The resource provider also makes final decisions on user access to grid resource, i.e., the resource provider could temporarily reject the user even if the latter has certain authorization to the resources.

Shibboleth and Community Authorization Services: Enabling Role-Based Grid Access

135

Fig. 2. Overview of CAS

Shibboleth. (Figure 3) is open source software developed by the Middleware Architecture Committee for Education - Internet2 (MACE-Internet2). For authentication purposes, Shibboleth is considered to be an extension of the typical local authentication and authorization approach where the user could access multiple resources of any participant of a federation by their local authentication information.

Fig. 3. Shibboleth architecture and workflow, redrawn from Jie et al [4]

The Certificate Pool System (CPS) is a novel system to manage certificates in a local system for remote users. Figure 4 illustrates the architecture of CPS, which has two major components: the Certificate Pool Manager (CPM) and Certificate Pool Database (CPD). CPS stores a certain number of certificates in the CPD, based on the policies set by the owning organization. The certificates are generated and signed by the CA for remote users who want to use local Grid resources. These certificates have shorter lifespans than certificates for local users. The main functions of the CPM are:

136

F. Gao and J. Tan

Fig. 4. Architecture of the Certificate Pool Manager (CPM)

• • •

• • • •

To check the validity of certificates stored in the CPD. To generate certificate status reports for users or administrators. To allocate a certain number of certificates to particular roles. For instance, assume 100 certificates stored in CPD. The CPM can allocate 10 for administrators, 70 for staff and 20 for students. The ratio of certificates for different roles is variable. To control the maximum number of concurrent remote user access to local resources. To respond to certificate requests from legitimate users or entities. To maintain certificates in the CPD, i.e., to destroy a certificate if revoked or if it expires, and to obtain or renew them via the CA when required. When the CPM sends a request to the local CA to sign the certificate, the CPM controls the maximum lifetime of certificates based on some policy.

The CPS can improve authentication and authorization since users do not need personal certificates from the local CA. It increases security by centralized certificate control over concurrent remote users and certificate life spans. It also simplifies the management of certificates for remote users, workload for the local system administrator decreases.

4 Framework Design The framework consists of one part for remote users and one for local users. The following are assumed: • • •

In our prototype, three roles are defined: administrator, staff and student. More sophisticated classifications may well suit other situations. The framework design does not reflect the influence of a firewall. HTTPS is used in all interactions in this framework, to provide SSL/TLS security.

Shibboleth and Community Authorization Services: Enabling Role-Based Grid Access

137

4.1 Framework Design for Remote Users The framework based on the typical Single Sign-On model [12] is redesigned for our own purposes. Figure 5 illustrates seven major components, which we describe below.

Fig. 5. System architecture and workflow for a remote user

The Identity Provider (IdP) provides authentication and authorization services via the user’s home institution. Note that it relies on some underlying security infrastructure, typically an authentication, authorization and accounting (AAA) system for local user accounts and accounting data. The Where Are You From (WAYF) Service is used to redirect a remote user to their home institution’s IdP to begin the authentication process. A Portal is considered a gatekeeper to local system services. It must verify the legitimacy of a remote user and obtain the user’s attributes. The portal’s Assertion Requester generates and presents an authentication request to a user to authenticate through his/her home institution. It is assumed that the local system and the user’s home institution are in the same federation. The Attribute Requester is used to obtain the user’s attributes from his/her home institution based on the local system requirements and delivers these attributes to the local system for subsequent authorization purposes. The

138

F. Gao and J. Tan

GridShib-CA is used to generate local proxy credentials based on the user’s role attributes and certificate, stored in the Certificate Pool System (CPS). The specific Subject Name (SN) (e.g. remote_administrator/01, remote_staff/02 and remote_student/05 etc.) is composed in that proxy credential on account of the user’s role. In effect, the GridShib-CA makes it possible for roles to enter the picture. It sends an attribute request to obtain a user’s attributes from the user’s home institution via the portal. It is also used to request local certificates for remote users from the CPS. The Certificate Pool System (CPS) is used to manage and store local certificates in the certificate pool for remote users. The Community Authorization Service (CAS) provides access control for remote users, based on a user’s Distinguish Name (DN) and the Subject Name (SN) in the proxy credentials, and based on local policies. CAS generates specific capabilities for a remote user, used to access authorized local resources. There is a “capability valid time” notice generated by the CAS to demonstrate a maximum valid time for those capabilities. The Resource Provider (RP) provides resources, e.g., computing resources, storage resources, applications and services. Figure 5 shows 19 steps of authentication and authorization for a remote user. Steps 1-8 are typical authentication steps of Shibboleth via portal for the remote user. Steps 9 through 15 present a procedure to generate role-based local proxy credentials for the remote user. Finally, steps 16 through 19 describe the local resource capability request procedure and access to local Grid resources. 4.2 Framework Design for a Local User In comparison with that for a remote user, the architecture and workflow for a local user is much simpler. For example, the IdP is part of the local system, so there is no need to use the Portal and WAYF service to redirect the user to the IdP, though doing so will still work. In addition, the Certificate Pool System (CPS) is unlikely to apply to local users since they each have local certificates, and local policies may not be restrictive to local users. They may also authenticate directly via the local Authentication System behind the IdP, rather than through the IdP itself.

5 Pros and Cons Single Sign-On allows users to be authenticated by their home institution’s authentication infrastructure. If authenticated, they obtain local proxy credentials and capabilities to repeatedly transact with resources without subsequent sign-on steps. Usability improves for users who are easily authenticated across domains based on the trust relationship within a federation. “Role-grained” Access Control comes from having a user’s role attribute allowing the local system to provide appropriate privileges without maintaining individual accounts for remote users. In a typical grid, resources are typically allocated via one-size-fits-all queues, or via groups to which individual users may be added manually. In our model, CAS policies apply to roles, i.e., “rolegrained” allocation. User authentication is effectively delegated in hierarchical fashion via home institutions, providing scalability. Apart from above benefits, other points of manageability can be made. Each institution needs only to manage its own IdP for its own users. New users or new institutions imply no major changes to local

Shibboleth and Community Authorization Services: Enabling Role-Based Grid Access

139

account management, assuming that generic accounts are already available for remote users in general. Moreover, resource providers define role-based authorization policies for their resources, based on roles rather than per user. The CPS avoids having users approach a registration authority operator prior to the approval of a certificate request. Each institution can autonomously use its own authentication system but still be federated. Different mechanisms such as VOMS [1] and PERMIS [13] could be used locally without impairing interoperability. Privacy is protected in Shibboleth, as local systems request minimal user attributes from the user’s home IdP, and our framework only adds role data to what is normally required. Of course, there are disadvantages as well. First, Shibboleth is a static trust based infrastructure, where each institution must trust each other. If the trust relationship is broken, the whole federation is affected. An unauthorized user who masquerades as a valid remote user can access other institutions in the federation. Moreover, if the IdP releases the wrong role data for a given user, CAS will effectively authorize higher privileges than is appropriate. Second, attributes defined in the remote user’s IdP may not meet the authorization requirement of other institutions. Moreover, if these definitions change at some point, mismatches may arise. Third, CAS does not support users from multiple VOs, as it only supports basic access control policies. Finally, CAS may become a performance bottleneck with only one CAS server. There are other issues to consider. The framework is not particularly designed to withstand attacks. A masquerade attack can let an adversary gain privileged roles. A Denial-of-Service (DoS) attack can bring the CAS server down, making the authorization service unavailable to all. Also, according to the characteristics of Shibboleth, a user who belongs to a specific institution can only possess one role. What if the user belongs to multiple institutions, having different roles? This can be a minor accounting problem, or a bigger one if there is reason to revoke the user’s privileges completely, but the user’s privileges exist across multiple identities with individual roles.

6 Conclusions and Future Work We discussed a proposed role-based access control system that addresses authentication, authorization and decentralized resource allocation and account management in a grid. It integrates Shibboleth and CAS to improve usability, scalability, manageability, privacy and performance of existing authentication and authorization systems using a Single Sign-On, role-based and “role-grained” access control mechanism. In our implementation, it supports workable workflows for remote and local users, and supporting interoperability across different mechanisms. A novel Certificate Pool System was employed to improve security, manageability and performance. Possible extensions to our work include: • • •

a distributed CAS system to improve robustness and scalability; implementing this framework with VOMS [1], PERMIS [13] and other mechanisms; adding support for users with multiple roles in different institutions.

140

F. Gao and J. Tan

References 1. Chakrabarti, A.: Grid computing security. Springer, New York (2007) 2. Gutmarm, P.: PKI: It’s not dead, just resting. Computer 35(8), 41–49 (2002) 3. Foster, I., Kesselman, C., Tsudik, G., Tuecke, S.: Security architecture for computational grids. In: 5th ACM Conf. on Computer and Communications Security (CCS 1998), pp. 83–92. ACM, NY (1998) 4. Jie, W., Arshad, J., Ekin, P.: Authentication and authorization infrastructure for Grids— issues, technologies, trends and experiences. J. Supercomput. 52(1), 82–96 (2010) 5. Sinnott, R.O., Jiang, J., Watt, J., Ajayi, O.: Shibboleth-based access to and usage of grid resources. In: Proc. 7th IEEE/ACM Int. Conf. Grid Computing (Grid 2006), pp. 136–143. IEEE Computer Society, Washington, DC (2006) 6. Daswani, N., Kern, C., Kesavan, A.: Foundations of security: what every programmer needs to know. Apress Media LLC, New York (2007) 7. Pereira, A.L., Muppavarapu, V., Chung, S.M.: Role-based access control for grid database services using the community authorization service. IEEE Trans. Dependable and Secure Computing 3(2), 156–166 (2006) 8. ITU-T Recommendation X.812 | ISO/IEC 10181-3:1996, Security Frameworks for open systems: Access control framework (1996) 9. Hemmes, J., Thain, D.: Cacheable decentralized groups for grid resource access control. In: 7th IEEE/ACM Int. Conf. Grid Computing (Grid 2006), pp. 192–199. IEEE Computer Society, Washington, DC (2006) 10. Ni, X., Luo, J., Song, A.: A trust degree based access control for multi-domains in grid environment. In: 11th Int. Conf. Computer Supported Cooperative Work in Design (CSCWD 2007), pp. 864–869. IEEE, Piscataway (2007) 11. Lang, B., Foster, I., Siebenlist, F., Ananthakrishnan, R., Freeman, T.: A multipolicy authorization framework for grid security. In: 5th IEEE Int. Symp. Network Computing and Applications (NCA 2006), pp. 269–272. IEEE, Los Alamitos (2006) 12. Jensen, J., Spence, D., Viljoen, M.: Grid single sign-on in CCLRC. In: Proc. UK eScience All Hands Meeting 2006, Nottingham, UK. National e- Science Centre, Edinburgh (2006) 13. Chadwick, D., Otenko, A.: The PERMIS X.509 role based privilege management infrastructure. Future Generation Computer Systems 19(2), 277–289 (2003)

A Secure Internet Voting Scheme Md. Abdul Based and Stig Fr. Mjølsnes Department of Telematics Norwegian University of Science and Technology (NTNU) {based,sfm}@item.ntnu.no Abstract. We describe information security requirements for a secure and functional Internet voting scheme. Then we present the voting scheme with multiple parties; this voting scheme satisfies all these security requirements. In this scheme, the voter gets a signed key from the registrar, where the registrar signs the key as blinded. The voter uses this signed key during the voting period. All other parties can verify this signature without knowing the identity of the voter, hence the scheme provides privacy of the voter. This voting scheme also satisfies voter verifiability and public verifiability. In addition, the scheme is receipt-free.

1

Introduction

Internet voting is a very interesting research topic in recent years. Many countries have started deploying Internet voting in their areas, and it has been taken as a very challenging task to have a secure Internet voting scheme. In Internet voting, we want privacy of the voter, and at the same time we want to verify that the ballot cast by a voter has been counted properly. In addition, we do not want to show how the ballot is cast to the vote buyer or coercer. These requirements make the Internet voting a tough topic for the researchers. Regarding physical coercion-free election, it is recommended that the voter will first go to an election booth controlled by the election officials. Then the voter will show some identity to get access to the election booth. After that the voter will vote over Internet. We can make the voting scheme remote so that people can vote from anywhere, but in that case we need to compromise the physical coercion. The voter may be coerced by the vote buyer to vote for a particular candidate in the remote Internet voting. In order to protect the voter to sell the vote, the voting scheme must be receipt-free. We can assume that there is a voter computer [1] inside the election booth to perform the cryptographic task on behalf of the voter. The voter will just choose a vote for a particular candidate and the voter computer will construct the ballot from that vote on behalf of the voter. Then the voter will not be able to sell the vote to the bad people or the coercer. We use bulletin board in the voting scheme to make it verifiable. There are two ways of verification: the voter can verify that the ballot is counted properly and anyone can verify that the counting is correct. The first one refers to voter verifiability and the second one is called public or universal verifiability. Our voting scheme is both voter verifiable and public verifiable. Y. Xiang et al. (Eds.): ICA3PP 2011 Workshops, Part II, LNCS 7017, pp. 141–152, 2011. c Springer-Verlag Berlin Heidelberg 2011 

142

M.A. Based and S.F. Mjølsnes

Outline of the paper: The related work is described in Section 2. The security requirements for a secure and functional Internet voting scheme are presented in Section 3. In Section 4, the voting scheme is presented. The corresponding message sequence diagrams are shown in Section 5. The security analysis is done in Section 6, where the conclusions and future plans are presented in Section 7.

2

Related Work

An Internet voting scheme with multiple parties is presented in [2], where a registrar was proposed to sign a key for the voter that is used during the voting period. But, the register could easily link the key with the identity of the voter in that scheme. We improve this part in our voting scheme such that the registrar signs a blinded copy of the key K. So the registrar has no way to link the key with the identity of the voter. The inductive approach to verifying cryptographic protocols is published by Lawrence et. al. in [3]. They describe protocols inductively as sets of traces, where a trace is a list of communication events. Kremer et. al. present the analysis of an Electronic Voting Protocol in the Applied Pi Calculus in [4]. Groth investigates the existing voting schemes based on homomorphic threshold encryptions and evaluates the security of the schemes in the UC framework in [5], and also proposes some modifications to make the voting schemes secure against active adversaries. The zero-knowledge protocols for voting systems and homomorphic cryptosystems are published in [5,6,7]. A non-interactive zeroknowledge protocol (NIZK) is described in [1]. The authors propose multiple counting servers to count the ballots jointly. The counting servers can individually verify the validity of the ballot without interacting with the voter. They also propose a voter computer to construct the ballot from a vote. We refer to their voter computer and NIZK protocol that can also be used in our voting scheme.

3

Security Requirements

We classify the information security requirements of Internet voting scheme as basic security requirements and expanded security requirements [8]. These requirements are mostly related to each other. 3.1

The Basic Requirements

The basic security requirements are: Eligibility of the Voter. To allow the authenticated voter to cast the ballot, eligibility of the voter is important. To have this, authentication and identification of voter are needed so that only the eligible voters will have access to cast the ballot.

A Secure Internet Voting Scheme

143

Confidentiality of the Ballot. The ballot cast by a voter must be confidential throughout the transmission from the voter to the servers. The servers represent the election officials or authentication servers or counting servers. In most cases, the Public Key Cryptography (PKC) is used in this purpose through the public key encryption techniques. The voter encrypts the ballot using the public key of the servers, so only the servers can decrypt and see the contents of the ballot. Integrity of the Ballot. Integrity of the ballot is to guarantee that the ballot cast by an eligible voter should not be altered or modified by any unauthorized party. To provide integrity, digital signature or group signature [9] techniques are mostly used [10]. Privacy and Secrecy. Privacy refers to both the privacy of the voter and the privacy of the ballot. The privacy should not be compromised in a voting scheme. Secrecy means the way in which a voter casts the ballot should not be revealed to any party [11]. Robustness. The voting scheme should ensure that the components (both the servers and the voter computers) of the voting scheme must be functional during the voting period such that if some components fail then the other components will still work without stopping the voting process. Fairness. The voting scheme should not publish any intermediate results such that no participant gains any knowledge about the partial results before the final tallying. This is important, because, the knowledge of the partial results may affect the intentions of the voters who have not cast their ballots before knowing the partial results to vote for a particular candidate. That is why, a voting scheme should be fair. Soundness. The counting process of a voting scheme should discard a ballot that is not valid. Ballot verification is needed before counting the ballots. Completeness. The counting process of a voting scheme is complete if the counting process counts all the valid ballots. In this case, the final tally is equal to the number of valid ballots cast by the voter. Verifiability of the counting process is important to provide completeness in a voting scheme. Unreuseability of the Ballot. The counting process of a voting scheme should count only one ballot for each eligible voter. The authentication process in some schemes does not allow a voter to vote twice or more, though some voting schemes allow a voter to vote multiple times. But, in that case, the counting process includes only the final ballot cast by the voter. 3.2

The Enhanced Requirements

The enhanced security requirements for Internet voting schemes are: Unlinkability and Untraceability. Unlinkability between the voter and the ballot is important for the privacy of the voter and the ballot. By untraceability neither the voter nor the ballot acquirer can add identifiable information to the ballot. This property supports the unlinkability property.

144

M.A. Based and S.F. Mjølsnes

Validity of the Ballot. The counting process of a voting scheme should validate the ballots before counting them to satisfy soundness, completeness, and unreuseability requirements of the counting process. In Internet voting scheme, we define a ballot as valid if the ballot is cast by an eligible voter and the content of the ballot is also valid. In general, the Zero Knowledge Proof (ZKP) protocols [14] are mainly used for this purpose. The ZKP protocols are classified into two types based on interaction between the prover and the verifier:

. Interactive Zero Knowledge Proof Protocol . Non-interactive Zero Knowledge Proof Protocol

Interactive zero knowledge protocol [7] allows the prover to interact with the verifier and prove the validity of the ballot. The voters play the role as the prover and the counting servers play the role as the verifier in the voting schemes. First, the voter commits to a value of the ballot, and sends the ballot to the servers. Then the servers send a random challenge string to the voter, and the voter responds to the challenge string. The verifiers then verify the ballot by verifying the responses. Like interactive zero-knowledge protocol, there is no interaction between the prover and the verifier in a Non-Interactive Zero-Knowledge (NIZK) protocol [10]. In NIZK, the verifier verifies the ballot without interacting with the prover. A NIZK requires no online communication between the prover and the verifier, hence usually it is faster and more efficient. On the other hand, a NIZK protocol requires that both the prover and verifier share a common random string (as challenge string), usually provided by a trusted third party and a pre-arranged use of this random string is required. Verifiability. Verifiability refers to the verifiability of the counting process. It is divided into two categories:

. Universal Verifiability or Public Verifiability . Voter Verifiability

In universal verifiability or public verifiability, any observer can verify that the counting process contains all the valid ballots and the final tally is correct [6]. On the other hand, if only the voters can verify the counting process and the tally, then the voting scheme is called voter verifiable [1,12]. In the scheme [1], the counting servers publish the final tally and the nonce values that were added to the ballot. The scheme is voter verifiable, since every voter can verify the nonce values. Receipt-freeness. In Internet voting, receipt-freeness [13] is another challenging requirement introduced by Benaloh and Tuinstra [15] to provide coercionresistance. Receipt-freeness means, in a voting scheme, the voters should not be allowed to produce a receipt of the ballot to prove for which candidate the ballot is cast. To protect vote-buying by the coercer or the vote buyer this is very important.

A Secure Internet Voting Scheme

145

Fig. 1. The Voting Scheme

Coercion-resistance. Coercion-resistance is achieved through receipt-freeness if the voting schemes do not allow the vote-buyer or coercer to enforce a voter to vote for a particular candidate. Re-voting or multiple voting [1] by a voter are allowed in some schemes to make them coercion-resistant. To protect the voter from physical coercion, election booth based voting schemes are recommended. In case of remote Internet voting, there is no cryptographic way to make it free from physical coercion.

4

The Voting Scheme

Fig. 1 shows the voting scheme. In this scheme, the voter interacts with a registrar to get a signed key and uses that key for voting. Then the voter sends the ballot to the ballot acquirer together with this signed key. The ballot acquirer verifies the signature of the registrar on this key and sends the ballot to the counting servers after voting period is over. The ballot acquirer signs the key and sends the key back to the voter. The voter publishes this key on the bulletin board. The counting servers also sign the key and publish the signed key on the bulletin board. So the voter and any observer can verify that the number of keys signed by the counting servers is equal or less than to the number of keys signed by the ballot acquirer. 4.1

The Parties

The various parties in the voting scheme are as follows:

. Voter - A voter is an eligible one to vote. We assume that the voter holds a smartcard containing some private information that identifies the eligibility

146

.

. .

.

.

4.2

M.A. Based and S.F. Mjølsnes

of the voter to the registrar. The voter creates a private/public key pair, where the public key is to be signed by the registrar so that the voter can use this key during the voting period. After the voter has chosen the candidate to vote for, the choice is encoded, and the ballot is split into shares and sent to the ballot acquirer. We do not describe the ballot construction here, rather refer to [1]. Voter Computer - Voter Computer (VC) is used to perform the cryptographic task on behalf of the voter to construct the ballot from a vote. We assume that VC is trusted and operates inside an election booth. The voter cannot see the cryptographic detail of the construction of a ballot, so cannot sell the ballot to the vote buyer or coercer. Registrar - The role of the registrar is to validate the eligibility of the voter and to sign the blinded copy of the public key supplied by the voter. Ballot Acquirer - The role of the ballot acquirer is to acquire the ballot from the voter and to verify the signature of the registrar on the key included with the ballot. After the ballot acquisition stage, the ballot acquirer mixes all the ballots and sends these to the counting servers. To defend coercion, a voter may vote multiple times. In that case, the ballot acquirer keeps a record and sends the latest ballot of the voter to the counting servers. Bulletin Board - The content of the bulletin board is public. This board is used to provide voter verifiability and universal verifiability. After voting, the voter sends the signed key (signed by the ballot acquirer) to the bulletin board. After counting, the same key is published by the counting servers. All counting servers will publish the same key, so no single counting server can alter or delete a key without detection. The voter can verify that the key signed by the ballot acquirer is the same key published by the counting servers. Any observer can observe that the number of keys signed by the ballot acquirer must be equal to or larger than the number of keys published by the counting servers. Because, the counting servers may ignore some invalid ballots after ballot verification. In that case, the key will not be published. But, the counting servers can maintain a list of the keys of the discarded ballots (if needed) for future claims by the voters. Counting Servers - A group of counting servers verifies that each individual ballot is correctly constructed and computes the result of the election using multiparty computations. No single counting server gets complete information about the individual ballot, but together they reach a result that all counting servers agree upon. Communication Model

We assume that the bulletin board is implemented as a centralized server and the contents of the bulletin board are public. The voter and the counting servers communicate with the bulletin board over the Internet. The counting servers are the same role, but different parties, the communication between them is done over authenticated and encrypted channels.

A Secure Internet Voting Scheme

147

Fig. 2. The Protocol between the Voter and the Registrar

The communication between the voter and the registrar is interactive over the Internet. The implementations of the Registrar, Ballot Acquirer, Counting Servers, Bulletin Board are owned, developed and deployed by different trusted parties and all Internet connections are authenticated and confidential based on standard technology such as TSL/SSL. Each party is assumed to have a public key known by all roles and have a private key to sign message with.

5

The Protocol

The protocol is shown as message sequence diagrams in this section. The voter shows some valid identity to enter the election booth and generates the key pair K, K −1 where K −1 is the private key and K is the corresponding public key. 5.1

The Protocol between the Voter and the Registrar

The protocol between the voter and the registrar is shown in Fig. 2. The voter sends a signed and encrypted blinded copy (the blinding scheme is not shown in detail here) of K to the registrar. That is, the voter sends [K v−1 ]R to the registrar. Here, K is the blinded copy of K, signed with the private key of the voter V −1 , and encrypted with the public key of the registrar R. The registrar verifies the signature of the voter, and if the voter is valid and not voted before, the registrar signs the blinded K with its private key R−1 , encrypts it with the public key of the voter V , and sends this signed and encrypted blinded key to the voter. Then the voter can unblind the message and gets a signed public key. 5.2

The Protocol between the Voter and the Ballot Acquirer

The protocol between the voter and the Ballot Acquirer is shown in Fig. 3. The voter generates the ballot (we can assume that a voter computer [1] will perform the cryptographic task to construct a ballot from a vote so that the voter does not know the cryptographic details of the vote), signs it with the private key K −1 , adds the signed KR−1 to the ballot, and encrypts it with the

148

M.A. Based and S.F. Mjølsnes

Fig. 3. The Protocol between the Voter and the Ballot Acquirer

Fig. 4. The Protocol between the Voter and the Bulletin Board

public key of the counting server (CS). The voter then adds the signed key again (KR−1 ) with this message and encrypts the message with the public key of the ballot acquirer (BA). The voter sends this signed and encrypted ballot to the ballot acquirer. The ballot acquirer can verify the signature on the key after decrypting the message. The ballot acquirer signs the key and sends the signed key KBA−1 to the voter. 5.3

The Protocol between the Voter and the Bulletin Board

The protocol between the voter and the bulletin board is shown in Fig. 4. The voter sends the signed key KBA−1 to the bulletin board. 5.4

The Protocol between the Ballot Acquirer and the Counting Servers

The protocol between the Ballot Acquirer and the Counting Servers is shown in Fig. 5. After verifying the signature on the key K, the ballot acquirer signs the encrypted message received from the voter with its private key BA−1 , and sends it to the counting server. The counting server only receives ballots signed by the ballot acquirer.

A Secure Internet Voting Scheme

149

Fig. 5. The Protocol between the Ballot Acquirer and the Counting Servers

Fig. 6. The Protocol between the Counting Servers and the Bulletin Board

5.5

The Protocol between the Counting Servers and the Bulletin Board

The protocol between the Counting Servers and the Bulletin Board is shown in Fig. 6. The counting servers (we assume that there are multiple counting servers to count the ballots [1]) decrypt the message and count the ballots and publish the tally. The counting servers also sign and publish the signed key KCS −1 on the bulletin board (BB).

6

Security Analysis

This section analyzes (informal) the security properties of the voting scheme presented in this paper. The voting scheme satisfies the following: Eligibility. The voter shows the identity to the election officials to enter the election booth and then sends the signed key (as blinded) to the registrar. The registrar verifies the signature of the voter. So, only the eligible voter can vote. Confidentiality and Integrity of the Ballot. In our scheme, the ballot is encrypted with the public key of the counting servers and signed using the private key K −1 of the voter. Only the counting servers can decrypt and see the content of the ballot. The counting servers can also verify the key KR−1 by verifying the signature of the registrar on this key (because the key must be signed by the registrar). This provides confidentiality and integrity of the ballot.

150

M.A. Based and S.F. Mjølsnes

Privacy and Secrecy. In our scheme, the voter first enters an election booth controlled by the election officials by showing some identity and credentials. Then the voter casts the vote inside the booth. No one knows any information how the voter casts the ballot inside the election booth. Using the blind signature and ballot encryption (encryption with the public key of the counting servers) techniques, our scheme provides privacy of the voter and the ballot. Robustness and Fairness. There are multiple counting servers in our voting scheme. As long as the threshold number of counting servers are functional, the voting scheme will work. No intermediate results should be published before the final tally. Since there are multiple counting servers to count the final tally, it is not possible for a single counting server to publish any intermediate result. The counting servers work together to count and publish the result of the voting. Soundness, Completeness, and Unreuseability of the Ballot. These properties are achieved through ballot verification, voter verifiability and universal verifiability. Unlinkability and Untraceability. In our scheme, the voter sends a signed (signed by the voter) blinded copy of the key K to the registrar. The registrar verifies the signature and signs the blinded copy of the key. Then the registrar sends this signed (signed by the registrar) blinded key to the voter. The voter unblinds it, gets the signed key and uses this key during the voting period. The registrar has no knowledge about the key, so it can not link the voter with this key. So, no one can add identifiable information to the ballot in our scheme. Validity of the Ballot. Regarding the validity of the ballot by a set of counting servers, our scheme is similar to the scheme presented in [10]. By using the noninteractive zero-knowledge protocol [10] the counting servers can individually verify the validity of the ballot. Voter Verifiability. After voting, the voter receives the signed key from the ballot acquirer (as shown in Fig. 4). This means that the ballot acquirer has received the ballot. The counting servers also publish the same key signed by these servers on the bulletin board. The voter can easily verify that the key included by the voter in the ballot is published by the counting servers. Universal Verifiability. Any observer can observe the contents on the bulletin board. The counting servers publish the signed key (signed by the counting servers) after ballot verification and counting. The voter publishes the signed key (signed by the ballot acquirer) after the ballot has been received by the ballot acquirer. The number of keys published by the counting servers must be equal to or less than the number of keys signed by the ballot acquirer (the counting servers may discard some invalid ballots after ballot verification). Since all the counting servers will publish the same key, no single counting server can alter or delete a key without detection.

A Secure Internet Voting Scheme

151

Receipt-freeness and Coercion-resistance. We assume a voter computer inside the election booth that performs the cryptographic task on behalf of the voter to construct the ballot from a vote. The voter does not know the detail construction of the ballot, so the voter can not prove to anyone how the voting was done. We assume this voter computer as a trusted ballot generator. Thus, the voting scheme is receipt-free and hence, coercion-resistant.

7

Conclusions and Future Plans

In this paper, we first present the security requirements for a secure Internet voting scheme. We then present a new scheme that satisfies all these security requirements. The scheme involves multiple parties for voter authentication (by registrar) and ballot generation (by voter computer), and multiple counting servers to count the ballots. We do not need to trust all the counting servers. As long as a single counting server is honest, the ballot verification and counting will be correct in our voting scheme. This essentially increases public trust in ballot counting. The voter and any observer can verify that the counting is correct by observing the contents of the bulletin board. Thus, voter verifiability and public verifiability are also satisfied. In summary, we can say that, we have presented a voting scheme here that satisfies all the basic and enhanced security requirements for Internet voting schemes. We have analyzed the security properties of our voting scheme informally. Ongoing work involves verification of these security properties by using formal verification tool, for example, Isabelle [3]. After implementation of some parts of the scheme (counting), performance evaluation is an important future work. Acknowledgement. The authors are thankful to Peter Ryan, University of Luxemberg, David Gray and Denis Butin, Dublin City University, for their valuable comments and suggestions regarding the voting scheme presented in this paper.

References 1. Based, M. A., Mjølsnes, S.F.: Universally Composable NIZK Protocol in an Internet Voting Scheme. In: Cuellar, J., et al. (eds.) STM 2010, LNCS, vol. 6710, pp. 147– 162. Springer, Heidelberg (2011) 2. Based, M.A., Reistad, T.I., Mjølsnes, S.F.: Internet Voting using Multiparty Computations. In: Proceedings of the the 2nd Norwegian Security Conference (NISK 2009), Tapir Akademisk Forlag, pp. 136–147 (2009) ISBN: 978-82-519-2492-4 3. Paulson, L.C.: The Inductive Approach to Verifying Cryptographic Protocols. Journal of Computer Security (2000) 4. Kremer, S., Ryan, M.: Analysis of an Electronic Voting Protocol in the Applied Pi Calculus. In: Sagiv, M. (ed.) ESOP 2005. LNCS, vol. 3444, pp. 186–200. Springer, Heidelberg (2005)

152

M.A. Based and S.F. Mjølsnes

5. Groth, J.: Evaluating Security of Voting Schemes in the Universal Composability Framework. Springer, Heidelberg (2004) ISBN 978-3-540-22217-0 6. Schoenmakers, B.: A Simple Publicly Verifiable Secret Sharing Scheme and its Application to Electronic Voting. In: Wiener, M. (ed.) CRYPTO 1999. LNCS, vol. 1666, pp. 148–164. Springer, Heidelberg (1999) 7. Iversen, K.R.: The Application of Cryptographic Zero-Knowledge Techniques in Computerized Secret Ballot Election Schemes. Ph.D. dissertation, IDT-report, 1991:3, Norwegian Institute of Technology (February 1991) 8. Meng, B.: Analyzing and Improving Internet Voting Protocol. In: Proceedings of the IEEE International Conference on e-Business Engineering, pp. 351–354. IEEE Computer Society, Los Alamitos (2007) ISBN 0-7695-3003-6 9. Boyen, X., Waters, B.: Compact Group Signatures Without Random Oracles. In: Vaudenay, S. (ed.) EUROCRYPT 2006. LNCS, vol. 4004, pp. 427–444. Springer, Heidelberg (2006) 10. Based, M.A., Mjølsnes, S.F.: A Non-interactive Zero Knowledge Proof Protocol in an Internet Voting Scheme. In: Proceedings of the the 2nd Norwegian Security Conference (NISK 2009), Tapir Akademisk Forlag, pp. 148–160 (2009) ISBN:97882-519-2492-4 11. Gray, D., Sheedy, C.: E-Voting: a new approach using Double-Blind Identity-Based Encryption. Presented in STM 2010: 6th International Workshop on Security and Trust Management, Athens, Greece (September 23-24, 2010) 12. Chaum, D.: Secret-ballot receipts: True voter-verifiable elections. IEEE Security and Privacy 2(1), 38–47 (2004) 13. Hirt, M., Sako, K.: Efficient Receipt-Free Voting Based on Homomorphic Encryption. In: Preneel, B. (ed.) EUROCRYPT 2000. LNCS, vol. 1807, pp. 539–556. Springer, Heidelberg (2000) 14. Krantz, S.G.: Zero Knowledge Proofs. AIM Preprint Series, 2007-46 (July 25, 2007) 15. Benaloh, J., Tuinstra, D.: Receipt-free secret-ballot elections. In: Proceeding of STOC 1994, pp. 544–553 (1994)

A Hybrid Graphical Password Based System Wazir Zada Khan1, Yang Xiang2, Mohammed Y. Aalsalem1, and Quratulain Arshad1 1

School of Computer Science, Jazan University, Saudi Arabia {wazirzadakhan,aalsalem.m}@jazanu.edu.sa, [email protected] 2 School of Information Technology, Deakin University, Australia [email protected]

Abstract. In this age of electronic connectivity, where we all face viruses, hackers, eavesdropping and electronic fraud, there is indeed no time when security is not critical. Passwords provide security mechanism for authentication and protection services against unwanted access to resources. A graphical based password is one promising alternatives of textual passwords. According to human psychology, humans are able to remember pictures easily. In this paper, we have proposed a new hybrid graphical password based system, which is a combination of recognition and recall based techniques that offers many advantages over the existing systems and may be more convenient for the user. Our scheme is resistant to shoulder surfing attack and many other attacks on graphical passwords. This resistant scheme is proposed for small mobile devices (like smart phones i.e. ipod, iphone, PDAs etc) which are more handy and convenient to use than traditional desktop computer systems. Keywords: Graphical passwords, Authentication, Network Security.

1 Introduction A password is a secret that is shared by the verifier and the customer. ”Passwords are simply secrets that are provided by the user upon request by a recipient.” They are often stored on a server in an encrypted form so that a penetration of the file system does not reveal password lists [2]. Passwords are the most common means of authentication because they do not require any special hardware. Typically passwords are strings of letters and digits, i.e. they are alphanumeric. Such passwords have the disadvantage of being hard to remember [3]. Weak passwords are vulnerable to dictionary attacks and brute force attacks where as Strong passwords are harder to remember. To overcome the problems associated with password based authentication systems, the researchers have proposed the concept of graphical passwords which use pictures instead of textual passwords and are partially motivated by the fact that humans can remember pictures more easily than a string of characters [4].Graphical passwords have been known from the mid 1990s. The idea of graphical passwords was originally described by Greg Blonder in 1996 [5]. The first and most important advantage is that they are easier to remember than textual passwords. Human beings have the ability to remember faces of people, places they visit and things they have

Y. Xiang et al. (Eds.): ICA3PP 2011 Workshops, Part II, LNCS 7017, pp. 153–164, 2011. © Springer-Verlag Berlin Heidelberg 2011

154

W.Z. Khan et al.

seen for a longer duration. Thus, graphical passwords provide a means for making more user-friendly passwords while increasing the level of security. Besides these advantages, the most common problem with graphical passwords is the shoulder surfing problem: an onlooker can steal user’s graphical password by watching in the user’s vicinity. Many researchers have attempted to solve this problem by providing different techniques [6]. Due to this problem, most graphical passwords schemes recommend small mobile devices (PDAs) as the ideal application environment. Another common problem with graphical passwords is that it takes longer to input graphical passwords than textual passwords [6]. The login process is slow and it may frustrate the impatient users. Graphical passwords serve the same purpose as textual passwords differing in consisting of handwritten designs (drawing), possibly in addition to text. The exploitation of smart phones like ipod and PDA’s is increased due to their small size, compact deployment and low cost. In this paper, considering the problems of text based password systems, we have proposed a new graphical password scheme which has desirable usability for small mobile device. Our proposed system is new graphical passwords based hybrid system which is a combination of recognition and recall based techniques and consists of two phases. During the first phase called Registration phase, the user has to first select his username and a textual password. Then objects are shown to the user to select from them as his graphical password. After selecting the user has to draw those selected objects on a touch sensitive screen using a stylus. During the second phase called Authentication phase, the user has to give his username and textual password and then give his graphical password by drawing it in the same way as done during the registration phase. If they are drawn correctly the user is authenticated and only then he/she can access his/her account. For practical implementation of our system we have chosen i-mate JAMin smart phone which is produced by HTC, the Palm Pilot, Apple Newton, Casio Cassiopeia E-20 and others which allow users to provide graphics input to the device. It has a display size of 240x320 pixels and an important feature of Handwriting recognition. The implementation details are out of the scope of this paper. Rest of the paper is organized as follows. In Section II, all existing graphical password based schemes are classified into four main categories. Section III gives a little review of existing research and schemes which are strongly related to our work. Section VI discusses the problems of all existing graphical password based schemes. In Section V our proposed system is described in detail. In Section VI we have compared our proposed system with existing schemes by drawing out the flaws in existing schemes. Section VII provides discussion. Finally Section VIII concludes the paper.

2 Classification of Graphical Password Based Systems Graphical passwords schemes can be broadly classified into four main categories: recognition-based, cued-recall based, pure-recall based and hybrid systems. Recognition Based Systems are also known as Cognometric Systems or Search metric Systems. Recognition based techniques involves identifying whether one has seen an image before. The user must only be able to recognize previously seen images, not generate them unaided from memory.

A Hybrid Graphical Password Based System

155

Pure Reacll Based Systems are also known as Drwanmetric Systems. In pure recallbased methods the user has to reproduce something that he or she created or selected earlier during the registration stage. Cued Recall Based Systems are also called Iconmetric Systems. In cued recall-based methods, a user is provided with a hint so that he or she can recall his his/her password. Hybrid Systems are typically combination of two or more schemes. Like recognition and recall based or textual with graphical password schemes.

3 Related Work Haichang Gao et al. [10] have proposed and evaluated a new shoulder-surfing resistant scheme called Come from DAS and Story (CDS) which has a desirable usability for PDAs. This scheme adopts a similar drawing input method in DAS and inherits the association mnemonics in Story for sequence retrieval. It requires users to draw a curve across their password images (pass-images) orderly rather than click directly on them. The drawing method seems to be more compatible with people’s writing habit, which may shorten the login time. The drawing input trick along with the complementary measures, such as erasing the drawing trace, displaying degraded images, and starting and ending with randomly designated images provide a good resistance to shoulder surfing. A user study is conducted to explore the usability of CDS in terms of accuracy, efficiency and memorability, and benchmark the usability against that of a Story scheme. The main contribution is that it overcomes a drawback of recallbased systems by erasing the drawing trace and introduces the drawing method to a variant of Story to resist shoulder-surfing. P.C.van Oorshot and Tao Wan [1] have proposed a hybrid authentication approach called Two Step. In this scheme users continue to use text passwords as a first step but then must also enter a graphical password. In step one, a user is asked for her user name and text password. After supplying this, and independent of whether or not it is correct, in step two, the user is presented with an image portfolio. The user must correctly select all images (one or more) pre-registered for this account in each round of graphical password verification. Otherwise, account access is denied despite a valid text password. Using text passwords in step one preserves the existing user sign-in experience. If the user’s text password or graphical password is correct, the image portfolios presented are those as defined during password creation. Otherwise, the image portfolios (including their layout dimensions) presented in first and a next round are random but respectively a deterministic function of the user name and text password string entered, and the images selected in the previous round.

4 Problem Domain There are many problems with each of the graphical based authentication methods. These are discussed below:

156

W.Z. Khan et al.

4.1 Problems of Recognition Based Methods Dhamijia and Perrig proposed a graphical password based scheme Déjà Vu, based on Hash Visualization technique [11]. The drawback of this scheme is that the server needs to store a large amount of pictures which may have to be transferred over the network, delaying the authentication process. Another weakness of this system is that the server needs to store the seeds of portfolio images of each user in plaintext. Also, the process of selecting a set of pictures from picture database can be tedious and time consuming for the user [7]. This scheme was not really secure because the passwords need to store in database and that is easy to see. Sobrado and Birget developed a graphical password technique that deals with the shoulder surfing problem [3]. In their first scheme the system displays a number of passobjects (pre-selected by user) among many other objects as shown in Fig: 3. To be authenticated, a user needs to recognize pass-objects and click inside convex hull formed by all the pass objects. They developed many schemes to solve the shoulder surfing problem but the main drawback of these schemes is that log in process can be slow. Another recognition based technique is proposed by Man et al. He proposed a shoulder-surfing resistant algorithm which is similar to that developed by Sobrado and Birget. The difference is that Man et al has introduced several variants for each pass-object and each variant is assigned a unique code. Thus during authentication the user recognize pre-selected objects with an alphanumeric code and a string for each pass-object. Although it is very hard to break this kind of password but this method still requires the user to memorize alphanumeric codes for each pass-object variants. “Passface” is another recognition based system. It is argued by its developer that it is easy for human beings to remember human faces than any other kind of passwords. But Davis et al [12] have found that most users tend to choose faces of people from the same race. This makes the Passface password somewhat predictable. Furthermore, some faces might not be welcomed by certain users and thus the login process will be unpleasant. Another limitation of this system is that it cannot be used by those people who are face-blind [6]. 4.2 Problems of Recall Based Methods The problem with the Grid based methods is that during authentication the user must draw his/her password in the same grids and in the same sequence. It is really hard to remember the exact coordinates of the grid. The problem with Passlogix is that the full password space is small. In addition a user chosen password might be easily guessable [6]. DAS scheme has some limitations like it is vulnerable to shoulder surfing attack if a user accesses the system in public environments, there is still a risk for the attackers to gain access to the device if the attackers obtained a copy of the stored secret, and, brute force attacks can be launched by trying all possible combinations of grid coordinates, ) Drawing a diagonal line and identifying a starting point from any oval shape figure using the DAS scheme itself can be a challenge for the users, and finally Difficulties might arise when the user chooses a drawing which contains strokes that pass too close to a grid-line, thus, the scheme may not be able to distinguish which cell the user is choosing.

A Hybrid Graphical Password Based System

157

“PassPoints” is the extended version of Blonder’s idea by eliminating the predefined boundaries and allowing arbitrary images to be used. Using this scheme it takes time to think to locate the correct click region and determine precisely where to click. Another problem with these schemes is that it is difficult to input a password through a keyboard, the most common input device; if the mouse doesn’t function well or a light pen is not available, the system cannot work properly [6]. Overall, with both “PassPoints” and “Passlogix”, looking for small spots in a rich picture might be tiresome and unpleasant for users with weak vision. In Viskey’s scheme’s main drawback is the input tolerance. Pointing to the exact spots on the picture has proven to be quite hard thus Viskey accepts all input within a certain tolerance area around it. It also allows users to set the size of this area in advance. However, some caution related to the input precision needs to be taken, since it will directly influence the security and the usability of the password. In order to practically set parameters, a four spot VisKey theoretically provides approximately 1 billion possibilities for defining a password. Unfortunately this is not large enough to prevent off-line attacks from a high-speed computer. Therefore no less than seven defined spots are required to overcome the likelihood of brute force attacks.

Fig. 1. A shoulder Surfing Resistant Graphical Password Scheme [3]

5 Proposed System Taking into account all the problems and limitations of graphical based schemes, we have proposed a hybrid system for authentication. This hybrid system is a mixture of both recognition and recall based schemes. Our proposed system is an approach towards more reliable, secure, user-friendly, and robust authentication. We have also reduced the shoulder surfing problem to some extent. 5.1 Proposed Algorithm Steps 1-3 are registration steps and steps 4-9 are the authentication steps. The algorithm of our proposed system is as follows:

158

W.Z. Khan et al.

─ Step 1 The first step is to type the user name and a textual password which is stored in the database. During authentication the user has to give that specific user name and textual password in order to log in. ─ Step 2 In this second step objects are displayed to the user and he/she selects minimum of three objects from the set and there is no limit for maximum number of objects. This is done by using one of the recognition based schemes. The selected objects are then drawn by the user, which are stored in the database with the specific username. Objects may be symbols, characters, auto shapes, simple daily seen objects etc. Examples are shown in Figure 2. ─ Step 3 During authentication, the user draws pre-selected objects as his password on a touch sensitive screen (or according to the environment) with a mouse or a stylus. This will be done using the pure recall based methods. ─ Step 4 In this step, the system performs pre-processing. ─ Step 5 In the fifth step, the system gets the input from the user and merges the strokes in the user drawn sketch. ─ Step 6 After stroke merging, the system constructs the hierarchy. ─ Step 7 Seventh step is the sketch simplification. ─ Step 8 In the eighth step three types of features are extracted from the sketch drawn by the user. ─ Step 9 The last step is called hierarchical matching. During registration, the user selects the user name and a textual password in a conventional manner and then chooses the objects as password. The minimum length for textual password is L=6. Textual password can be a mixture of digits, lowercase and uppercase letter. After this the system shows objects on the screen of a PDA to select as a graphical password. After choosing the objects, the user draws those objects on a screen with a stylus or a mouse. Objects drawn by the user are stored in the database

A Hybrid Graphical Password Based System

159

Fig. 2. Some examples of objects shown to the user

with his/her username. In object selection, each object can be selected any number of times. Flow chart of registration phase is shown in Figure 3. During authentication, the user has to first give his username and textual password and then draw pre-selected objects. These objects are then matched with the templates of all the objects stored in the database. Flow chart of authentication phase is shown in Figure 4. The phases during the authentication like the pre-processing, stroke merging, hierarchy construction, sketch simplification, feature extraction, and hierarchical matching are the steps proposed by Wing Ho Leung and Tsuhan Chen in their paper [13]. They propose a novel method for the retrieval of hand drawn sketches from the database, finally ranking the best matches. In the proposed system, the user will be authenticated only if the drawn sketch is fully matched with the selected object’s template stored in the database. Pre-processing of hand drawn sketches is done prior to recognition and normally involves noise reduction and normalization. The noise occur in the image by user is generally due to the limited accuracy of human drawn images. [14]. A number of techniques can be used to reduce noise that includes Smoothing, filtering, wild point correction etc. Here in the proposed system Gaussian smoothing is used which eliminates noise introduced by the tablet or shaky drawing.

or specifically in two dimensions

Where r is the blur radius (r2 = u2 + v2), and σ is the standard deviation of the Gaussian distribution. In case, if user draws very large or a very small sketch then the system performs size normalization which adjusts the symbols or sketches to a standard size. The Stroke merging phase is use to merge the strokes which are broken at end points. If the end points are not close, then that stroke is considered as open stroke and it may

160

W.Z. Khan et al.

be merged with another open stroke if the end point of one stroke is close to the end point of the other. The strokes are then represented in a hierarchy to simplify the image and to make it meaningful for further phases [13]. In the next step of sketch simplification, a shaded region is represented by a single hyper-stroke. After sketch simplification three types of features are extracted from the user re-drawn sketch. These features are hyper stroke features, Stroke features, and bi-stroke features.

Fig. 3. Flow chart for Registration Phase

In the last step of hierarchical matching, the similarity is evaluation the top to bottom hierarchical manner. The user is allowed to draw in an unrestricted manner. The overall process is difficult because free hand sketching is a difficult job. The order in which the user has selected the objects does matter in our proposed system i.e. during the authentication phase, the user can draw his pre-selected objects in the same order as he had selected during the registration phase. So, in this way the total combinations of each password will be 2n –1, ‘n’ being the number of objects selected by the user as password during the registration phase.

A Hybrid Graphical Password Based System

161

6 Comparison of Proposed System with Existing Schemes Our system offers many advantages over other existing systems as discussed below: Comparing to the “Passface” system, our system can also be used for those who are face-blind. We have used objects instead of human faces for selecting as password because later on during the authentication phase, the user has to draw his/her password and it is a much more difficult task to draw human faces than simple objects. Also we believe that as compared to human faces, objects are easier to remember which are in daily use. Our system has eliminated the problems with grid based techniques where the user has to remember the exact coordinates which is not easy for the user. Our system just compares the shapes of the objects drawn by the user during authentication. Our scheme is less vulnerable to Brute force attack as the password space is large. It is also less vulnerable to online and offline dictionary attacks. Since

Fig. 4. Flow Chart for Authentication Phase

162

W.Z. Khan et al.

stylus is used, it provides ease to the user for drawing objects and also it will be impractical to carry out dictionary attack. Our scheme is better than Man et al scheme. This is because in his scheme the user has to remember both the objects and string and the code. In our method the user has to remember the objects he selected for password and also the way he has drawn the objects during registration. Our proposed system differs from CDS in that the user has to first select a textual password and then a graphical password, making it more secure. Comparing to Van Oorschot’s system Two Step Authentication system, our system is more secure since users not only select graphical password but also draw their password, making it difficult to hack and even if the textual password is compromised, the graphical password cannot be stolen or compromised since the user is also drawing the graphical password. Our proposed system works in the same way as Two Step Authentication system i.e the user has to choose a textual password before choosing a graphical password but difference is that in our system during authentication, after giving the username and textual password, the user has to draw his graphical password which is matched with its stored template drawn by the user during the registration phase. This approach protects from hacking the password and prevents them from launching different attacks. Thus our system is more secure and reliable than two step authentication system. As with all graphical based systems our system will also be slow. The normalization and matching will take time. An important issue of our system is that it is somewhat user dependent during authentication. It depends upon the user’s drawing ability. Thus, the system may not be able to verify the objects drawn by the user and as a result the actual user may not be authenticated. The possible attacks on graphical passwords are Brute force attack, Dictionary attacks, Guessing, Spy-ware, Shoulder surfing and social engineering. Graphical based passwords are less vulnerable to all these possible attacks than text based passwords and they believe that it is more difficult to break graphical passwords using these traditional attack methods. Our System is resistant to almost all the possible attacks on graphical passwords.

7 Conclusion and Future Work The core element of computational trust is identity. Currently many authentication methods and techniques are available but each with its own advantages and shortcomings. There is a growing interest in using pictures as passwords rather than text passwords but very little research has been done on graphical based passwords so far. In view of the above, we have proposed authentication system which is based on graphical password schemes. Although our system aims to reduce the problems with existing graphical based password schemes but it has also some limitations and issues like all the other graphical based password techniques. To conclude, we need our authentication systems to be more secure, reliable and robust as there is always a place for improvement. Currently we are working on the System Implementation and Evaluation. In future some other important things regarding the performance of our system will be investigated like User Adoptability and Usability and Security of our system.

A Hybrid Graphical Password Based System

163

References 1. van Oorschot Tao Wan, P.C.: TwoStep: An Authentication Method Combining Text and Graphical Passwords. In: 4th International Conference, MCETECH 2009, Ottawa, Canada (May 4-6, 2009) 2. Authentication, http://www.objs.com/survey/authent.htm (last visited on May 15, 2011) 3. Sobrado, L., Birget, J.C.: Graphical Passwords, The Rutgers Schloar, An Electronic Bulletin for Undergraduate Research, vol. 4 (2002), http://rutgersscholar.rutgers. edu/volume04/sobrbirg/sobrbirg.htm 4. Elftmann, P.: Diploma Thesis, Secure Alternatives to Password-Based Authentication Mechanisms, Aachen, Germany (October 2006) 5. Blonder, G.E.: Graphical password. U.S. Patent 5559961, Lucent Technologies, Inc., Murray Hill, NJ (August 1995) 6. Suo, X., Zhu, Y., Owen, G.S.: Graphical Passwords: A Survey. In: Proceedings of Annual Computer Security Applications Conference (2005) 7. Approaches to Authentication, http://www.e.govt.nz/plone/archive/services/see/see-pkipaper-3/chapter6.html?q=archive/services/see/see-pki-paper3/chapter6.html (last visited on May 15, 2011) 8. Roman, V.Y.: User authentication via behavior based passwords. In: Systems, Applications and Technology Conference, Farmingdale, NY (2007) 9. Biometric Authentication, http://www.cs.bham.ac.uk/~mdr/teaching/modules/security/lect ures/biometric.html (last visited on May 02, 11) 10. Gao, H., Ren, Z., Chang, X., Liu, X., Aickelin, U.: A New Graphical Password Scheme Resistant to Shoulder-Surfing. In: 2010 International Conference on CyberWorlds, Singapore (October 20-22, 2010) 11. Perrig, A., Song, D.: Hash Visualization: A New Technique to improve Real-World Security. In: International Workshop on Cryptographic Techniques and E-Commerce, pp. 131– 138 (1999) 12. Davis, D., Monrose, F., Reiter, M.K.: On User Choice in Graphical Password Schemes. In: 13th USENIX Security Symposium (2004) 13. Leung, W.H., Chen, T.: Hierarchical Matching For Retrieval of Hand Drawn Sketches. In: Proceeding of International Conference on Multimedia and Expo (ICME 2003), vol. 2 (2003) 14. Khan, H.Z.U.: Comparative Study Of Authentication Techniques. International Journal of Video & Image Processing and Network Security IJVIPNS 10(04) 15. Token Based Authentication, http://www.w3.org/2001/sw/Europe/events/foafgalway/papers/fp /token_based_authentication/ (last visited on May 02, 2011) 16. Knowledge Based Authentication, http://csrc.nist.gov/archive/kba/index.html (last visited on May 02, 2011) 17. Knowledge based Authentication, http://searchsecurity.techtarget.com/definition/knowledgebased-authentication (last visited on May 02, 2011)

164

W.Z. Khan et al.

18. A Survey on Recognition based Graphical User Authentication Algorithms, http://www.scribd.com/doc/23730953/A-Survey-on-RecognitionBased-Graphical-User-Authentication-Algorithms (last Visited on May 02, 2011) 19. Jain, A., Bolle, R., Pankanti, S. (eds.): Biometrics: personal identification in networked society. Kluwer Academic, Boston (1999) 20. Hurson, A.R., Ploskonka, J., Jiao, Y., Haridas, H.: Security issues and Solutions in Distributed heterogeneous Mobile Database Systems. In: Advances in Computers, vol. 61, pp. 107–198 (2004) 21. Biddle, R., Chiasson, S., van Oorschot, P.C.: Graphical Passwords: Learning from the First Twelve Years, Carleton University - School of Computer Science, Technical Report TR-11-01 (January 4, 2011) 22. Weinshall, D.: Cognitive authentication schemes safe against spyware, (short paper). In: IEEE Symposium on Security and Privacy (May 2006) 23. Hayashi, E., Christin, N., Dhamija, R., Perrig, A.: Use Your Illusion: Secure authentication usable anywhere. In: 4th ACM Symposium on Usable Privacy and Security (SOUPS), Pittsburgh (July 2008) 24. Davis, D., Monrose, F., Reiter, M.: On user choice in graphical password schemes. In: 13th USENIX Security Symposium (2004) 25. Passfaces Corporation. The science behind Passfaces, White paper, http://www.passfaces.com/enterprise/resources/white_papers.h tm (last visited on May 05, 11) 26. De Angeli, A., Coventry, L., Johnson, G., Renaud, K.: Is a picture really worth a thousand words? Exploring the feasibility of graphical authentication systems. International Journal of Human-Computer Studies 63(1-2), 128–152 (2005) 27. Moncur, W., Leplatre, G.: Pictures at the ATM: Exploring the usability of multiple graphical passwords. In: ACM Conference on Human Factors in Computing Systems (CHI) (April 2007) 28. Pering, T., Sundar, M., Light, J., Want, R.: Photographic authentication through untrusted terminals. In: Pervasive Computing, pp. 30–36 (January-March 2003) 29. Wiedenbeck, S., Waters, J., Sobrado, L., Birget, J.: Design and evaluation of a shouldersurfng resistant graphical password scheme. In: International Working Conference on Advanced Visual Interfaces (AVI) (May 2006) 30. Bicakci, K., Atalay, N.B., Yuceel, M., Gurbaslar, H., Erdeniz, B.: Towards usable solutions to graphical password hotspot problem. In: 33rd Annual IEEE International Computer Software and Applications Conference (2009) 31. Jermyn, I., Mayer, A., Monrose, F., Reiter, M., Rubin, A.: The design and analysis of graphical passwords. In: 8th USENIX Security Symposium (August 1999) 32. Valentine, T.: An Evaluation of the PassfaceTM Personal Authentication System, Technical Report. Goldmsiths College University of London, London (1998) (the first report known in the literature)

Privacy Threat Analysis of Social Network Data Mohd Izuan Hafez Ninggal and Jemal Abawajy School of Information Technology, Deakin University, 3217 Victoria, Australia {mninggal,jemal}@deakin.edu.au

Abstract. Social network data has been increasingly made publicly available and analyzed in a wide spectrum of application domains. The practice of publishing social network data has brought privacy concerns to the front. Serious concerns on privacy protection in social networks have been raised in recent years. Realization of the promise of social networks data requires addressing these concerns. This paper considers the privacy disclosure in social network data publishing. In this paper, we present a systematic analysis of the various risks to privacy in publishing of social network data. We identify various attacks that can be used to reveal private information from social network data. This information is useful for developing practical countermeasures against the privacy attacks. Keywords: Privacy disclosure, Social networks, Threat analysis, Data publications.

1 Introduction Online social networking has become one of the most popular activities on the web [1-2]. The dramatic increase in the number, size, and variety of online social networks has generated interesting data management and data mining problems. An important concern in the release of these data for study is their privacy, since social networks usually contain personal information. Privacy is an important issue when one wants to make use of data that involves individuals’ sensitive information. While expressly acknowledging the immense potential and the importance of social networks as a communication tool, many real-world social networks contain sensitive information and serious privacy concerns have been raised. Social networks often contain some private attribute information about individuals as well as their sensitive relationships. Simply removing or replacing the identifying attributes such as name and SSN by meaningless unique identifiers is far from sufficient to protect privacy [3]. Serious concerns on privacy protection in social networks have been raised in recent years [4-5], particularly when social network data is published [3]. The increasing availability of rich social media, popular online social networking sites, and sophisticated data mining techniques have made privacy in social networks a serious concern. As a result, the thriving phenomenon of social networks has also attracted the attention of lawmakers. In a significant number of countries (primarily, but not exclusively, in Europe and North America) Data Protection Authorities have Y. Xiang et al. (Eds.): ICA3PP 2011 Workshops, Part II, LNCS 7017, pp. 165–174, 2011. © Springer-Verlag Berlin Heidelberg 2011

166

M.I.H. Ninggal and J. Abawajy

started to scrutinize both, the business models of social networks as well as the practices performed on such platforms, where such attention usually is reserved to general privacy issues (such as: identity theft, on-line frauds, data security) or to sector specific data protection problems (such as: financial transactions, e-commerce, electronic communication, children's protection), but not yet to marketing performed on such platforms (exceptions – with provisions specifically governing commercial communication on social networks – are a number of countries, members to the European Union). The US has in place Statute Law provisions or Self-Regulation guidelines specifically dealing with privacy issues on social networks. In most countries, general provisions and requirements in place for processing of personal data will apply to social networks. In some jurisdictions (inclusive of several nonEuropean ones in North and South American as well as in Asia and Oceania) dedicated rules are in preparation and likely to come into force in a near future. In this paper, we consider the privacy disclosure in social network data publishing. We present a systematic analysis of the various risks to privacy in published social network data. Privacy becomes an important issue when one wants to make use of data that involve individuals’ sensitive information. Social network data often contains sensitive information about the users [4-6]. Simply removing all identifiable personal information (such as names and social security number) before releasing the data is insufficient. The conventional methods proposed for micro data privacy [7], cannot be used directly to ensure privacy of social network data due to complex dependencies between the data and various relationships. We will survey and analyze various possible privacy leak concern when a social network data is published. We identify various attacks such as different structural queries that can be used to reveal private information from social network data. This information is useful for developing countermeasures to reduce the risk of such attacks. The rest of the paper is organized as follows. Section 2 presents a high level architecture of social network and discuses the motivation for social network data publication. Section 3 analyses various threats to social network data privacy concerns. The privacy breach is presented in Section 4. Finally, Section 5 gives the conclusion.

2 Publishing Social Network Data Social networks have become an important data source. Certainly, they have made data collection on individuals much easier. This new phenomenon has generated a wealth of data that is collected and maintained by the social media service providers. Sometimes it is necessary or beneficial to release such data to the public. The data generated by social media services often referred to as the social network data. In many situations, the data needs to be published and shared with others. The usefulness of social network data in capturing real world social activities has attracted many parties demanding the data for analysis purposes. Social network analysis has been a key technique in modern sociology, geography, economics, and information science [3]. Researchers in sociology, epidemiology, and health-care related field collect data about geographic, friendship, family, and sexual networks. Social network data is useful to them to study disease propagation and risk. In addition, there is also

Privacy Threat Analysis of Social Network Data

167

increased interest by researchers in governments institution in mining social network data for information and security purposes [8]. All these quests require the data to be shared or published.

Fig. 1. High level system components of social network

Social networks describe entities (often people) and the relationships between them. Social network analysis is often used to understand the nature of these relationships, such as patterns of influence in communities, or to detect collusion and fraud. A high level system component of a social network is shown in Fig. 1. In the architecture, there are users, social media services, and third party data recipients. Example of social media services include collaborative projects, blogs, content communities, social networking sites, virtual game worlds, and virtual communities [9]. Facebook is the most popular social networking sites currently has more than 500 million active users and they spend over 700 billion minutes per month of using the application [10]. Social media service users can be any real world entity that uses the service like individual or organization. When a user uses an online social media service, they usually are asked to create a profile and to give information about themselves. This information includes personal identifiable information like social security number, name and phone number which uniquely identify a person. They may also give semiidentifiable information like home address, former school he/she went or former company he/she has worked as well as private or sensitive information that users may like to make available to selected entity while keep it hidden from the public view. Sensitive information can include religion, political view, type of disease (as in healthcare network) or generated income (as in financial network). On top of that, there are also data generated from the social activity from the services. Some of this data may also carry sensitive information. All these information is kept and maintained by the service provider. In many situations, the data needs to be published and shared with others. The overall intent is for the data to be used for the public good such as in the evaluation of economic models, in the identification of social trends, and in the pursuit of the stateof-the-art in various fields. Usually, such data contain personal information such as medical records, salaries, and so on, so that a straightforward release of data is not appropriate. For examples, business companies are analyzing the social connections in social network data to uncover customer relationship that can benefit their services and product sales. The analysis result of social network data is believed to potentially provide an alternative view of real-world phenomena due to the strong connection

168

M.I.H. Ninggal and J. Abawajy

between the actors behind the network data and real world entities [11-12]. In fulfilling the demands for the data, online social media operators have been sharing the data it maintains with external third parties such as business advertisers, application developers, as well as academic researchers. Therefore, releasing the data to third parties has to be done in a way that can guarantee the privacy of the users. In other word, the data must undergo a privacy-preserving phase before being released to third parties. Online social media service providers who maintain the data may has specific interest in specific analysis outcomes of their data but due to the lack of in-house expertise to conduct the analysis, outsourcing the task to external parties often comes as the alternative option. In different situation, the owner of the data shares the data with third parties. For example, advertising partners tend to be interested in the sort of information that is held by the service provider. The data usually contain valuable information that can enable better social targeting of advertisements. On the other hand, the request to use the data can also come from third party applications embedded in the social media application itself. For instance, Facebook has thousands of third-party applications and the number is growing exponentially [13]. Even though the process of data sharing in this case is implicit, the data is indeed passed over from the data owner (service provider) to different party (the application). The data given to these applications is usually not sanitized to protect users’ privacy [14]. Social network data usually contain users’ private information; it is important to protect this information in any sharing activities. When data is shared, risk to data privacy violation will likely result as the key problem both, for all those participating in activities on social networks (be it or for private or for business purposes) as well as for companies running such platforms. Thus, publishing the data may violate individual privacy. Individual privacy is defined as “the right of the individual to decide what information about himself should be communicated to others and under what circumstances” [15]. A privacy breach occurs when sensitive information about the user is disclosed to an adversary. There are well-known examples of unintended disclosure of private information such as the AOL search data [16] and attacks on Netflix data [9]. The problem is challenging due to the diversity and complexity of data and their relationships, on which an adversary can use many types of background knowledge to conduct an attack. In the following section, we analyze various threats to social network data. Although in many ways a user gives ‘consent’ when they sign up to an online social network site, most are unaware of the implications of voluntarily providing personal information on profiles as well as not being aware of how this information may be processed. Privacy implications associated with online social networking depend on the level of identifiability of the information provided, it’s possible recipients, and its possible uses [17]. According to a survey released for EU Data Protection Day 2010, almost 50% of human resources professionals active in Europe perform on-line checks on candidates and 25% of all job applications are rejected based on the results of searches performed on candidates’ on-line reputation and profiles, where dismissal is usually grounded on ‘inappropriate comments’ or ‘unsuitable photos or videos’ found on the Internet. Interestingly the same survey shows that on the other side consumers continue to significantly underestimate the risks deriving from their on-line profiles: in the UK only 9% (in France 10% and in

Privacy Threat Analysis of Social Network Data

169

Germany 13%) of job seekers held that personal information available about them on the Internet would exercise influence on the outcome of their applications. In health care area, Personal Health Record (PHR) systems such as Google Health (health.google.com) allows users to store and manage personal information, including health information, emergency contacts, insurance plans, medications, immunizations, past procedures, test results, medical conditions, allergies, medications, family histories and lab results. Sharing of this information across user accounts is also supported. Placing detailed histories of health information online could expose users to significant risks [18]. Social finance network services like Wikinvest (www.wikinvest.com) are also changing the way finance has been done. Unacceptable disclosure of this type of data can result in serious consequences for individual ranging from scam and frauds to physical threats.

3 Threat Analysis In this section, we examine how a social network is translated into a data graph, what kind of sensitive information may be at risk and how an adversary may launch attack on individual privacy. 3.1

Data Representation

The data generated by social media services is usually viewed or represented as a graph network that contains vertices and connections between them. The network considered here is binary, symmetric, and without self-loops. Formally, a social network can be represented as , where , ,…, is a set of vertices and is the set of edges. We define as the degree of vertex . , which is the number of the vertices connected to node Harinda

Andy Eddie

Izuan

Claudia Bob Davood

Fed Gary Jemal

a) Original network

b) Naïve anonymized network

Fig. 2. Graphical social network representation

Fig. 2 illustrates an example of social network as a graph. The vertices usually represent real world actors or entities like individuals or organizations. Each vertex has a profile that usually contains personal attributes, such as name, gender, birth date, political view, religion etc. These individuals are usually connected by edges to represent some sort of social tie or link made between them. For example, in Social Networking Sites, these edges represent the connected friends each member has.

170

M.I.H. Ninggal and J. Abawajy

Therefore, edge can also have its attributes to describe the properties of the connection. For example in content communities type of social media, this attribute can be the content of the document they are collaborating with or the text comments and its timestamp that a user made in his/her subscribed blogs. This may also be the type of relationship between two users in social networking sites (e.g. political affiliation). 3.2

Background Knowledge

The first step in understanding attacks on privacy is to know what external information of a graph may be acquired by an adversary. Background knowledge is the piece of information that is known to the attacker and used by the attacker to perpetrate privacy attack. In social network data, the information that could be used as background knowledge to intrude user privacy is personal attribute and structural (or topological) attribute. The personal attribute is the information that describes a person such as name, address, date of birth, political view etc. Some attributes act as an identifier itself where it can be unique to individuals. Some other attributes are categorized act as semi identifiers or quasi attributes. Several quasi attributes, when combined together is potential to identify a person. Therefore, it is usually exploited as mapping parameter to find targeted individual in social network data.

Gary

Gary

Gary

Fig. 3. Structural information of Gary

The structural attribute is the information that can describe how an entity is connected to other entities in social network data. This information includes: i. Degree - This is the number of direct social links or relationships that an entity has. in Fig. 3 shows the social link that Gary has. This information does not carry direct sensitive information but is potential to be used as an effective mapping parameter in searching target individual in a network. Furthermore, acquiring this information from the network is relatively easy. For example in Facebook, the number of friends that appear in the user profile is the direct social tie the user has. This information does not uniquely represent an individual. However, in situation where the data range is small –e.g. only involve on a specific social group in the network - the return results could be very minimal or possibly unique. Using , adversary can map Gary as vertex 7 in naïve anonymized network of Fig. 2b. ii. Neighborhood – This refers to a set of neighboring entities that has direct social links to a target entity which they also have mutual link between them. For example in Fig.3b. If an adversary knows that Gary has four best friends who also three of them know each other, the adversary could still map Gary as vertex 7 even though there are several other vertices share the same number of degree.

Privacy Threat Analysis of Social Network Data

171

iii. Sub-graph – This refers to a set of relationships which the target entity is connected to which is a subset of the whole graph (Fig. 3c). To assume an adversary know this richer information is quite too strong. However, an adversary may create a set of dummy profiles and create social link between these profiles with certain patterns. Having that, the adversary then use those dummy profiles to establish a social link to target individual. The social link can be established as easy as adding target to the friend list or address book. Another way is the adversary simply constructs a coalition with other friends which also forming a small uniquely identifiable sub-graph. Having knowledge about specific pattern of relationship that he/she purposely created, the attacker later uses that pattern to locate the target individual in the released data. This is known as active attack and passive attack respectively [5]. iv. Network graph metrics – A network graph has many matrices. Some of them can implicitly reveal an individual. For instance, in a closed network communities which is a political movement group, the centrality metric could potentially reveal the leader of the group. There is another situation where an individual does not know about the other relationship of his/her neighborhoods. The closeness metric could give false assumption to the individual if it is found some of their neighborhoods have non-supposed affiliation. If a social network is published as simply a graph with no other information (such as attributes of a node) then re-identification attacks are still possible. It has been shown that if an attacker (or set of attackers) participates in the social network, then in many cases it is possible for the attacker to identify nodes corresponding to accounts under his control [5]. 3.3

Data Mapping Mechanism

The adversaries normally access the data by performing queries. The queries usually performed with several parameters like auxiliary information. Structural Queries is a series of knowledge queries that able to provide answers to a restricted knowledge about a target node in the network [19]. These queries exploit the structural information that may be available to an adversary, including complete and partial descriptions of vertex neighborhoods, and connections to hubs in the network. i. Vertex refinement queries - this query is a locally expanding structural query. It describes the structure of local neighborhood from a vertex’s perspective (a targeted individual perspective) in an iterative way. The weakest knowledge query, , simply returns the attribute information of the vertex . For unlabelled graph, the queries return null; returns the degree (the number returns the degree of each neighbor of of social links) that vertex v has; vertex v. These iterative queries can be defined as , where H

,… ,

returns the multiset of degree of each vertices adjacent to .

172

M.I.H. Ninggal and J. Abawajy

Example 3.1. Fig. 3 shows the computation of, , and for each node. is the vertex label of every vertices. In this case, the graph is set unlabeled, then is and they are uniform for all vertices. Let assume the targeted = {4}, which is Gary’s degree. individual is Gary in Fig. 1a, then = {2, 2, 3, 3} which represents Gary’s neighbors’ degrees. ii. Sub-graph queries - these queries identify the existence of a sub-graph around the targeted vertex by counting the number of edges in described subgraph. By these queries, the adversary is assumed to be able to gather some fixed number of social link focused around the target . By exploring the neighborhood of , the adversary learns the existence of a sub-graph around representing partial information about the structure around . iii. Hub fingerprint queries – these queries give information about how the vertex is linked to a set of selected hubs in the network. A hub is identified as a vertex with high degree and high betweenness centrality and they are often outliers in a network. For example in social networking sites like Facebook, a hub may correspond to a very famous person who has very high number of social links. These queries represent a range of structural information that may be available to adversaries, including complete and partial descriptions of vertex's local neighborhoods, and vertex's connections to hubs in the network.

4 Classification of Privacy Breach The privacy breaches in social networks can be categories into identity disclosure, sensitive link disclosure and sensitive attribute disclosure [20-21]. Identity disclosure happens when an adversary is able to map a record to specific individual. The identity disclosure may be considered as the key of privacy violation in social networks because it usually leads to the disclosure of content information as well as the information about relationship they have got. It could also lead to the revelation of an individual’s existence in a closed community network where he/she has strong privacy expectation of their existence in that group. For example, Facebook allows its user to create a network group with invited only member. This closed community network group could theme from secret society to political movement to religious purpose. Therefore, revealing someone existence in such group would also violate their privacy. In the sensitive link disclosure attack, the relationships between two individuals are revealed - The link among vertices in social network data can be a symbolism of relationship between individuals or organizations. This information is generated from social activities when using social media services. There are relationships that are safe for public to know, but some individuals may not prepare to reveal specific relationship they have. An adversary may want to know the degree of relationship between two entities. The disclosure occurs when the adversary is able to find out the existence of a relationship between two users, which the involved individuals prefer to keep it private. For examples, in social network data, based on the friendship relationships of a person and the public preferences of the friends such as political affiliation, it may be possible to infer the personal preferences of the person in

Privacy Threat Analysis of Social Network Data

173

question as well. Two entities in a social network may have many connections. Some are safe for the public to know and others that should remain private. If the relationship of two individuals can be determined by certain path, then the privacy is compromised. In the sensitive attribute disclosure attack, the sensitive data associated with each vertex or edge is compromised. Attribute disclosure occurs when an adversary is able to determine the value of a sensitive user attribute, which the user intended to keep it as private. Sensitive attributes may be associated with an entity as well as link relationship. In application level, the visibility of the attribute information is often variable. A member's profile can be set to be viewed publicly as well as by limited people in the network. In social network sites, content which commonly viewable by public usually something about hobbies and interests. However, certain application requires the user to give specific information accordingly to the application theme. In health based application, there could be information such as drinking and drug habits or type of disease that the user gives in the profile for monitoring purpose by other user in the system such as doctor. On the other hand, in online sexual network, there are sexual-based information like preferences and orientation. Meanwhile, there is also sensitive information generated from the interaction between users. For example in messaging network and email, the sensitive content are usually the text message, the timestamp, the frequency of interaction and other information correspond to both parties. Users usually have strong perception that this information is kept private [5].

5 Conclusion Although in many ways a user gives ‘consent’ when they sign up to an online social network site, most are unaware of the implications of voluntarily providing personal information on profiles as well as not being aware of how this information may be processed. Privacy implications associated with online social networking depend on the level of identifiability of the information provided, it’s possible recipients, and its possible uses. In this paper, we considered the privacy disclosure in social network data publishing. We present a systematic analysis of the various risks to privacy in publishing of social network data. We identify various attacks that can be used to reveal private information from social network data. This information is useful for developing practical countermeasures against the privacy attacks.

References 1. Alexa. The top 500 sites on the web (2011), http://www.alexa.com/topsites 2. Bonneau, J., Preibusch, S.: The privacy jungle: On the market for data protection in social networks. Economics of Information Security and Privacy, 121–167 (2010) 3. Zhou, B., Pei, J., Luk, W.: A brief survey on anonymization techniques for privacy preserving publishing of social network data. ACM SIGKDD Explorations Newsletter 10(2), 12–22 (2008) 4. Kleinberg, J.M.: Challenges in mining social network data: processes, privacy, and paradoxes. ACM, New York (2007)

174

M.I.H. Ninggal and J. Abawajy

5. Backstrom, L., Dwork, C., Kleinberg, J.: Wherefore art thou r3579x?: anonymized social networks, hidden patterns, and structural steganography. ACM, New York (2007) 6. Srivastava, J., et al.: Data mining based social network analysis from online behavior (2008) 7. Fung, B., et al.: Privacy-preserving data publishing: A survey of recent developments. ACM Computing Surveys (CSUR) 42(4), 1–53 (2010) 8. Rosenblum, D.: What anyone can know: The privacy risks of social networking sites. IEEE Security & Privacy, 40–49 (2007) 9. Facebook. Facebook Statistic (2011), http://www.facebook.com/press/info.php?statistics 10. Kaplan, A.M., Haenlein, M.: Users of the world, unite! The challenges and opportunities of Social Media. Business Horizons 53(1), 59–68 (2010) 11. Watts, D.J., Strogatz, S.H.: Collective dynamics of ‘small-world’networks. Nature 393(6684), 440–442 (1998) 12. Watts, D.J.: Networks, dynamics, and the small-world phenomenon. American Journal of Sociology, 493–527 (1999) 13. Narayanan, A., Shmatikov, V.: De-anonymizing social networks. IEEE, Los Alamitos (2009) 14. Felt, A., Evans, D.: Privacy protection for social networking APIs. In: 2008 Web 2.0 Security and Privacy, W2SP 2008 (2008) 15. Westin, A.F.: Privacy and freedom, London, vol. 97 (1967) 16. Hansell, S.: AOL removes search data on vast group of web users. New York Times 8, C4 (2006) 17. Gross, R., Acquisti, A.: Information revelation and privacy in online social networks. ACM, New York (2005) 18. Williams, J.: Social networking applications in health care: threats to the privacy and security of health information. ACM, New York (2010) 19. Hay, M., et al.: Resisting structural re-identification in anonymized social networks. Proceedings of the VLDB Endowment 1(1), 102–114 (2008) 20. Liu, K., et al.: Privacy-preserving data analysis on graphs and social networks. Next Generation of Data Mining, 419–437 (2008) 21. Zheleva, E., Getoor, L.: Preserving the privacy of sensitive relationships in graph data. Springer, Heidelberg (2007)

Distributed Mechanism for Protecting Resources in a Newly Emerged Digital Ecosystem Technology Ilung Pranata, Geoff Skinner, and Rukshan Athauda University of Newcastle, School of Design, Communication and IT, University Drive, Callaghan NSW, Australia {Ilung.Pranata,Geoff.Skinner,Rukshan.Athauda}@newcastle.edu.au

Abstract. Digital Ecosystem (DE) is characterized as an open and dynamic environment where the interaction and collaboration between its entities are highly promoted. A major requirement to promote such intensive interaction and collaboration in a DE environment is the ability to secure and uphold the confidentiality, integrity and non-repudiation of shared resources and information. However, current developments of such security mechanisms for protecting the shared resources are still in their infancy. Most of the proposed protection frameworks do not provide a scalable and effective mechanism for engaging multiple interacting entities to protect their resources. This is even a greater issue when multiple resources are exchanged and shared in an open and dynamic environment. Therefore, we proposes a distributed mechanism for individual enterprises to manage their own authorization processes and resource access permissions with an aim to provide a rigorous protection of entities’ resources. Keywords: Authentication, authorization, digital ecosystem.

1 Introduction Since its first introduction in 2002, a new emerging concept of Digital Ecosystem (DE) has grasped numerous attentions from researchers, businesses, ICT professionals and communities around the world. This concept aimed at achieving a set of predetermined goals of Lisbon summit in March 2000 which primarily focuses on dynamic formation of knowledge based economy [1]. Further, the knowledge based economy will lead to a creation of more jobs and a greater social inclusion in sustaining the world economic growth [2]. DE is a multi-dimensional concept that encompasses several current technology models such as collaborative environment [3], distributed system [4], and grid technology [5]. The combination of concepts from these models provides the ability for a DE environment to deliver an open, flexible and loosely coupled resource sharing environment. On the other hand, this combination also develops several complicated security issues which need to be addressed before the full implementation of a DE concept. Unfortunately, the evaluation on DE security dimensions from the current literature signifies a number of deficiencies in its security architecture particularly in protecting the enterprise resources and information. There is a need for a comprehensive resource protection solution that is able to provide a Y. Xiang et al. (Eds.): ICA3PP 2011 Workshops, Part II, LNCS 7017, pp. 175–185, 2011. © Springer-Verlag Berlin Heidelberg 2011

176

I. Pranata, G. Skinner, and R. Athauda

strong and rigorous mechanism to safeguard the critical resources and further to reduce the possibility of information leakage to the unauthorized parties. A key challenge for enterprises who involved in a DE environment is to determine the right users who are able to access the services, resources, knowledge and information hosted by these enterprises. This challenge is occurred due to several reasons. First, the occurrences of multiple resources published and shared by each enterprise in a DE environment and second, the situation where various clients are able to access each individual resource. Due to these reasons, enterprises urgently need a mechanism that effectively manages their clients’ access control and authorization permissions with an aim to protect their resources. In this paper, we attempt to deliver a comprehensive framework allowing enterprises to protect their resources and information from any unauthorized use.

2 Related Work In a DE environment where multiple interacting entities exist, the required efforts to enforce a strong authentication and authorisation mechanism are extensive. We identify three core issues that appear to be the challenging tasks to enforce such mechanisms. First, as the DE community expands its size to incorporate more entities, the resource providers face a challenge to identify the legal entities that are able to access their resources. Second, the fact that each entity would have different set of access permissions to access multiple resources further complicates the issue. Third, it is probable that each resource provider would host multiple resources and services in a DE environment. This situation, in turn, creates a great issue to authorize the right entities to the right resources with the right permissions. Failure in assigning right permissions to the entities would compromise the usage of resources and would bring negative impact for resource provider. The current internet mechanisms are still far from adequate to provide a reliable authentication and authorisation processes for a DE environment. This view is reflected from our literature analysis over a number of internet mechanisms. The most prominent mechanism to manage the client credentials is through the implementation of Identity Provider (IdP) or Credential Provider [6, 7]. IdP mainly focuses at storing and providing the client credential to any resource providers for their client authentication process. On every authentication process, resource provider will request the client credential from its trusted IdP where it receives any access request from a client. In latter development, several technology standards such as SAML [8] and Liberty Alliance [9] are adopted in this mechanism to provide the federation mechanism of multiple entities for Single Sign On (SSO) services. Similarly, the Credential Server (CRES) [10] and the Grid Security Infrastructure (GSI) MyProxy [11] utilizes the IdP concept, and they further leverage its concept for large number of dispersed servers over a wide geographical area. Both mechanisms store the clients’ credentials in the local server; however, authentication of a remote client can be facilitated by requesting his credential from the trusted remote server. In both MyProxy and CRES mechanisms, the resource provider requests the client credential from the local server on every authentication process. The local server then creates a certificate token which contains client information. Subsequently, the

Distributed Mechanism for Protecting Resources

177

certificate token is sent to the resource provider as the acknowledgement of the authentic client. When the resource provider receives the token, it allows the client to access the resources based on the trust established with the publisher of token. The SSL/TLS technology [12] has been extensively used in e-commerce transactions for secure authentication and communication. This technology is designed with a highly reliance on the Certificate Authority (CA) to ensure the legitimacy of an entity. Therefore, SSL/TLS technology also presents a centralized credential management. Although these approaches could be deployed well in a DE environment, the conspicuous issue of single server failure must be carefully considered. In an event where the credential provider server is down, there possibly a chaos in a DE community due to the unavailability of credential services for client authentication. Apparently, our literature review identifies that several internet authorisation mechanisms take similar approach as its authentication mechanisms. The most prominent authorisation mechanisms, such as CAS [13], Akenti [14], and PMI [15], utilize a central server to assign the multiple access permissions to the clients individually although their implementation are differ between each other. These mechanisms also inherit several issues pertinent to the central management of authorisation permissions. First, the central management would face real issue with the bottleneck and failure on its servers. Security breach would occur if the central servers fail to perform their authorisation processes over the clients. Although it is possible to replicate the central server, the replication process will bring abundance administrative issues, considering a huge amount of data that needs to be replicated. Second, challenges occur when the central server attempt to assign the access permissions to the DE member entities. As a large number of resource providers that host one or more resources, the central server needs to register each resource and its access permissions individually. Further, this situation becomes even more challenging as a single resource could be associated with multiple different access permissions, and each client may have different access permissions assigned to him. Therefore, the central management is not practical when there is huge number of entities in a DE environment. Third, serious administration issues would occur as a DE environment grows in size and diversity due to the great benefits that they can achieve. A central server will be experiencing huge burden to manage all client and resource providers’ accounts and permissions even with the use of super computers or grid collections of computers. Several DE literatures clearly reveal that DE is characterized as an open environment on which a centralized structure is minimized. DE must be engineered to provide a high resilience infrastructure while avoiding single point of control and failure [16, 17]. Therefore, a completely distributed control mechanism is required that immune to the super control failure. It is evident that the aforementioned internet mechanisms are inappropriate to be implemented in a DE environment due to its centralized management. In this paper, we propose a solution to manage the authentication and authorization in a full distributed approach that focuses on each entity to manage the authentication and authorization mechanism with the utilization of a capability token. We termed our solution as Distributed Resource Protection Mechanism (DRPM) [18, 19]. In this paper, we enhance our solution by eliminating its reliance on the central credential server and further secure the mechanism by utilizing the Public Key Infrastructure [15].

178

I. Pranata, G. Skinner, and R. Athauda

3 Overview of DRPM 3.1 Identifying Entity through Client Profile The present mechanism for service discovery in a DE environment requires a client to search for resources by utilizing a semantic discovery portal through its browsers or rich applications [20]. This discovery portal would search and list all resources which are provided by DE resource providers. Once the client finds the resource, it then contacts the resource provider and requests for that resource. At this stage, resource provider does not know any information about this client and its intended purpose on the resource. This may put the resource at risk as it may contain highly sensitive information which must be protected from any misuse and malicious act. Therefore, it is crucial for a resource provider to understand its client’s information before any access to the resource is granted. Taking this into consideration, we adopted a method of creation for a client profile that aims to capture all required, but voluntarily provided, information about a client. The information which is contained in a client profile provides necessary data about who the client is and about their intentions and purpose for using the requested resources. The aim of implementing a client profile is to ensure the resource provider that resources are not going to the wrong entities and further impose the confidentiality and integrity of the resources. The use of client profiles also facilitates the auditing process on who is accessing a resource. For example, there may be a situation where a resource provider needs to trace back which client was delegated an access to the resource in case there was an incident involving a dispute or counterfeiting of the resource. In order to fully implement a client profile, it is necessary that a client registration portal is employed in DRPM. A client profile is generated through this registration portal. Further, resource providers are able to customize the registration portal to contain only the information which is important to them. New clients wishing to access a specific resource are initially redirected into this portal. If they wish to access the resource, they must continue to fill in all the necessary information required by the resource provider to produce a client profile. Once it is produced, the client profile is stored in the resource provider repository. Utilisation of this functional procedure and process provides an additional and enhanced method for determining who is accessing a particular resource at a particular time inside a DE environment. 3.2 Storing Permission in a Capability Token It is always a challenge to enforce client access permissions on the available resources within a DE environment. This challenge is due to the occurrences of a high amount of entities that actively interact in a DE environment. Further, these entities could also make the same request for a particular resource either at the same or at different time. To solve the issue of managing multiple resource access permissions on a diverse range of DE clients, we utilize and further evolve the concept of capability introduced by CAS server that is used in a Collaborative Environment. In CAS, capability is used to store all access rights of a user which are determined by a community policy. However, the implementation of the capability in our framework is slightly different to the capability implementation in a CAS server. In our framework, capability contains all the necessary

Distributed Mechanism for Protecting Resources

179

right permissions for each client to perform a set of operations on a particular resource. This capability is produced by the resource provider that hosts a particular resource. This capability would be used to grant the client access to the resources, and it further facilitates the authorization process for the clients. Once a client profile is created, a list of client authorization permissions are assigned into the capability token. The client’s access permissions and policies are expressed in XML [21] due to its simplicity, wide usability and self-descriptive characteristics. Our basic design of a capability token contains the client profile identifier, resource provider identifier, resource ide