Collaborative Agents - Research and Development

Lecture Notes in Artiﬁcial Intelligence Edited by R. Goebel, J. Siekmann, and W. Wahlster Subseries of Lecture Notes in...

Author: Christian Guttmann | Frank Dignum | Michael Georgeff

53 downloads 1440 Views 4MB Size Report

This content was uploaded by our users and we assume good faith they have the permission to share this book. If you own the copyright to this book and it is wrongfully on our website, we offer a simple DMCA procedure to remove your content from our site. Start by pressing the button below!

Report copyright / DMCA form

DOWNLOAD PDF

Lecture Notes in Artiﬁcial Intelligence Edited by R. Goebel, J. Siekmann, and W. Wahlster

Subseries of Lecture Notes in Computer Science

6066

Christian Guttmann Frank Dignum Michael Georgeff (Eds.)

Collaborative Agents Research and Development International Workshops CARE@AI09 2009 / CARE@IAT10 2010 Melbourne, Australia, December 1, 2009 and Toronto, Canada, August 31, 2010 Revised Selected Papers

13

Series Editors Randy Goebel, University of Alberta, Edmonton, Canada Jörg Siekmann, University of Saarland, Saarbrücken, Germany Wolfgang Wahlster, DFKI and University of Saarland, Saarbrücken, Germany Volume Editors Christian Guttmann School of Primary Health Care Monash University, Melbourne, Australia and Etisalat British Telecom Innovation Centre (EBTIC) Khalifa University, Abu Dhabi, United Arab Emirates E-mail: [email protected] Frank Dignum Department of Information and Computing Science Universiteit Utrecht, Utrecht, The Netherlands E-mail: [email protected] Michael Georgeff School of Primary Health Care Monash University, Melbourne, Australia [email protected] ISSN 0302-9743 e-ISSN 1611-3349 ISBN 978-3-642-22426-3 e-ISBN 978-3-642-22427-0 DOI 10.1007/978-3-642-22427-0 Springer Heidelberg Dordrecht London New York Library of Congress Control Number: 2011930957 CR Subject Classiﬁcation (1998): I.2, H.5.3, H.3, C.2.4, H.4, J.1 LNCS Sublibrary: SL 7 – Artiﬁcial Intelligence

© Springer-Verlag Berlin Heidelberg 2011 This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, speciﬁcally the rights of translation, reprinting, re-use of illustrations, recitation, broadcasting, reproduction on microﬁlms or in any other way, and storage in data banks. Duplication of this publication or parts thereof is permitted only under the provisions of the German Copyright Law of September 9, 1965, in its current version, and permission for use must always be obtained from Springer. Violations are liable to prosecution under the German Copyright Law. The use of general descriptive names, registered names, trademarks, etc. in this publication does not imply, even in the absence of a speciﬁc statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. Typesetting: Camera-ready by author, data conversion by Scientiﬁc Publishing Services, Chennai, India Printed on acid-free paper Springer is part of Springer Science+Business Media (www.springer.com)

Preface

Collaboration is required when multiple agents achieve complex goals that are diﬃcult or impossible to attain for an individual agent. This collaboration takes place under conditions of incomplete information, uncertainty, and bounded rationality, much of which has been previously studied in economics and artiﬁcial intelligence. However, many real-world domains are characterized by even greater complexity, including the possibility of unreliable and non-complying collaborators, complex market and incentive frameworks, and complex transaction costs and organizational structures. The focus of the International Workshop on Collaborative Agents – Research and Development (CARE) is on collaborative and autonomous agents that plan, negotiate, coordinate, and act under this complexity. This volume contains the accepted papers presented at the First and Second CARE workshops (CARE 2009 and CARE 2010, respectively). CARE 2009 was held in conjunction with the 22nd Australasian Joint Conference on Artiﬁcial Intelligence AI09, at Melbourne University, Australia. CARE 2010 was held in conjunction with the 2010 IEEE/WIC/ACM International Conference on Intelligent Agent Technology (IAT), at York University, Toronto, Canada. The two workshops together received 45 submissions from which we selected 12 papers for this volume. Each submission was reviewed by at least three Program Committee members to ensure high-quality contributions. We also included two contributions of highly respected researchers in this area that gave invited presentations at the workshops. This compilation of papers provides a good overview of the area. We would like to thank all invited speakers, Program Committee members and volunteers that made this workshop series possible. We gratefully acknowledge the sponsorship provided by Monash University, Precedence Health Care, and the Finkel Foundation. We used EasyChair to manage the workshop submissions and to create this volume. April 2011

Frank Dignum Christian Guttmann Michael Georgeﬀ

Organization

Program Committee Alexander Pokahr Andrew Gilpin Cees Witteveen Christian Guttmann Cristiano Castelfranchi David Morley Frank Dignum Franziska Kl¨ ugl Gord Mccalla Iyad Rahwan Kobi Gal Kumari Wickramasinghe Lars Brauchbach Lawrence Cavedon Liz Sonenberg Marcelo Blois Ribeiro Mathijs De Weerdt Michael Georgeﬀ Michael Luck Michael Winikoﬀ Philippe Pasquier Rafael Bordini Rainer Unland Samin Karim Sascha Ossowski Simon Goss Simon Thompson Toby Walsh Wayne Wobcke Wei Chen

University Hamburg, Germany Carnegie Mellon University, USA Delft University of Technology, The Netherlands EBTIC, UAE and Monash University, Australia Institute of Cognitive Sciences and Technologies, Italy SRI International, USA Utrecht University, The Netherlands ¨ Orebro University, Sweden University of Saskatchewan, Canada Masdar Institute of Technology, UAE and Massachusetts Institute of Technology, USA Massachusetts Institute of Technology, USA Monash University, Australia University Hamburg, Germany NICTA and RMIT University, Australia Melbourne University, Australia Pontiﬁcal Catholic University of Rio Grande do Sul, Brazil Delft University of Technology, The Netherlands Monash University, Australia King’s College London, United Kingdom University of Otago, New Zealand Simon Frasier University, Canada Federal University of Rio Grande do Sul, Brazil University of Duisburg-Essen, Germany Melbourne University, Australia University Rey Juan Carlos, Spain Defence Science and Technology Organisation DSTO, Australia British Telecom, United Kingdom NICTA and University of New South Wales, Australia University of New South Wales, Australia Intelligent Automation Inc., United States of America

Table of Contents

Multi-agent Coalition Formation for Distributed Area Coverage . . . . . . . . Ke Cheng and Prithviraj Dasgupta

1

Modeling Warehouse Logistics Using Agent Organizations . . . . . . . . . . . . . Marcel Hiel, Huib Aldewereld, and Frank Dignum

14

Agent-Based Crisis Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Huib Aldewereld, John Tranier, Frank Dignum, and Virginia Dignum

31

Male Optimal and Unique Stable Marriages with Partially Ordered Preferences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Mirco Gelain, Maria Silvia Pini, Francesca Rossi, K. Brent Venable, and Toby Walsh Agent-Based Simulation of Joint Fire Support Teams – Collaboration in Network-Centric Warfare Scenarios . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Christian Gerstner, Robert Siegfried, and Nane Kratzke Intelligent Collaborative Care Management (ICCM): Achieving and Managing Contractual Relationships and Service Delivery . . . . . . . . . . . . . Christian Guttmann, Ian Thomas, Kumari Wickramasinghe, Michael Georgeﬀ, Hamid Gharib, Simon Thompson, and Heinz W. Schmidt A Centralized Framework to Multi-robots Formation Control: Theory and Application . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Soheil Keshmiri and Shahram Payandeh

44

56

68

85

Flexible Behaviour Regulation in Agent Based Systems . . . . . . . . . . . . . . . Michael Luck, Lina Barakat, Jeroen Keppens, Samhar Mahmoud, Simon Miles, Nir Oren, Matthew Shaw, and Adel Taweel

99

Eﬀect of Network Topology in Opinion Formation Models . . . . . . . . . . . . . Brenton J. Prettejohn and Mark D. McDonnell

114

Behavior Learning in Minority Games . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Guanyi Li, Ying Ma, Yingsai Dong, and Zengchang Qin

125

Two Decades of Multiagent Teamwork Research: Past, Present, and Future . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Matthew E. Taylor, Manish Jain, Christopher Kiekintveld, Jun-young Kwak, Rong Yang, Zhengyu Yin, and Milind Tambe

137

VIII

Table of Contents

Designing Intelligent Healthcare Operations . . . . . . . . . . . . . . . . . . . . . . . . . Nilmini Wickramasinghe, Christian Guttmann, and Jonathan Schaﬀer

152

Collaborative Learning in Uncertain Environments . . . . . . . . . . . . . . . . . . . Waiho Wong and Joseph G. Davis

163

Improving Task-Based Plan Coordination . . . . . . . . . . . . . . . . . . . . . . . . . . . Chetan Yadati Narasimha, Cees Witteveen, and Yingqian Zhang

175

Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

187

Multi-agent Coalition Formation for Distributed Area Coverage Ke Cheng and Prithviraj Dasgupta Computer Science Department University of Nebraska Omaha, NE 68182, USA {kcheng,pdasgupta}@mail.unomaha.edu

Abstract. In the multi-robot area coverage problem, a group of mobile robots have to cover an initially unknown environment using a sensor or coverage tool attached to each robot. Multi-robot area coverage is encountered in many applications of multi-robot systems including unmanned search and rescue, aerial reconnaissance, robotic demining, automatic lawn mowing, and inspection of engineering structures. We envisage that multi-robot coverage can be performed eﬃciently if robots are coordinated to form small teams while covering the environment. In this paper, we use a technique from coalitional game theory called a weighted voting game that allows each robot to dynamically identify other team members and form teams so that the eﬃciency of the area coverage operation is improved. We propose and evaluate a novel technique of computing the weights of a weighted voting game based on each robot’s coverage capability and ﬁnding the best minimal winning coalition(BMWC). Also we designed a greedy method and a heuristic method to ﬁnd BMWC in O(n log n) time and O(n2 ) time respectively. We tested these two algorithm with our base line method. Keywords: Area Coalitions.

1

Coverage,

Weighted

Voting

Games,

Winning

Introduction

In recent years there has been an increasing interest for area coverage using multi-robot systems. Compared with single robot coverage, multiple robots have potentially better performance because increasing the number of robots reduces the time taken to cover the area. Additionally, multi-robot systems oﬀer robustness of the system against failure of individual robots[7,11]. However, it is challenging to design eﬃcient techniques for coordinating multi-robots to perform area coverage. As Hazon et al. [10] mention, optimal single robot coverage algorithms cannot be directly used or produce identical results in multi-robot cases. Area coverage with multi-robot systems involves challenges such as exchanging coverage maps judiciously between robots, allowing robots to move intermittently out of each other communication ranges and dispersing robots C. Guttmann, F. Dignum, and M. Georgeﬀ (Eds.): CARE 2009/2010, LNAI 6066, pp. 1–13, 2011. c Springer-Verlag Berlin Heidelberg 2011

2

K. Cheng and P. Dasgupta

in adequate numbers forwards uncovered regions. To address these challenges, we envisage that multi-robot team formations can be used as an appropriate technique the multi-robot coverage problem. Multi-robot collaborations and cooperative decision-making play an important role on multi-robot team formation because the achievement of a robot team is based on the performance of each robot. We propose to use weighted voting games (WVG), which provide a model of decision-making in many political voting procedures, as a model for structured team formation in our multi-robot system. In a WVG, each player has a weight, and a coalition of players wins the game if the sum of the weights of its participants exceeds a certain threshold. As weights can aﬀect the outcome, one of the key issues to use this model in multi-robot area coverage is how to dynamically compute such a weight for each robot. Finding a stable and unique coalition which is a well-known problem in coalitional games [12], is another practical challenge. In this paper, we illustrate a method of calculating the weight of a robot that can be used in a WVG, based on its coverage history. Also, we extend the weighted voting game with robot domain knowledge to calculate a unique coalition called the best minimal winning coalition(BMWC). We theoretically prove the feasibility of this model, and give algorithms to ﬁnd veto players and the BMWC. We have also experimentally veriﬁed the performance of our BMWC strategy with the nearest neighbor strategy and the utility based strategy on a robot simulation platform.

2

Preliminary Definitions

Coalitional Games: A coalitional game (CG) with transferable utility is given by (N, v), where N = {1, 2, . . . , n} is a set of agents or players (the grand coalition); and v : 2N → is a utility function that maps each group of agents S ⊆ N to a real-valued payoﬀ. Intuitively, v(S) is the proﬁt that the members of coalition S can attain by working together [12,13]. A lower case coalitional game is simple if v(S) can only take values 0 and 1, e.g., v : 2N → {0, 1}. In such games, we call a coalition C (C ⊆ N ) a winning coalition if v(C) = 1, and a losing coalition if v(C) = 0. An agent i is a veto player, if i ∈ C, v(C) = 1, v(C\{i}) = 0. In other words, veto players are the players included in all winning coalitions. On the contrary, dummy players are those players which can be removed from a winning coalition without aﬀecting the winning condition of the coalition, e.g., for any C ⊆ N , and i ∈ N , we have v(C ∪ {i}) = v(C) = 1, i is called a dummy player. Weighted Voting Games: A weighted voting game G is a simple game that is described by a vector of players’ weights w = (w1 , ..., wn ) and a quota q [1]. In such games, a coalition is winning if the sum of its total weights meets or exceeds the threshold quota q. Formally, weights wi assigned to each player i ∈ N .

Multi-agent Coalition Formation for Distributed Area Coverage

3

Let W be i∈N w(i), and q ∈ [0, W ]. For any C ⊆ N , v(C) = 1 if i∈N w(i) > q ; v(C) = 0 otherwise. Minimal Winning Coalition: S is a winning coalition, if ∀pi ∈ S, the coalition T = S − {pi } is a losing coalition, S is denoted as minimal winning coalition [15]. For example, let us consider the game G = (N, q : 6, 3, 2), which has three players N = {n1 , n2 , n3 } who cast 6, 3, and 2 votes, respectively. If we set the quota q at the ”majority” level, q = 6, then we have the following collection of minimal winning coalitions: {n1 }. So the veto player of the game is n1 . Players n2 and n3 are dummies in this game because they are not members of any minimal winning coalition. If we set quota q less than ”majority” level, for example, q = 5, the minimal winning coalitions of this weighted voting game are {n1 } and {n2 , n3 }. There are no veto players or dummy players. Imputation: Given a coalitional game G = (N, v), the imputation set, denoted P , is deﬁned as {x ∈ ρ|∀i ∈ N, xi ≥ v(i)} [13]. Core: A payoﬀ vector x is in the core of a coalitional game (N,v) if and only if ∀S ⊆ N, xi ≥ v(S) [13]. It is the signiﬁcant solution concept of coalitional games. The core of a coalitional game contains imputations such that no other sub-coalitions can obtain a better imputation for themselves by defecting from the grand coalition. Intuitively, no players plan to leave the coalition C, which is in the core, because they can not get better payoﬀs by forming any other coalitions. The core provides the stability for coalitional games. Weighted voting games are widely used representations of coalitional games in practice. For example, the voting system of the European Union is a combination of weighted voting games [2]. With exponentially large number of players, [8] shows that the computational complexity of general representation of coalitional game is infeasible. In other words, it is a NP-hard problem. In [1,9], Elkind gives a dynamic programming method to solve weighted threshold game in polynomial time.

3 3.1

System Modeling and Algorithm Design Single Robot Coverage Capability

The goal of multi-robots area coverage is to search or explore an unknown environment eﬃciently while reducing the redundancy of the area covered. This implies that each robot has to maximize the area it can cover within a certain time period, and simultaneously reduce the overlap between regions covered by itself and other robots. To facilitate our modeling, we posit that the contribution of a robot towards the team it belongs to is measured in teams of the area of a region the robot covers within a certain, ﬁxed number of timesteps T . We call this metric the robot ’s coverage capability. For physical robots, the coverage capability can vary dynamically depending on the local environment conditions,

4

K. Cheng and P. Dasgupta

(a)

(b)

Fig. 1. (a) 6 robots calculate two minimal winning coalitions in the communication range, (b) 4 robots form a best minimal winning coalition and head to a new team direction

function FindVetoPlayers returns set I{ inputs: set N , double Quota ; variables: player i; double W , W−i ; W = i∈N wi ; for each i ∈ N W−i = j∈N−{i} wj ; if(W−i < Quota) I = I ∪ {i}; }

Fig. 2. The algorithm with linear time used to ﬁnd veto players in weighted voting games

such as the size and number of obstacles, as well as intrinsic on robot capabilities including motor speed, and sensor detectable range. We propose to use robot coverage capability as the weights in a WVG. We model the environment as a grid-based graph, which each cell is the area a robot can cover in one time step. We assume that each robot has 4-DOF. The coverage map maintained by a robot r is used to store the regions (number of cells) in the vicinity of its current location that it has covered. A robot updates its coverage map every T timesteps. Each location in the coverage map is associated with a numeric value calculated using a node-counting technique [11], which represents the number of times the location has been visited by the robot. A robot uses the coverage information from its coverage map to calculate its coverage capability. The coverage capability of robot i is denoted as Ci = a × θi − b × ηi + C0 . θi is the coverage rate of robot i in recent time period T , cov and ηi is the redundancy rate. The coverage rate is θi = VVmap , where Vcov is the

Multi-agent Coalition Formation for Distributed Area Coverage

5

area robot i covered in the last T timesteps, and Vmap is the area of its whole coverage map. For example, if robot i can record a 100 × 100cm2 local area in its coverage map, and in T timesteps it has covered a 50 × 50cm2 area, we can red , where Vred is calculate the value θi is 0.25. The redundancy rate is ηi = VVmap the area of the revisited region within the coverage map, and Vmap is the area of the whole coverage map. a and b are normalizing constants, and C0 is the initial value of coverage ability. Thus, we can get Ci in the range [0, 1.96], when a = 2, b = 1, C0 = −0.04. The coverage capability Ci of robot i is upper and lower bounded. 3.2

Best Coalition Formation

Coalition structures represent the diﬀerent combinations of teams which a robot can form within other robots within its communication range. Every time, there are two or more teams of robots in each other’s communication range, we can search the coalition structures in the solution space. Our goal is to ﬁnd an efﬁcient partition, which increases the payoﬀ of the robot team without reducing the payoﬀ of any single robot. For example, suppose there are 4 robots that are within each other’s communication range. These robots can form 24 = 16 possible coalitions. To get a winning coalition, we can set a robot’s own weight as a linear function of the robot coverage capability deﬁned by wi = α × Ci + β, where Ci is the robot’s coverage capability introduced in former section, and α and β are the adjustment constants to make wi remain within a certain interval. For example, wi ∈ [0,1] with α = 12 and β = 0.02. The weight wi associated with a robot gets updated with its coverage map and coverage capability after every T timesteps. We set the winning threshold of the WVG to q = c × n, where n is the number of robots and c is an adjustment constant. In each round of the voting game, the weight wi of each robot and the quota q are both ﬁxed values. For example, we consider a weighted voting game with n = 4 robots and individual robot weights at w1 = 1.3, w2 = 1.1, w3 = 1.2, w4 = 0.2, in a concise form G = {3 : 1.3, 1.1, 1.2, 0.2}. We can get the minimal winning coalition {r1 , r2 , r3 }, when we set q = 3, with c = 0.75. A problem with applying the winning condition from general WVGs to our context is that sometimes there could be more than one minimal winning coalition. Suppose there are 6 robots with the weight set {1.1, 1.1, 1.1, 0.4, 0.4, 0.1}, and quota q = 3.5, as shown in Figure 1 (a). We can get two winning coalition {r1 , r2 , r3 , r4 } and {r1 , r2 , r3 , r5 }. As the minimal winning coalition is not unique, we design a g×xk +e×di,j +f ×ϕi,j

function ξ = f (xk , di,j , ϕi,j ) = argminS∈I ( i,j,k∈S ). We call |S| the coalition which has the optimized value ξ the best minimal winning coalition(BMWC). The value xk is a prime number, which can be a unique number such as the robot’s ID. di,j is the distance between two robots i and j. ϕi,j is the angle between the two robots i and j. e , f ,and g are adjustment constants. I is the set of minimal winning coalitions, and S is a minimal winning coalition in I. ξ has the minimum value of each minimal winning coalition. The deﬁnition of ξ considers the angle and distance between robots in the minimal winning coalitions as well as the intrinsic value of, such as an id of each robot to come up

6

K. Cheng and P. Dasgupta

with a unique value for each coalition. An example of determining the BMWC is shown in Figure 1 (b). If there is no winning coalition, we just keep the status of each robot unchanged. As veto players are present in all winning coalitions, we can check the exist of veto players of a voting game to determine the nonemptiness of the core. Also we can ﬁnd the unique and stable BMWC of the voting game as proved in [5]. Theorem 1. A minimal winning coalition with veto players is in the core. Proof. For T ⊇ N , let M WT denote the set of all best minimal winning coalitions. There exist A ⊆ T with veto players. Suppose B belongs to the core, and B = {j} A, which means B is not in T. Let xj be the payoﬀ of j. Based on x = v(N ) and the deﬁnition of the core in section 2 , i i∈N i∈B xi ≥ v(B), ∀B ⊆ N . i∈B xi = i∈A xi + xj ≥ v(B) > v(A). However, A is in the winning coalition set T, so the payoﬀ v(A) ≥ v(N ) ≥ v(B). Therefore, there can not exist a set B which belongs to the core, but does not belong to the minimal winning coalition set T with veto players. Thus, if we can ﬁnd a best minimal winning coalition A, it must be the core. Theorem 2. The ξb value of best minimal winning coalition is smaller than any ξ of minimal winning coalitions. Proof. Suppose, there exists a value ξsmall which is the minimal ξ of minimal winning coalitions. There also exists a positive real number ε, where ε = ξb − g×xk +e×di,j +f ×ϕi,j ξsmall . Based on the BMWC, ξb = argminS∈I ( i,j,k∈S ), there |S| exists a distance between two robots or the robot id that is a negative real number. However, it is not true for physical robots. Thus, ξb is the smallest value of all ξ value of minimal winning coalitions. 3.3

Algorithm Analysis of Finding the BMWC

There are three steps to ﬁnd the BMWC [5]. First, we need to ﬁnd the veto players in the communication range that have more coverage capability than the non-veto players. In other words, veto players can get beneﬁts in the winning coalition. Next, we can use the set of veto player to ﬁnd the minimal winning coalition set. As proved in Theorem 3, the minimal winning coalition with veto players is a stable coalition. Finally, we can ﬁnd the unique BMWC while different strategies. In [5], we described an O(n3 ) time complexity method as the baseline algorithm to fulﬁll these steps. Here we describe an algorithm that can reduce the worst case time complexity for ﬁnding the BMWC from cubic time to log-linear time without guaranteeing optimality. To guarantee that the BMWC is optimal, we also describe a quadratic time algorithm. Veto players are the players who appear in all winning coalitions. Therefore, if we remove all the veto players from a certain coalition, the resulting coalition should not remain a winning coalition anymore. In other words, if the coalition C = {N − I} ∪ {I} is a winning coalition and I is the set of veto players, then the players in {N − I} can never form a winning coalition by themselves. We

Multi-agent Coalition Formation for Distributed Area Coverage

7

function BMWCGreedy returns Best minimal winning coalition set BM W C{ inputs: veto player set I; all player set N ; positive number Quota; variables: temp set S, C; player Pi ; positive number Wmax , Vtmp ; C = I; S = N \I; Wveto = 0; for each Pi in I Wveto = Wveto + W eight(Pi); end for Wmwc = Wveto ; sort S in descending order; // The veto players can form the // minimal winning coalition by themselves. if Wveto ≥ Quota then BM W C = C; else for each Pi in S C = C ∪ {Pi }; Wmwc = Wmwc + W eight(Pi ); if Wmwc ≥ Quota then BM W C = C; break; end if endfor end if return BM W C; } Fig. 3. The greedy algorithm with O(n log n) time used to ﬁnd the best minimal winning coalition

use this intuition to determine the set of veto players in linear time as shown in Figure 3. First, we calculate W , the sum weights of all the players in linear time. Then, for each player i, we calculate W−i as the sum of weights excluding that player. We add the player i to the set of veto players I if the W−i falls below the quota without player i. To get the unique BMWC, we propose two methods: a greedy method as shown in Figure 3 and a heuristic method as shown in Figure 4. The heuristic method can ﬁnd the optimal solution in O(n2 ) time. In contrast, the greedy method can only guarantee to ﬁnd a suboptimal solution. However, it can ﬁnd this solution in O(n log n) time. Thus, there is a tradeoﬀ between the

8

K. Cheng and P. Dasgupta

function BMWCHeuristics returns Best minimal winning coalition set BM W C{ inputs: veto player set I; Minimal winning coalition set M W C; variables: ith member of MWC set Mi ; set S; positive number ξwmc ; player Pi , Pc ; //Pc is the virtual player at the central point // of all the veto players with the average angle. ξwmc = 0; Location(Pc ) ← arg i∈I (Distance(Pi )); Angle(Pc ) ← arg i∈I (Angle(Pi )); for each Mi in M W C S = Mi \I; for each Pi in S ξwmc = Distance(Pi , Pc ) + Angle(Pi , Pc ) + ID(Pi ); end for if ξwmc is minimal one then BM W C = Mi ; end if end for return BM W C; } Fig. 4. The heuristic algorithm with O(n2 ) time used to ﬁnd the best minimal winning coalition

computational time and the optimal solution. In Figure 3, we ﬁrst sort the non-veto player set in descending order by a single robot’s weight. By selecting the sorting algorithm judiciously and assuming that the veto player weights are in randomized order, the worst case time complexity of the sorting algorithm becomes O(n log n). Although there can be more than one MWC, we only choose the ﬁrst MWC we get as the BMWC. For example, if we have a eight player voting game with the weight set {1.0, 1.0, 1.0, 0.3, 0.3, 0.3, 0.2, 0.1} and quota q = 3.6, we get veto player set {P1 , P2 , P3 }, and non-veto player set {P4 , P5 , P6 , P7 , P8 }. Based on greedy method, we can easily choose MWC set {P1 , P2 , P3 , P4 , P5 } as BMWC. However, this method does not calculate the remaining MWCs: {P1 , P2 , P3 , P4 , P6 }, {P1 , P2 , P3 , P4 , P7 , P8 }, {P1 , P2 , P3 , P5 , P6 }, {P1 , P2 , P3 , P5 , P7 , P8 }, and {P1 , P2 , P3 , P6 , P7 , P8 }. It stops after the ﬁrst solution is found. In the worse case, P6 , P7 , P8 are much closer to P1 , P2 , P3 than P4 , P5 . We can easilty see that the solution found by the greedy algorithm {P1 , P2 , P3 , P4 , P5 } is not the optimal coalition. The heuristic method gives a way to ﬁnd the optimal coalition, as shown in Figure 4. By checking every member of all MWC sets with all members of the

Multi-agent Coalition Formation for Distributed Area Coverage

9

veto player set, we can ﬁnd the optimal coalition with the minimal ξ value. Instead of calculating the distance and the angle between non-veto players to veto players in each MWC, we compute these values with the virtual player Pc , which is the centroid of all the veto players. Also, to ﬁnd Pc in linear time, we compute the average coordinates, and the angle of each veto player. Finally, we remove veto players inside each of the MWCs. Suppose, we have m MWCs, k veto players, and n total players. The computational time of the heuristic method is O(m × (n − k)). In the worse case and the average case, the computational time is O(n2 ). However, in the best case, the computational time becomes linear, which c = n − k is a constant value.

4 4.1

Adaptive Flocking Teams with WVGs Team Shape Maintenance

In the former section, we described how robots ﬁnd the best team based on a weighted voting game module. After they built their group, how to keep robots moving in a particular formation is another important issue. In other words, in the higher level, robots select a team based on maximizing their single utility or payoﬀ to fulﬁll a certain task. In the lower level, robots need to keep their team in an optimal physical shape to get the most team utility and reduce extra moving costs. Our goal is to keep the robot team moving eﬃciently while maintaining a certain shape. We use our former ’V’ shape ﬂocking module, which has a team leader and several followers [3,4,5] to achieve this. Particularly, we select the robot with the highest weight as the team leader in the team. In other words, the team leader has the highest coverage capability. 4.2

WVGs for Team (Re)formations

Based on calculating the best minimal winning coalition, a robot can make the decision of joining a team or leaving a team. In a certain time period, such as 25 time steps, the robot with highest weight of the team will calculate the BMWC. If a team member’s coverage capability is low, it is the leader of the team or a follower of the team, it will not be in the new BMWC and leave the team. When two or more than two teams meet in the communication range, based on recalculation of the BMWC, a team leader may become a team follower, and a team follower may become a team leader. In this process of reformation, the number of robots of each team can also be increased or decreased to gain the maximum coverage capability. In short, the stability and uniqueness of BMWC guarantee the distributed decision-making of each robot can obtain team behaviors.

5

Experiment and Evaluation

We did two classes of experiments: one is used to test the run time of three algorithms including the baseline algorithm, the greedy algorithm and the heuristic

10

K. Cheng and P. Dasgupta

Run Time in Millisecond

10

10

10

10

10

4

the Baseline Algorithm the Greedy Algorithm the Heuristic Algorithm

3

2

1

0

10

20

30

40

50

60

70

80

90

100

Number of Robots

Fig. 5. The running time (in the logarithm scale) of three algorithms for ﬁnding the BMWC with diﬀerent numbers of robots

algorithm; the other is used to evaluate the performance of the utility based method and the nearest neighbor method comparing with our BMWC method by the percentage of coverage in the environment, as well as the coverage redundancy. To evaluate the run time of three algorithms to ﬁnd BMWC, we implemented them in a laptop computer with 2 GHz processor run time and 1 GB memory. These hardware are similar to the hardware of CoroBot [6]. We ran the three algorithms respectively over 10 times using 10 to 100 robots, as shown in Figure 5 in the logarithm scale. In fact, these three line should show no-linear relationship in the normal scale. Not surprisingly, the greedy algorithm ran fastest within 20 millisecond for all the tests. The lowest one is the base line algorithm, which ran over 1 second for 100 robots. The middle one is the heuristic algorithm which has 10 to 100 millisecond run time by increasing the number of robots. All these Table 1. The Kruskal-Wallis statistics tests of The ξ values of the baseline algorithm and the greedy algorithm using diﬀerent numbers of robots Algorithms Mean Rank Baseline 10.00 Greedy 11.00 Test Score Chi-Square 0.143 df 1 p-value 0.705

Multi-agent Coalition Formation for Distributed Area Coverage

11

Fig. 6. The ξ values of the baseline algorithm and the greedy algorithm using diﬀerent numbers of robots

algorithms ran very fast around 1 millisecond when there are less than 10 robots. However, if the numbers of robots or agents are really large, the greedy algorithm becomes a better choice. As the greedy algorithm is an approximate solution. We checked the ξ value between the baseline algorithm and the greedy algorithm, as shown in Figure 6. The average ξ values of the two diﬀerent algorithms are nearly the same. The range of ξ value is larger for the greedy algorithm than the base line algorithm. To statistically analysis the ξ values, we used Kruskal-Wallis statistics tests as our sampling module. In Table 1, since p-value = 0.705 > 0.05 = α, we accept the null hypothesis which means the two sample groups are statistically equal. Thus, at α = 0.05 level of signiﬁcance, there exists enough evidence to conclude that there is no diﬀerence among the ξ value of the two diﬀerent algorithms based on the test scores. In other words, if we use the greedy algorithm, we can get the approximately optimal solution.

Table 2. Diﬀerent parameter values used in the experimental settings for our simulations Symbol |N | ψ ξc r q s

Parameter Value Number of robots {5, 10, 20, 40} Size of the arena {4 × 4 m2 } Communication range {0.7 m } IR Sensor range {4-7 cm } Quota for WVGs {0.5 Wmax } Speed of a robot {2.8 cm/s}

12

K. Cheng and P. Dasgupta

Table 3. Percentage of the environment that is covered by 5, 10, 20, or 40 robot with diﬀerent decision-making strategies

Max. Avg. Min. Max. Nearest Neighbor Avg. Min. Max. Utility Based Avg. Min. BMWC

6

Number of robots 5 10 20 40 88.29 97.99 99.39 100 85.78 97.17 99.27 99.79 82.38 96.21 99.20 99.63 84.50 97.33 99.39 99.52 79.05 96.17 98.88 99.38 71.94 95.13 98.35 99.16 82.17 97.24 99.17 99.48 77.77 95.61 98.48 99.00 70.87 93.22 98.26 98.80

Future Work

In this paper, we proposed and veriﬁed a technique of computing the weights of a weighted voting game based on robot’s coverage capability and ﬁnding the stable and unique best minimal winning coalition. We studied the feasibility of our model, and give theoretical proof and solutions to ﬁnd the BMWC. The result of our experiments show that this model can be applied for multi-agent coalition formation for distributed area coverage. We plan to evaluate the robustness of our approach to sensor and actuator noises. We are working towards evaluating our technique with physical robots, such as the CoroBot. We envisage that this technique can support heterogeneous robots with diﬀerent memory and communication limitations as well.

References 1. Bachrach, Y., Elkind, E.: Divide and Conquer: False-Name Manipulations in Weighted Voting Games. In: Autonomous Agents and Multiagent Systems (AAMAS), pp. 975–982 (2008) 2. Bilbao, J.M., Fernandez, J.R., Jiminez, N., Lopez, J.J.: Voting power in the European Union enlargement. European Journal of Operational Research 143, 181–196 (2002) 3. Cheng, K., Dasgupta, P., Yi, W.: Distributed Area Coverage Using Robot Flocks. In: World Congress on Nature and Biologically Inspired Computing (NaBIC 2009), Coimbatore, India, pp. 678–683 (2009) 4. Dasgupta, P., Cheng, K., Fan, L.: Flocking-based Distributed Terrain Coverage with Mobile Mini-robots. In: IEEE Swarm Intelligence Symposium, pp. 96–103 (2009) 5. Cheng, K., Dasgupta, P.: Weighted Voting Game Based Multi-robot Team Formation for Distributed Area Coverage. In: AAMAS 2010 Workshop on Practical Cognitive Agents and Robots, Toronto, Canada (2010)

Multi-agent Coalition Formation for Distributed Area Coverage

13

6. http://robotics.coroware.com/corobot (accessed by March 25, 2010) 7. Correll, N.: Coordination Schemes for Distributed Boundary Coverage with A Swarm of Miniature Robots: Synthesis, Analysis and Experimental Validation, Ph.D. Dissertation, Ecole Polytechic Federale Laurence (2007) 8. Deng, X., Papadimitriou, C.H.: On the complexity of cooperative solution concepts. Math. of Oper. Res. 19(2), 257–266 (1994) 9. Elkind, E., Goldberg, L.A., Goldberg, P.W., Wooldridge, M.: Comutational Complexity of Weighted Threshold Games. In: Proc. of AAAI 2007 (2007) 10. Hazon, N., Kaminka, G.: On Redundancy, Eﬃciency, and Robustness in Coverage for Multiple Robots. Robotics and Autonomous Systems 56(12), 1102–1114 (2008) 11. Koening, S., Szymanski, B., Liu, Y.: Eﬃcient and Ineﬃcient Ant Coverage Methods. Annals of Mathematics and Artiﬁcial Intelligence 31(1-4), 41–76 (2001) 12. Myerson, R.B.: Game Theory. Harvard University Press, Cambridge (1997) 13. Shoham, Y., Leyton-Brown, K.: Multiagent Systems: Algorithmic, Game Theoretic and Logical Foundations. Cambridge University Press, Cambridge (2009) 14. Svennebring, J., Koening, S.: Building Terrain Covering Ant Robots: A Feasibility Study. Autonomous Robots 16, 313–332 (2004) 15. Taylor, A., Zwicker, W.: Simple Game: Desirablility Relations, Trading, Pseudoweightings. Princeton Univerity Press, Princeton (1999)

Modeling Warehouse Logistics Using Agent Organizations Marcel Hiel, Huib Aldewereld, and Frank Dignum Utrecht University - Institute of Information and Computing Sciences P.O.Box 80089, 3508 TB Utrecht, The Netherlands {hiel,huib,dignum}@cs.uu.nl

Abstract. Warehouse management systems are traditionally highly optimized to a speciﬁc situation and do not provide the ﬂexibility required in contemporary business environments. Agents are advocated to provide adaptiveness and ﬂexibility, and have been used to solve speciﬁc problems in the warehouse logistics domain. However, for creating a general warehouse management system, it is not clear for developers how and when to implement them. In this paper, we describe many of the design decisions when constructing a warehouse management system, and illustrate how and when agents (and agent-organizations) are useful.

1

Introduction

Traditional control systems for warehouse management are centralized monolithic systems that are highly optimized for a speciﬁc situation. However, in the current business environment, where mergers, acquisitions and rapid product development happen frequently, companies are in a continuous state of ﬂux. The warehouses used by these companies (as customer or owner) are therefore subject to a lot of diﬀerent changes. Examples of such changes range from withdrawal or addition of (types of) products, slow moving products becoming fast moving products (and vice versa) to the addition or removal of hardware. The hardware that is used in warehouses has been subject to evolution and a component-based approach is currently used to create for example, picking stations and conveyor belts. However this evolution was not reﬂected in the software that controls these machines and many of warehouse management systems are still centralized and monolithic. In order to introduce more ﬂexibility, recently multi-agent approaches were proposed as a solution in production and warehouse management (among others, [1,2,3,4,5]). Agents, characterized by properties such as autonomy and pro-activeness serve as an alternative for the centralized approach, potentially alleviating problems in ﬂexibility, robustness and scalability. In order to structure these multi-agent systems and ensure that global business objectives are met agent organizations were introduced. Although agents promise to alleviate problems in ﬂexibility and adaptiveness, design questions such as: How many components should the system consist of? When to use agents? What should C. Guttmann, F. Dignum, and M. Georgeﬀ (Eds.): CARE 2009/2010, LNAI 6066, pp. 14–30, 2011. c Springer-Verlag Berlin Heidelberg 2011

Modeling Warehouse Logistics Using Agent Organizations

15

be considered as an agent? and how to use an agent-organization? let alone the question what all this means for the eﬃciency of the overall system hamper the commercial usage of agents. In this paper, we provide a structured overview of a number of design decisions in the domain of warehouse logistics. These design decisions help developers determine whether an agent-based approach should be considered. To be able to make these design decisions, we distinguish three aspects of a system, namely the data (information), the business rules (decisions) and the communication between components (and/or agents). Based on these aspects, we specify how to design a warehouse management system based on agents. We use our experience in creating a warehouse management simulation tool to illustrate and clarify these design decisions. The paper is structured as follows: Section 2 provides background information on the warehouse logistics domain and discusses our simulation prototype. After this, in Section 3, we describe when to use agents and how to model them. Based on the model of a single agent, we present the design decisions and agent organizations for scheduling and planning in Section 4. We relate to other work in Section 5 and conclude this paper in Section 6.

2

Warehouse Logistics

We conduct our research within the domain of warehouse logistics. A warehouse stores and collects products for customer orders. These products are typically packed and/or placed in boxes. In this paper, we use the term Transport Storage Unit (TSU), to indicate any box or container that holds products. In the following, we explain this domain in more detail by means of (a hierarchy of) the hardware components, a separation of three layers of operation within a warehouse management system and our simulation prototype. 2.1

Hierarchy of Hardware Components

Figure 1 illustrates an ontology of the hardware components used in our (very simple) warehouse logistics domain. The relations between rectangles indicate “is a” relationships, for example a conveyerbelt is a subclass of transport components. Although in this ﬁgure for every class either no subclass or only one subclass is given, multiple subclasses may exist. For example, a distinction can be made between a automated picking station and an operator picking station (manual picking), both are a subclass of the picking class. Or instead of a conveyerbelt a forklift could be used for transport. The transport class and the conveyerbelt subclass are not always included in the analysis of warehouse management systems (e.g. [6]). Motive for not including them, is that from an abstract perspective the functionality of transport components are included in those of the storage class, namely moving a product from one point to another in some period of time. Or functioning as a buﬀer with time delay between diﬀerent work stations. Advantages and disadvantages of using conveyerbelts are discussed in, among others, [7]. It makes sense to exclude

16

M. Hiel, H. Aldewereld, and F. Dignum

Fig. 1. Component hierarchy

the transport from the model if the hardware component is very straightforward and thus the functionality is given by a simple function that indicates the time it takes to move TSUs from one place to another. However, modern conveyorbelts can have more functionalities. They can be used to route TSUs and sometimes as an additional buﬀer (by letting TSUs circle around). So, the capabilities of the hardware determine (partly) whether the components should be modeled separately or not. In our research, where we analyze whether diﬀerent components should be considered as agents we prefer to model transport separately (even though it is very simple in the simulation). Although we will not discuss hardware conﬁgurations much further in this paper, we also want to mention that the conﬁguration of course impacts eﬃciency and ﬂexibility of the system. E.g. in our simulation depicted in ﬁgure 2 we see that the conveyorbelt connects all other components. This seems logical for such a small set-up. However, if one has 20 or more storage racks it might be very ineﬃcient to connect all of them through the same conveyorbelt. In the other hand, making 10 modules like the one depicted in ﬁgure 2 might impact the ﬂexibility (because storage can then only serve a limited amount of picking stations). So, both functionality as well as conﬁguration of the hardware puts constraints on the possibilities of the software and should be taken into account from the start when designing the agent system that controls the logistics. 2.2

Planning, Scheduling and Plant Layer

In warehouse logistics typically a separation is made between three layers of operation, namely the plant (execution), scheduling and planning. We deﬁne these three layers within our scenario as follows: Planning: “Planning is the process of generating (possibly partial) representations of future behavior prior to the use of such plans to constrain or control that behavior”[8]. In our domain this means that orders are assigned to be handled by certain components. In our plans there is no concept of time. Scheduling: “Scheduling is the problem of assigning a set of tasks to a set of resources subject to a set of constraints” [9]. In our domain scheduling

Modeling Warehouse Logistics Using Agent Organizations

17

encompasses two aspects, namely providing the machines with actions to perform and transferring TSUs from one component to another. Scheduling decides at what time these aspects occur. Plant (Execution): The plant layer represents the hardware components and the people (called operators) that process products in a workstation. The distinction between these layers creates a separation of concern which encapsulates a number of design decisions and therefore makes it easier to develop a warehouse management system. Furthermore, it makes decoupling of the control system easier and allows outsourcing part of the warehouse management to diﬀerent (specializing) companies. For example, a company may specialize in creating the hardware (and device controllers) while other enterprises can take care of the scheduling and/or planning. 2.3

Factors of Complexity

Although both the ontology of hardware components and the layering creates a separation of concern that makes the design easier, other factors make the design of a warehouse management system more complex. We distinguish here two factors: Complexity imposed by the hardware components Hardware components provide certain capabilities in handling TSUs, such as moving or storing them. These capabilities restrict the possible schedules and thereby also the plans that can successfully be handled in this warehouse. On the other hand, if the component contains a long list of capabilities then the number of diﬀerent situations it is capable of handling grows rapidly. The complexity of the hardware components should be reﬂected in the controlling software if the hardware is going to be optimally used. Complexity of the scenarios Besides the separation of these three layers, other properties of a warehouse exist that make the domain more complex. In warehouse logistics two major factors were stated to inﬂuence the complexity, namely the number of orderlines and the number of stock-keeping units (SKUs) [10]. From these two factors other factors such as for example the picking strategy or special storage conditions can be deduced. However, these two factors were gathered from existing warehouses and do not consider that a warehouse may change. Warehouses may be changed during their life-time to accommodate changes in the market by adding or replacing hardware components. Dynamism is therefore an important aspect, which results in additional complexity by being forced to handle exceptions and changes to hardware components. Below we provide a number of example factors, among others, that increase the complexity of a warehouse management system: Orders and products: The number and diﬀerent types of products that can be processed (sizes, weight, storage and handling conditions etc). Also the number of orders and orderlines as well as prioritization of orders.

18

M. Hiel, H. Aldewereld, and F. Dignum

Hardware: The number and homogeneity of the hardware components in the warehouse. For example the number of diﬀerent types of picking systems used. Processes: What processes are supported by the WMS? For example, is order batching, cross-docking or custom clearance supported? Exceptions: How well can the warehouse management system handle exceptions in the execution layer? As there are many diﬀerent types of exceptions, being able to handle them all creates a complex system. Changes: How well are changes anticipated? For example, changes in hardware components, or changes in connected software (for example, CRM or ERP applications). In principle one can say that the more of the factors above hold for the warehouse system you are designing, the more likely it is you will beneﬁt from an agent (organization) approach. Basically the argument is that agents modularize the complexity such that it can be handled more easily, while using an agent organization takes care that global (eﬃciency) constraints are taken into consideration. 2.4

Prototype Simulation

Most design decisions only become clear when implementing the system. For this reason we developed a simulation tool that allows you to try out diﬀerent conﬁgurations and parameters before actually building the expensive hardware and software system. Figure 2 illustrates a basic set-up for evaluating the design of a warehouse management system. In this prototype we have a conﬁguration of two miniloads, one conveyerbelt and two workstations. The orange boxes in Figure 2 represent moving TSUs. The choices we made in creating this simulation tool, i.e., the warehouse design, and the possible alternatives we describe in the following sections.

Fig. 2. Simulation prototype

Modeling Warehouse Logistics Using Agent Organizations

3

19

Modeling Components

An important design decision in our research is whether a component should exert adaptive behavior, i.e., whether it should be considered an agent. Based on this decision either a traditional object-oriented approach or an agent-based approach is better. If an agent-based approach is chosen, then the follow-up question is how to model these agents. We discuss these two design decisions in the two sections below. 3.1

What Should Be Considered an Agent?

The decision whether a component should be considered an agent, depends on the complexity (and dynamicity) of the environment (c.f. [11]). Making adaptive software (agents) is intellectually more challenging and therefore more costly. Although the concept of an agent is intuitive and easy to understand, it requires skilled developers that know how to program active components (similar to thread programming) to be able to ensure that the developed software functions correctly. In a simple environment (small number of SKUs and low number of orderlines) with little anticipated changes (market is stable and unlikely to change) using agents would be more costly than a system based on traditional software engineering techniques. Furthermore, it would not yield a system that is performing better. However, as the complexity of the environment rises due to introduced or anticipated business changes, the number of diﬀerent situations that have to be handled rise with it. I.e., the number of SKUs rises and/or the number of orderlines becomes bigger. As the number of diﬀerent situations rises, a point arrives where programming all these situations becomes ineﬃcient and costly. At this point, adaptive components that employ AI techniques such as machine learning or logical reasoning become desired. Agents, advocated to embody adaptive behavior, become the choice for modeling and implementing these systems. For example consider the conveyerbelt in Figure 2. Assume that although it is cyclic, traversing the cycle is not allowed. In this setup the conveyerbelt acts as buﬀer with a single delivery time. Therefore, it could be modelled as a table with the time to traverse to a certain destination. This table could be modelled as part of the miniloads. However, if this assumption is dropped, i.e., TSUs are allowed to travel the cycle, then the conveyerbelt becomes more complex. It could then be modelled as a static component which provides a serie of times at which it is able to deliver the TSU. If more cycles are introduced, and the conveyerbelt is also able to vary the speed of each of these cycles, then the number of situations of when and how a TSU can be transported becomes larger. A larger number of diﬀerent situations provides more ﬂexibility to the overall warehouse system. For instance in case of an exception, e.g., the destination is oﬀ-line, then a conveyerbelt may decide to keep this TSU on a certain loop for a speciﬁc time or decide to send the TSU back to the origin. However, in order to exploit this ﬂexibility the conveyerbelt should be modelled as an agent that is capable of negotiating with the miniload and/or workstation.

20

3.2

M. Hiel, H. Aldewereld, and F. Dignum

How to Model an Multi-agent System?

Next to the question whether a component should be considered to be an agent, the question rises of how to model the agents. In our research, we use MASQ [12] to help make this decision. In the following we ﬁrst brieﬂy introduce MASQ after which we use it to model agents in our warehouse logistics domain. MASQ. MASQ is a meta-model for structuring the basic elements that compose the interaction process of agents, environments, organizations and institutions. The model deﬁnes four perspectives on agent-based interaction, according to two axis, introducing a separation of internal/external and of individual/collective perspectives. For this paper, we focus mostly on the separation between an internal and external aspect of an agent, see Figure 3.

Fig. 3. The mind-presence(body) relation [12]

In MASQ, the separation of internal and external means that an agent has a single mind but can have several presences (also called bodies). A presence can be either a physical, virtual or social representation of the agent in an environment. The rules and protocols that agents use to interact in their environment are called spaces. A distinctive feature of MASQ is that agents can have presences in multiple spaces. The presences are the means how an agent can interact with its environment (space), typically called capabilities. The mind corresponds to the internal structure of an agent and which also serves as the “glue” between the diﬀerent presences. Using MASQ in Warehouse Logistics. As MASQ supports multiple spaces we create for each of the layers in a warehouse management system, namely: plant, scheduling and planning, a space. Each space incorporates its own implemented business rules and interaction protocols and thereby improves modularity and maintainability.

Modeling Warehouse Logistics Using Agent Organizations

21

Fig. 4. The diﬀerent presences of the agents in a warehouse

Figure 4 illustrates the presences of the agents for each layer in our warehouse scenario. A presence is represented by a box. An agent is represented by a dotted line around a group of boxes. For instance, the miniload agent has three presences, one in each layer. As can be seen from Figure 4, the conveyerbelt agent is not involved in the planning, and has only a scheduling and plant presence. This means that the conveyorbelt is not taking into account while doing the planning. However, it can inﬂuence the scheduling. The broker on the other hand does not have a physical representation and is only a virtual entity with a presence in the planning space. We explain the function of the broker in more detail in Section 4.2. Business rules (i.e., the decisions of the agent) are located and handled in the respective presences of the agent. Business rules are closely related to the functions that implement the decision. Therefore placing the business rules in the presences creates a modularity per implementation of a business rule. For example, two types of conveyerbelt schedule presences can be created, one that supports only FIFO and one that supports more intelligent scheduling. Furthermore having the business logic in the presences, and not in the mind, had two additional advantages. First it supports simultaneous development in diﬀerent spaces as the overlap (in the mind) is minimized. Second, it prevents the agent’s mind becoming more and more complex with each presence that it controls. This complexity would impair any potential ﬂexibility and adaptability of the agents. Thus, having the business rules in the presences creates a separation of concerns per space and improves maintainability (and traceability) in case speciﬁc business rules have to be adapted. In our model the mind functions as the “glue” that keeps all the presences of an agent together. The mind triggers the tasks that needs to be performed in the separate presences and any decisions that have to be taking for executing these tasks are handled in the presences themselves. Figure 4 suggests that all components that can be divided into mind and presences should be considered agents. However, as described in Section 3.1, it can be questioned whether the conveyerbelt should be modelled as an agent. Especially as the conveyerbelt is modelled in Figure 4 in only two layers. Advantages of considering it as an agent are the homogeneity of the system (all components are

22

M. Hiel, H. Aldewereld, and F. Dignum

agents), and furthermore, if more complexity is added later, having it already implemented as an agent makes redesign easier. Also, the argument that agents are more costly becomes less important if the other components are already modeled as agents. In the following, we describe the organizational model for scheduling and planning. As the plant layer is solely concerned with controlling the hardware, we do not have an organizational model for the plant layer.

4

Agent Organizations

An agent organization is a structure that supports the overall model of a multiagent system. This model restricts the behavior of the agents functioning in that organization and ensures that overall objectives are met. In this paper we use the OperA framework [13] for representing agent organizations. OperA contains three aspects: Social: The social aspect involves roles, objectives and dependencies between roles based on objectives. Roles identify activities necessary to achieve organizational objectives and enable the developer to abstract away from the speciﬁc actors that will perform these activities. Roles are dependent on each other for the achievement of their objectives and therefore require interaction. Interaction: In systems containing multiple components, interaction is an important aspect. Through interaction the objectives of the organization are realized. The interaction in OperA is structured in scenes where each scene represents the parties that are involved for reaching a certain objective. The interaction is structured using landmarks. A landmark describes the state that should be reached during the interaction. By specifying only landmarks, the actual protocols may vary (as long as they achieve the same results). Norms: Norms specify how actors ought to behave in a certain situation. Business rules are one way of implementing norms. However, one of the most distinctive features of norms is that agents might violate them if circumstances require. E.g. if orders should be handled in a FIFO order an agent might deviate from this rule in case one of the products for that order is not in stock or when a emergency order arrives. Within the framework one also describes how the system should handle the violation situations (which lends the system its ﬂexibility). 4.1

An Organizational Model for Scheduling

As stated in Section 2, scheduling in our warehouse management system concerns two aspects, namely providing actions to execute for the hardware components and transferring TSUs between (hardware) components. The ﬁrst aspect, providing actions for the plant, is an internal aspect of the agent. The plant presence states that it is ready to execute the next action and the scheduling presence of

Modeling Warehouse Logistics Using Agent Organizations

23

the same agent provides this action. On this level also failure of actions in the plant can be handled that have consequences for subsequent schedules. The second aspect, namely transferring TSUs between components, can be modeled in a simple organizational model. The model contains only two roles, namely the sender of a TSU and the receiver of a TSU. As we schedule here the physical transfer of a TSU from one hardware component to another, a constraint is that the scheduling presences that communicate must be physical neighbors. Figure 5 illustrates the communication protocol for establishing a schedule for transferring a TSU and the message structure of the messages used in this protocol. In this ﬁgure two design decisions are hidden. The ﬁrst design decision applies in the situation when a component has multiple channels to a neighboring component. If a TSU needs to be transferred to that component, it can be send on any of these channels. The decision which channel should be selected is a design decision. Either the sender or receiver selects the channel. In this protocol only the receiver has this information and therefore makes the decision.

Fig. 5. Communication for transferring a TSU

The deadline in the communication protocol indicates the second design decision, namely what to do if a component is not ready to receive a TSU. For example if the buﬀers of the component are full or if it is oﬄine. Either the sender or the receiver has to maintain a list of sent messages in order to track whether a TSU is scheduled to be transferred. For instance, if the receiver does not reply to the request, this means that it is not ready to receive/schedule the TSU. If the sender maintains this list after a certain time it will send the request again. Advantage of this approach is that it also covers the situation if the receiver was oﬄine for a period of time. If the receiver maintains this list, then it may never respond as it did not receive the request. The deadline in Figure 5 indicates that the sender will maintain this information. Note that we did not specify any landmarks for this interaction protocol. Motive is that the only sensible landmark in this interaction would be: ”TSU is transferred”. The protocol given is the simplest, but in the mean time most

24

M. Hiel, H. Aldewereld, and F. Dignum

obvious implementation protocol that will reach this landmark. If the channel selection also had to be dealt with and possibly other aspects on the transfer one might conceive of other protocols that would still achieve this same landmark. Furthermore, a landmark violation indicates a problem (for example an agent is oﬄine) that is already detected (and handled), see previous paragraph. The same line of reasoning can be used for norms. Norms are not required for scheduling as the constraints for scheduling are hard constraints imposed by the planning layer. 4.2

An Organizational Model for Planning

There are several design decisions in the agent-based planning approach for warehouse management. We divide these decisions based on three questions, namely (1) Who talks to whom? (2) Who to talk to for delivery/replenishment/relocation?, and (3) How to talk?. We discuss these three questions in detail below. After these questions, we shortly discuss alternative organizational models that serve to illustrate the design decisions implicit in an organizational model. Who talks to whom? This question is answered via an organizational structure. The organization deﬁnes the parties, and the way that these parties should interact in order to reach the organization’s objective. Figure 6 illustrates the roles in our warehouse logistics domain. These roles for a large part coincide with the types of hardware components distinguished in Figure 1, the idea behind these classes is that their subclasses all have the same function, i.e., they achieve the same objective. For example, a manual picking workstation or an automated picking workstation both achieve the objective of the picker role. The objectives of the roles in the planning space indicate what type of things will be optimized by the diﬀerent roles. The picker role might have as objective to pick as many TSUs as possible. The storage role might have as objective to minimize storage use or alternatively to maximize the throughput of TSUs. The shipping role is responsible for handling orders and thus will try to minimize the handling time of orders (or the average handling time or maybe the deviation of the average handling time per order). Finally the receiver may want to receive products either in large batches in order to eﬃciently handle the receiving operation or maybe it wants many small batches in order to have as much just in time handling of orders and minimize storage space. Notice that objectives of diﬀerent roles might not be all in line. The places where they contradict is where choices have to be made in their interaction on how to balance the objectives. The origin of the arrows in Figure 6 indicate the initiator of the conversation between these roles. By deﬁning these roles, we allow an agent, and thereby also the hardware component it controls, to adopt a role appropriately for the situation. For example, a miniload, indicated as Storage in Figure 1, may not only play the role of Storage but when necessary also that of Picker (for relocation). In small domains, the roles in the organization will be played by agents (roleenacting agents), however in a larger domain the roles may be played by ’smaller’

Modeling Warehouse Logistics Using Agent Organizations

25

Fig. 6. Organization for Planning

Fig. 7. Nesting of organizations

organizations (role-enacting organizations). Nesting of organizations allows creating hierarchies of modules on the plant layer, thereby supporting modularization per area. Figure 6 can be used as a building block for creating multiple layers. Interactions in the organization of organizations are the same as in the ’simple’ organization. This means that a logical area is split into diﬀerent parts, each with its own agents that play picker, storage, receiving, shipping and broker roles. Figure 7 shows an example where the picker role is enacted by an organization. Who to talk to for delivery/replenishment/relocation? A problem of open dynamic systems is how agents know who they should talk to for a particular service. In warehouse management, three types of services are typically distinguished, namely delivery, replenishment and relocation. Delivery is the actual delivery of the order to the customer. It is where the products leave the system. For example, shipping asks a picker for delivery of an order.

26

M. Hiel, H. Aldewereld, and F. Dignum

Replenishment is the act of asking other components for products in order to be able to process an order. For example, the picker will ask storage for replenishment if it does not have the products needed to pick for an order. Relocation: Products that are not needed are to be stored. For example, a picker that has a TSU containing a product is does not need will try to relocate this TSU to storage. A problem of open dynamic systems is how agents know who they should talk to for a particular service. As agents may join and leave the system the providers for a particular service (deliver, replenish, relocate) can change. Furthermore, even the type of services available at a speciﬁc time may change. This makes it necessary to maintain and keep up to date information on who provides what service. This information is stored and provided by the broker. Although hardware components of the warehouse are not subject to frequent changes, by providing the controlling software the capability of handling diﬀerent situations, performance during abnormal situations can be improved considerably. For example, by incorporating the broker, our system allows hardware components to be taken oﬄine (for instance for maintenance) and keep a (albeit reduced) steady performance.

Fig. 8. Service Brokerage

Service Brokerage is displayed in Figure 8. For more information the interested reader is referred to [14]. The enacting agents register their service or service-need at the broker, who provides the agent with a list of service providers. Furthermore, the broker keeps registered services informed about changes. That is, informing customers about new available providers or unavailability of a provider. How to talk? Besides determining which agent to talk to, it is important to decide on how the agents talk to each other. As a basis for the communication between the agents we use FIPA Agent Communication Language (ACL). FIPA ACL deﬁnes the structure of the message being passed between the agents. A message is composed of the following elements: – Performative (cfp, propose, inform, etc.). – Sender (agent id) – Receiver (agent id)

Modeling Warehouse Logistics Using Agent Organizations

27

– Content (the information) – Conversation ID (to track conversations). In particular, we make use of the FIPA contract-net protocol (see Figure 9, and [15] and [16]). All interaction on the planning layer related to service consumption in our simulation is based on the contract-net protocol. The reason is that it is very simple and eﬃcient to implement. It also appears to be enough for the situations that we wanted to handle. However, one might also consider to use the CNP with decommitment, which might be useful in situations where emergency (high priority) orders arrive frequently (but unexpectedly) and where it pays of to relocate tasks.

Fig. 9. Communication in the planning organization

Figure 9 gives an example of the communication in an organization. Planning of an order goes as follows: Shipping receives an order and translates this in a cfp to the pickers (all agents that have registered to be able to play the Picker role). The pickers answer with a proposal containing the amount of time it would take that picker to complete the order. Shipping weighs the received proposals and

28

M. Hiel, H. Aldewereld, and F. Dignum

sends an accept-proposal to the picker that oﬀers the best proposal. The other pickers receive a reject-proposal message. If the picker needs a resupply to fulﬁll the order, it starts a replenish interaction with the Storage roles. This interaction can interleave the order interaction, as shown in Figure 9 (order interaction is colored red, replenish interaction is colored green). Alternative Organizational Model. In the organizational model presented above, the shipper starts the process by asking the pickers whether they are able to process an order. The pickers in their turn ask storage for replenishment and storage asks receiving for replenishment. The underlying assumption for starting at the picker is that they are the last in the process to handle a TSU, therefore they they should be asked ﬁrst. However, an alternative model is that the shipper does not ask the picker but storage for replenishment. Storage on its turn then asks the picker(s) which one is able to handle the order. Motivation for beginning at the storage and not the picker is that the processing time of the storage (retrieving a TSU from storage) is in general much longer then that of the picker (picking product from a TSU). The above shows that even given an organizational structure one can change the operation of the organization by changing the interaction structure or even just the party that takes the initiative to interact. What is the best solution is not always obvious on forehand, but one heuristic is that roles that are ”bottlenecks” in the logistic process should have the initiative in order to maximize ﬂexibility.

5

Related Work

Diﬀerent types of agent organizations have been proposed to structure a multiagent system. For example, Holons present an approach introduced in MAS by [3]. A survey of diﬀerent types of organizations, such as holarchies, hierarchies and markets, including the advantages/disadvantages of these organizations is presented in [17]. Most research on warehouses focuses on speciﬁc parts, mainly on optimizing certain performance criteria for a speciﬁc hardware conﬁguration or component. for example, order picking or conveyerbelts, research that considers the whole warehouse [6]. A further study, based on literature, on a general design methodology for warehouses is provided by [18]. Conclusion of the authors is that in general all existing approaches agree on the steps, however in which order and to what detail diﬀers. An approach that includes a ﬂexibility analysis in the design is [7]. In their design methodology the authors include a speciﬁc step for evaluation of the design ﬂexibility. Meaning that the design should be evaluated on the type of business scenarios that can be handled. For example, anticipating changes in volume (sudden growth or decline of demand), time (high-priority orders) and quantity (switch from item picking to case-picking). Flexibility in scheduling (and in planning) can be gained using one of two (or both) strategies: 1) schedule as late as possible or 2) by supporting re-scheduling.

Modeling Warehouse Logistics Using Agent Organizations

29

Scheduling as late as possible ensures that no constraints exist that may hamper incoming changes (for example, receiving a high priority order). However, late scheduling is not always possible or desirable. In the situation where a schedule was already made, changes may require re-scheduling. A paper that describes re-scheduling is [19].

6

Conclusion

In this paper, we highlighted a number of design decisions when developing a warehouse management system. We showed how agents can be modeled and how agent organizations can help structure the communication in an intuitive manner. To come to these design decisions we implemented a simulation prototype that we use to test the agent organizations performance. Flexibility, adaptability and adaptiveness are important qualities of software in general and of warehouse management systems in particular. It is our intention to further develop our prototype and experiment with diﬀerent organizational models in order to create a warehouse design tool that is able to provide for each situation the optimal warehouse layout and control strategy.

Acknowledgment This work has been carried out as part of the FALCON project under the responsibility of the Embedded Systems Institute with Vanderlande Industries as the industrial partner. This project is partially supported by the Netherlands Ministry of Economic Aﬀairs under the Embedded Systems Institute (BSIK03021) program. The authors like to thank Jacques Verriet and Bruno van Wijngaarden for their valuable comments and discussions.

References 1. Liu, J.-S., Sycara, K.P.: Coordination of Multiple Agents for Production Management. Annals of Operations Research 75, 235–289 (1997) 2. Maturana, F.P., Balasubramanian, S., Norrie, D.H.: Intelligent Multi-agent Coordination System for Advanced Manufacturing. In: Proc. SPIE (1997) 3. Fisher, K.: Agent-based Design of Holonic Manufacturing Systems. Robotics and Autonomous Systems 27, 3–13 (1999) 4. Ito, T., Abadi, S.M.M.J.: Agent-based Material Handling and Inventory Planning in Warehouse. Journal of Intelligent Manufacturing 13, 201–210 (2002) 5. Chen, R.S., Lu, K.Y., Chang, C.C.: Intelligent warehousing management systems using multi-agent. Int. J. Comput. Appl. Technol. 16(4), 194–201 (2003) 6. Rouwenhorst, B., Reuter, B., Stockrahm, V., van Houtum, G., Mantel, R., Zijm, W.: Warehouse Design and Control: Framework and Literature Review. European Journal of Operational Research 122, 515–533 (2000) 7. Rushton, A., Croucher, P., Baker, P.: The Handbook of Logistics and Distribution Management. Kogan Page (2006)

30

M. Hiel, H. Aldewereld, and F. Dignum

8. Austin Tate in the MIT Encyclopedia of Cognitive Science: Planning, http://cognet.mit.edu/library/erefs/mitecs/tate.html 9. The Computational Intelligence Research Laboratory, http://www.cirl.uoregon.edu/research/scheduling.html 10. Faber, N., de Koster, R., van de Velde, S.L.: Linking Warehouse Complexity to Warehouse Planning and Control Structure. International Journal of Physical Distribution and Logistics Management 32, 381–395 (2002) 11. Silva, V., Garcia, A., Brandao, A., Chavez, C., Lucena, C., Alencar, P.: Taming Agents and Objects in Software Engineering. In: Software Engineering for LargeScale Multi-Agent Systems: Research Issues and Practical Applications, pp. 1–26. Springer, Heidelberg (2003) 12. Stratulat, T., Ferber, J., Tranier, J.: MASQ: Towards an Integral Approach to Interaction. In: AAMAS 2009: Proceedings of The 8th International Conference on Autonomous Agents and Multiagent Systems, pp. 813–820 (2009) 13. Dignum, V.: A Model for Organizational Interaction: based on Agents, founded in Logic. PhD thesis, Universiteit Utrecht (2004) 14. Papazoglou, M.P., van den Heuvel, W.: Service Oriented Architectures: Approaches, Technologies and Research Issues. VLDB Journal 16, 389–415 (2007) 15. Smith, R.G.: The contract net protocol: High-level communication and control in a distributed problem solver. IEEE Transactions on Computers C-29, 1104–1113 (1980) 16. FIPA: FIPA Contract Net Interaction Protocol Speciﬁcation, http://www.fipa.org/specs/fipa00029/SC00029H.html (2002) 17. Horling, B., Lesser, V.: A Survey of Multi-Agent Organizational Paradigms. Knowl. Eng. Rev. 19, 281–316 (2005) 18. Baker, P., Canessa, M.: Warehouse Design: A Structured Approach. European Journal of Operational Research 193, 425–436 (2009) 19. Rubrico, J.I., Higashi, T., Tamura, H., Ota, J.: Online Rescheduling of Multiple Picking Agents for Warehouse Management. In: Robotics and Computer-Integrated Manufacturing, p. xx (2010) (in press)

Agent-Based Crisis Management Huib Aldewereld, John Tranier, Frank Dignum, and Virginia Dignum Utrecht University, The Netherlands {huib,tranier,dignum,virginia}@cs.uu.nl

1

Introduction

Crisis management is the process by which an organisation deals with a major unpredictable event that threatens to harm the organisation or the whole society. Three elements are common to most deﬁnitions of crisis: (a) a threat to the organisation, (b) the element of surprise, and (c) a short decision time [11]. Crisis management, or disaster management, is a relatively new ﬁeld of management. Typically, proactive crisis management activities include forecasting potential crises and planning how to deal with them, for example, how to recover if your company’s computer system completely fails, but also how to evacuate the city of Rotterdam in case of a major ﬂood [10]. Hopefully, organisations have time and resources to complete a crisis management plan before they experience a crisis. Crisis management in the face of a current, real crisis includes identifying the real nature of a current crisis, intervening to minimise damage and recovering from the crisis. The level of preparedness for a crisis can greatly limit the eﬀects of the crises. Simulations provide a means to study the consequences of escalation of crises with respect to structures involved. In the real world, simulations are often enacted using active personnel [6], real people. However, such simulations are expensive, both in terms of the cost of execution and the cost of the time required for the emergency service personnel involved. Computer models of crises, provide a more cost eﬀective means to study the potential of diﬀerent organisational structures in very many diﬀerent scenarios. An added complication in the case of large incidents, is the fact that responses usually require the intervention of several diﬀerent organisations (e.g. ﬁre brigade, police, medics) each with their own regulations, structures and mandates. The Netherlands has an extensive crisis management structure to respond to crisis that aﬀect public order. A layered model, the Coordinated Regional Incident-Management Procedure (GRIP), is a nationwide emergency management procedure based on the severity of the disaster, and allows local, regional and national authorities to take action where necessary. The aim of the procedure is to enable the adequate response to the situation, with a minimum level of disruption of the public life. The procedure is currently used by emergency services, diﬀerent layers of government and government agencies. As a crisis escalates, organisational structures are systematically updated to reﬂect the changes in the nature of the crisis and the number of parties involved: the C. Guttmann, F. Dignum, and M. Georgeﬀ (Eds.): CARE 2009/2010, LNAI 6066, pp. 31–43, 2011. c Springer-Verlag Berlin Heidelberg 2011

32

H. Aldewereld et al.

numbers of organisations involved increases, communication lines change, roles change. However, in this domain agents, roles and groups can be clearly identiﬁed, the interaction patterns between (groups of) agents can be coordinated, and the interaction between (groups of) agents with their changing environment can be modelled. Most current systems provide coordination and planning capabilities for teams of agents, often assuming the emergence of group behaviour. Moreover, highly formalised organisation processes deﬁned by governments and aid agencies deﬁne a strict frame for action. But interactions between organisations are less well deﬁned while hugely impacting the performance. The study of crisis response scenarios is an already active area of research in the ﬁeld of Multi-Agent Systems (MAS) [7,10]. This paper extends this research by examining how a multi-agent system can be used to analyse the eﬀect of interaction structures to coordinate the diﬀerent organisations involved in crisis response. The focus of the paper is thus not on crisis management in itself, but rather the interaction between organisations when they have to cooperate in such a situation. The paper is organised as follows. In section 2 we introduce the concept organisational structure and in section 3 we discuss measures and ways to measure the eﬀect of structure on organisational performance. In section 4 we describe the simulation framework used for the study of organisational structures, which is tailored in section 5 to model the crises management scenario. Results of our experiments are reported in section 6 and discussed in section 7.

2

Organisations

Organisations are used in agent-based systems as a coordination mechanism to express the possible interactions between participants on a high level of abstraction [9]. They determine the roles that play a part in the expressed domain, and deﬁne the possible interactions between these roles. Organisational structure can be deﬁned as that “what persists when components or individuals enter or leave an organisation, i.e. the relationships that make an aggregate of elements a whole” [2]. Structure is thus an independent construct that enables interaction between, or that describes the organisation of, two or more agents in the same system. Organisations describe speciﬁc global objectives, possibly deﬁned by a number of sub-objectives that identify the roles that are played in the organisation. Structure is related to the optimisation of agents’ activities, the management and security of the information ﬂow among agents, and the enforcement of certain outcomes. In our work, we adopt the OperA [1] framework for representing the organisation. (In fact for the limited parts of the organisational models that we use in this paper we could have used other frameworks. However, we will extend this work later on with normative components that are also covered in OperA, but not in other frameworks). OperA proposes an expressive manner for deﬁning organisations by introducing an organisational model, a social model and an

Agent-Based Crisis Management

33

interaction model. OperA explicitly distinguishes between the organisational model and the agents that act in it. Agents adopt roles in the organisation via contracts that specify the organisational rules. The organisational model enables the speciﬁcation of organisational structures in terms of roles and their objectives and relations. The social model deﬁnes the relation between the roles and the actors that enact them. The interaction model deﬁnes the actual interactions between actors to realise organisational objectives. In [4] role relations have been further described in terms of power, coordination and control, that make explicit diﬀerent structural dimensions. Formally, an organisational structure OS is deﬁned as a tuple Roles, RP ow , RCoord, RContr where Roles is a ﬁnite set of roles, and RP ow , RCoord, RContr are three irreﬂexive binary relations on Roles characterising, respectively, the Power, the Coordination and the Control structures. In the case of coordinated response to crisis situations, usually more than one formal organisation is involved. Even though management approaches foresee an overall coordination role (in the Netherlands, this is the LOCC, or coordination center) this functions as a point of contact for the organisations involved and has often no direct formal power over the diﬀerent actors on the ﬁeld. There is therefore a need to analyse the consequences of interaction between diﬀerent organisations, each with a possibly diﬀerent form of coordination structure.

3

Structural Evaluation

Organisational structures determine for a large part the characteristics of the organisation. In fact, considerations about structure explain why in certain environment conditions, for instance, a network structure is more ﬂexible than a hierarchy, but a hierarchy is more eﬃcient. As discussed in the previous section, in crisis management situations diﬀerent organisations are involved, which are dependent on each others actions but are coordinated independently. In the scenario used in this paper, medics and ﬁremen belong to two diﬀerent organisations, each with its own regulations and coordination structures, but are dependent on each other in the ﬁeld. The interaction diagram (part of the OperA model) depicted in ﬁgure 1 depicts this dependency: medics can only safely reach a casualty after ﬁremen have secured the area.

Fig. 1. Interaction structure for crisis management scenario

34

H. Aldewereld et al.

We are interested in identifying the consequences of organisational conﬁgurations to the eﬃcacy of solving the crisis situation at hands. According to management literature, organisational performance is characterised by robustness, ﬂexibility and eﬃciency. The work described in [5] provides a deﬁnition of organisational properties in terms of graph measures, using a graph representation of organisations. Here we brieﬂy recall the main intuitions behind the characterisation of these three organisational properties. Robustness. “Robustness is simply a measure of how stable the yield is in the face of anticipated risks. That is, the maintenance of some desired system characteristics despite ﬂuctuations in the behavior of its component parts or its environment”. [12]. That is, robustness asks for redundancy in the power and coordination structural dimensions needed for distributing tasks within an organisation. Flexibility. Flexibility is the ability of an organisation to cope with changing tasks. This requires, roles’ capabilities to be diversiﬁed. Intuitively, this means a less articulated power structure. Eﬃciency. Eﬃciency mostly refers to the amount of resources used by the organisation to perform its tasks. organisational structure plays a role in this sense, since “links are not without cost in a social systems” [8]. This explains why tree structures usually are very eﬃcient organisational forms. It can be intuitively seen that it is not possible to maximise all three measures simultaneously. Furthermore, graph theoretical reasoning on the eﬀect of organisational structures on these measures does not enable the incorporation of dynamics of the environment. The approach taken in this paper, complements these theoretical approaches with simulations of environment changes. Crisis management scenarios reﬂect dynamic environments where changes occur uncontrollably and at a possibly fast pace. Assuming the bounded rationality and resources of agents, it is easy to understand that agents in the ﬁeld are not able to perceive and deliberate about all changes. On the other hand, they are probably able to react quickly to the events they do perceive. This leads to a situation in which, when left to its own resources, each agent takes a local optimisation strategy, which may not be the most appropriate at global level (that is, while the ﬁreman agent is dealing with a minor local ﬁre that it has identiﬁed, it is not able to contribute to extinguish a much larger one outside its scope). On the other hand, if agents must report their perceptions to one (or more) controller that will decide on the global course of action, they may be not quick enough to react.

4

Organisational Simulation

Following the discussion in the previous sections, the aim of our research is to develop a simulator that enables the study of the eﬀects of organisation structures on the performance of societies consisting of multiple agents. We are interested in investigating both the properties of such systems, and the degree of

Agent-Based Crisis Management

35

complexity necessary to build agents that are able to reason about organisation structure. The simulation developed for crisis management meets the following requirements: – be simple enough to enable empirical evaluation of the results. – be complex enough to emulate real world situations where organisation structure really matters. As basis for the simulation of crisis management, we use ABC Lab which allows to describe and simulate organisational conﬁgurations taking into account individual, social, and environmental factors. For a more extensive description of ABC Lab see [13]. ABC Lab is an implementation of the MASQ meta-model for MAS, which is described extensively in [3]. The MASQ meta-model has been designed with the aim of describing a MAS in all its aspects (agents, environment, interaction, organisations and institutions). The MASQ meta-model provides four basic constructs – Mind, Object/Body, Space, Culture – to describe a complex social system. All these concepts are related as follows: a mind is embodied in one or several bodies; a body belongs to a space which is its context of activity; a space establishes a structure that relates its objects and bodies to each another; minds are connected to the cultures to which they belong. Finally, a mind can act upon and perceive diﬀerent spaces through its bodies, and interpret what occurs in those spaces according to its cultures. Using the MASQ model, we see ﬁreﬁghters and medics as having a mind, encapsulating their own decision-making strategies, and a body, that is their representation in the environment. Cultures are evidenced by the diﬀerent organisation structures and regulations used as coordination mechanisms by diﬀerent groups. Agents interact in a virtual space that describes the crisis environment, containing also ﬁres, casualties and other ‘geographical’ components.

5

Simulation Scenario

In this section, we describe the application of ABC Lab to the design of a simulation environment for the crisis management scenario. First, we describe the overall model for crisis management. Then we describe speciﬁc experimental scenarios representing diﬀerent environment conditions. 5.1

The Model

In the scenario, we consider two diﬀerent organisations: the ﬁre brigade and the emergency health service. The agents in the ﬁre brigade represent individual ﬁremen, which have the goal of responding to and extinguishing ﬁres. The agents in the emergency health service, the medics, have the goal of rescuing casualties. We consider two diﬀerent organisation structures for each of these organisations: in the hierarchical case, a leader is in charge of evaluating the situation, deciding on a global coordinated plan, and gives orders to all the agents in its organisation; in the network case, agents act based on their own evaluation of the situation and coordinate locally.

36

H. Aldewereld et al.

The environment: Spaces, Objects and Bodies. We model activities within the crisis management scenario with three types of spaces: communication space, organisation space, and ﬁeld space. The communication space supports distant communication between agents (local communication is handled in the ﬁeld space). Each agent holds a communication body in this space, which provides the capability to send and receive messages. This space deﬁnes the acquaintance relationship between agents (i.e., which body can communicate with which other one). For our purpose, leaders can send messages to any other agent whereas active units (ﬁremen and medics) can only send messages to their leader (when they have one). The communication body determines how many messages an agent can send per tick, and the communication space deﬁnes the communication lag (how many ticks are required to deliver a message). Those properties are set as parameters of the model. Two organisation spaces represent respectively the two organisations: ﬁre brigade and emergency health services. Each agent holds a role body in the space of its organisation, and this body provides it with the capability to perceive the role of other agents in its organisation. For the ﬁre brigade, the roles are leader and ﬁreman, and for the emergency health services, the roles are leader and medic. The ﬁeld space represents the geographical area in which emergency situations occur and operational units (ﬁremen and medics) are located. Fires and casualties are represented by objects in that space, and ﬁremen and medics agent hold respectively a ﬁreman body and a medic body. The ﬁeld space speciﬁes a grid topology on which all entities are located. Fireman and medic bodies have the capability to move around in the grid, to perceive in a limited range, and to send messages to other bodies in the perception range. Medic bodies may perceive both ﬁres and casualties but ﬁreman bodies only have the capability to perceive ﬁres. Medic bodies have the capability of rescuing a casualty when its location does not contain a ﬁre, and ﬁreman bodies have the capability to extinguish ﬁres. Three types of ﬁre are distinguished: small, medium and large. A small ﬁre can be extinguished by one ﬁreman, a medium ﬁre requires two ﬁremen and a large ﬁre requires three ﬁremen working at the same time. Each casualty is characterised by a number of life points which decreases over time by one unit per tick when it is in a ﬁre-free location, and by three units per tick when there is a ﬁre close by. Decision-making: Minds. Six types of minds are designed to model respectively the ﬁremen leader (FL), the medics leader (ML), the network ﬁremen (NF), the hierarchical ﬁremen (HF), the network medic (NM) and the hierarchical medic (HM). Both FL and ML (which have no body in the ﬁeld space) listen to information provided by their subordinates and generate plans to be followed by their subordinates. Moreover, ML have to decide when and how to request support from ﬁremen when it is needed. The strategy here depends on the ﬁre brigade organisation: if it is a hierarchical one, ML will send its requests to FL, but if it is a network one, ML will broadcast to all NFs.

Agent-Based Crisis Management

37

HF and HM are quite simple minds since they only inform their respective leader about what they perceive locally1 , and react to orders given by the leader. However, NF and NM have to decide by themselves where to move, and how to coordinate locally with others when they cannot handle the situation alone. Local coordination is achieved through direct communication on the map, and is restricted by the perception range of bodies. NF have to organise themselves to deal with medium and large ﬁres, and NM have to request support from ﬁremen (either through FL or directly with NF). All these minds have been given limited cognitive capabilities which are expressed by the number of messages they can process per tick. If too many messages are received during a tick, messages will be processed during the following ticks. Moreover, in the experiments reported in this paper we have only considered one type of reasoning for each type of mind. We are currently extending the model to include heterogenous minds. 5.2

Experimental Scenarios

The experiments described in this section were designed to study the behaviour of organisational structures for crisis management with respect to eﬃciency (cf. section 2). For the analysis of other structural properties, the scenarios have to be modiﬁed to represent communication and control failures (for the study of robustness) and endow agents with alternative capabilities (to analyse ﬂexibility). For this study of eﬃciency, we have designed four diﬀerent conﬁgurations each involving two organisation structures of ﬁremen and medics: hierarchy (HF and HM , respectively) and network (N F and N M , respectively), resulting in the following cases: Table 1. Organisational configurations Firemen Network Hierarchy Network NF-NM HF-NM Medics Hierarchy NF-HM HF-HM

Each simulation was run with a population of 8 ﬁremen and 4 medics. The minds had a cognitive capability to process 10 messages per tick. Default settings for ﬁeld, communication and organisation spaces are given in table 2. Based on these settings, we developed 12 simulation scenarios, as listed in table 3. We use these scenarios as an initial set to discover whether our hypothesis on what the outcome of the cooperation between diﬀerent organisations will be makes any sense. Thus we are not claiming that this set of scenarios is a complete set, but rather that it is a good starting point to study inter-organisational cooperation. For all these scenarios, the initial location of entities (ﬁremen, medics, ﬁres and casualties) are set randomly. 1

Their local area is defined by their perception range.

38

H. Aldewereld et al. Table 2. Base settings for all scenarios Field Space Map size 20 × 20 Initial casualty life range 50 Communication lag instantaneous Perception range medic bodies 5 Perception range firemen bodies 5 Communication Space Communication lag Send message capability Organisation Space Perception range leader

1 tick 10/tick

absolute

Table 3. Simulation scenarios Scenario 1 2 3 4 5 6 7 8 9 10 11 12

3 3 3 8

Small, Small, Small, Small,

Fires 10 Small 10 Small 25 Small 25 Small 10 Large 10 Large 25 Large 25 Large 3 Medium, 3 Medium, 3 Medium, 8 Medium,

3 3 3 8

Large Large Large Large

Casualties 10 25 10 25 10 25 10 25 10 25 10 25

Each simulation round had a maximal duration of 200 ticks (beyond this limit the run is a failure). Results are interpreted in terms of the following measures: average time to extinguish ﬁre, number of failures in extinguishing ﬁres, and percentage of dead casualties. Each scenario was run 200 times for each of the 4 organisational conﬁgurations described in table 1; the results presented in the following section are an average over these 200 runs.

6

Results

The results obtained from running the simulation scenarios described in the previous section are displayed in ﬁgures 2, 3 and 4, where each colour represents a diﬀerent organisational conﬁguration (cf. table 1). Figure 2 shows the diﬀerences

Agent-Based Crisis Management

39

Fig. 2. Average time to extinguish all the fires

Fig. 3. Number of failures for fire fighters

in time to extinguish the ﬁres for the diﬀerent organisations, ﬁgure 3 refers to the diﬀerences in failures for ﬁremen (when they could not extinguish all the ﬁres) and ﬁgure 4 gives the number of deaths per organisational conﬁguration. We will ﬁrst discuss the eﬃciency of the ﬁremen organisation. With respect to small, homogeneous crises (scenarios 1 to 4), network organisations (for both ﬁremen and medics) seem to be more eﬃcient. This is because network agents react immediately to their perception whereas the hierarchical ones will ﬁrst inform their leader and wait for its instructions. There is an important diﬀerence in number of failures between scenarios 1 and 2, and scenarios 3 and 4. In the latter case, the large number of ﬁres induces a lot of moves for the ﬁremen, which makes them discover new ﬁres. For scenarios 1 and 2, many of the ﬁres remain out of the perception of the ﬁremen, so they are never extinguished. For large, homogenous crises (scenarios 5 to 8) coordination between ﬁremen is required because large ﬁres can only be estinguish by a group of three ﬁremen. We hypothesise that the HF structure is more appropriate to handle such situations because of this need of coordination, but results show (cf. ﬁgure 2) that NF structures are still more eﬃcient for these scenarios. This is an emergent result from the simulation: even if the process of forming a group of 3 agents is more

40

H. Aldewereld et al.

Fig. 4. Percentage of dead casualties

complex and requires more time for networked agents, than in hierarchies, once it is formed, the group of three agents keep together and collaborate easily.2 On the other hand, it should be noticed that if NF are more eﬃcient than HF on scenarios 5 to 8 in terms of time when they succeed, they fail to handle the situation more often (cf. ﬁgure 3). Because they are only guided by their local perception of the ﬁeld they can only coordinate on sight. Situations in which ﬁremen do not ﬁnd others result in failure to handle the crisis. In the case of a hierarchical structure, because the leader has global view of the situation, it will guide ﬁremen to each other and all ﬁres can be handled. Scenarios 9 to 12 represent heterogeneous crises consisting of diﬀerent types of ﬁres simultaneously and lead to quite diﬀerent results. The ﬁremen have to coordinate in order to handle medium and large ﬁres, but this time, because the ﬁres are heterogeneous forming or maintaining a group of three agents is not the best solution. In this situation, HF are more eﬃcient because the leader provides a better allocation of ﬁremen to diﬀerent ﬁres. Concerning the eﬃciency of the medics organisation, ﬁgure 4 shows that NM manage to save much more casualties than HM in all the scenarios with many casualties (i.e. scenarios 2, 4, 6, 8, 10 and 12). Medics do not need to coordinate between themselves as the ﬁremen do, because one medic is enough to rescue a casualty. In that case, local decisions generate shorter paths than global planning, meaning that NF have more time to rescue casualties. On scenarios with few casualties (i.e. scenarios 1, 3, 5, 7, 9 and 11), time is not an issue. Figure 4 shows that HF usually rescue slightly more casualties than HN except in scenario 11. The fact that, due to the global planning, HF tend to travel much more than NF, becomes now an advantage since they have more chance to discover new casualties on their way. Scenario 11 is diﬀerent from the other scenarios with few casualties because ﬁremen need a lot of time to extinguish ﬁres, and so medics have to wait a long time before some locations are secured. Here again the NM rescue more casualties than HM because they are faster.

2

This is a consequence of the use of homogeneous minds: they just take the same decisions.

Agent-Based Crisis Management

41

Concerning the overall crises organisation, that is combining ﬁremen and medics, we can see that in scenarios 1 to 4 and 9 to 12, the combination NF-NM (blue bars) has slightly better results in term of time (cf. ﬁgure 2) than the combination NF-HM (red bars). When HM need support from NF, the medics leader broadcast a request to all the ﬁremen, which tends to lead several ﬁremen to a certain location. This means that ﬁremen are less spread on the ﬁeld leading to reduced eﬃciency. This phenomena also happens of course for the scenarios 5 to 8, but in these cases forming groups is an advantage to deal with large ﬁres. In combinations NM - NF, the requests for support are only performed locally, and so this group concentration does not occur. Finally when NM works with HF, the requests for support are sent to the ﬁre leader who can decide to send the proper number of ﬁremen secure the location for the medics. Finally, for all scenarios, we can notice that NM increase the number of failures for ﬁremen (both NF and HF) in comparison to HM. For the situation involving NF and HM, the medics leader reduces the number of failures of ﬁremen because it broadcasts requests for support to them, and so they get to know about new ﬁres from it. For the situation involving HF and NM , the ﬁre leader receive the requests for support from all the medics, in additional to its usual communication with ﬁremen. For this reason the ﬁre leader tends to be overloaded with messages and takes more time to react to new situations.

7

Conclusions and Future Work

In this paper, we have presented a model and a simulation tool to investigate the impact of organisation structures on crisis management. Our main objective was to compare the eﬃciency of (combinations of) hierarchical and network structures in the context of crisis management. As expected, our experiments conﬁrm that a hierarchy structure is more eﬃcient when agents have to deal with heterogeneous situations that require more than one agent to be solved, and when diﬀerent organisations need to coordinate their eﬀorts. On the other hand, network structures are more eﬃcient when agents are able to handle issues individually because of their faster response time. However, even the simple scenarios described in this paper can lead to unexpected results, as in the case where the hierarchical nature of the medics organisation served to improve eﬃciency of the ﬁremen. In fact, the dynamicity and complexity of a crisis scenario means that it is impossible to compute a priori all consequences of the organisational model. The results of this paper are promising as they show that even fairly simple simulations provide results which by a posteriori step-by-step analysis lead to relevant insights into the situation. This is facilitated by the possibilities to observe visualisations of activities and traces of the simulation runs. The experiments have also revealed an unexpected phenomena, which is that even for environments that contain issues that cannot be solved individually, if

42

H. Aldewereld et al.

those issues are homogeneous a network structure can be eﬃcient because of the emergence of local collaborating groups. In this paper, we have concentrated our eﬀort on evaluating organisation efﬁciency, which is only one of the possible measures to evaluate organisation performance. We are currently extending the crisis management model to enable the analysis of other organisational characteristics, such as ﬂexibility and robustness. Flexibility is related to dynamic environments. The simulation tool presented in this paper already oﬀers the possibility to generate new ﬁres and new casualties during a simulation run, but to understand ﬂexibility we will need to perform a systematic study with scenarios that also include richer cognitive models for the minds and more capabilities for the bodies. To analyse robustness, we have to consider situations for which the integrity of an organisation is threatened, such as communication failures. This requires an extension of the communication space in order to simulate communication problems. We are currently extending the crisis management model to enable the inclusion of more complex environments. Furthermore, we plan to add other measures to the model (e.g. average time or total time to rescue casualties, health of casualties, number of ﬁres that have been extinguished, etc.) that will provide richer analysis tools.

References 1. Dignum, V.: A Model for Organizational Interaction: based on Agents, founded in Logic. PhD thesis, Universiteit Utrecht (2004) 2. Ferber, J., Gutknecht, O., Michel, F.: From agents to organizations: An organizational view of multi-agent systems. In: Giorgini, P., M¨ uller, J.P., Odell, J.J. (eds.) AOSE 2003. LNCS, vol. 2935, pp. 214–230. Springer, Heidelberg (2004) 3. Ferber, J., Stratulat, T., Tranier, J.: Towards an integral approach of organizations: the MASQ approach. In: Dignum, V. (ed.) Multi-agent Systems: Semantics and Dynamics of Organizational Models. IGI (2009) 4. Grossi, D., Dignum, F., Dastani, M., Royakkers, L.: Foundations of organizational structures in multiagent systems. In: AAMAS 2005, pp. 690–697. ACM Press, New York (2005) 5. Grossi, D., Dignum, F., Dignum, V., Dastani, M., Royakkers, L.: Structural aspects of the evaluation of agent organizations. In: COIN@ECAI 2006 (2006) 6. Jenvald, J., Morin, M.: Simulation-supported live training for emergency response in hazardous environments. Simulation & Gaming 35, 363–377 (2004) 7. Kitano, H.: Robocup rescue: a grand challenge for multi-agent systems. In: Proc. ICMAS 2000, pp. 5–12 (2000) 8. Krackhardt, D.: Graph theoretical dimensions of informal organizations. In: Carley, C.M., Prietula, M.J. (eds.) Computational Organization Theory, pp. 89–110. Lawrence Erlbaum Associates, Mahwah (1994) 9. Malone, T., Crowston, K.: The interdisciplinary study of coordination. ACM Computing Surveys, 26(1) (March 1994)

Agent-Based Crisis Management

43

10. Quillinan, T., Brazier, F., Aldewereld, H., Dignum, F., Dignum, V., Penserini, L., Wijngaards, N.: Developing agent-based organizational models for crisis management. In: Proc. AAMAS 2009 Industrial Track (2009), IFAAMAS.org 11. Seeger, M.W., Sellnow, T.L., Ulmer, R.R.: Communication, organization and crisis. Communication Yearbook, 231–275 (1998) 12. Stimson, W.: The Robust Organization: Transforming Your Company Using Adaptive Design. Irwin Professional Publishing (1996) 13. Tranier, J., Dignum, V., Dignum, F.: A multi-agent simulation framework for the study of intermediation. In: Proc. 6th. Conference of European Social Simulation Association (2009)

Male Optimal and Unique Stable Marriages with Partially Ordered Preferences Mirco Gelain1 , Maria Silvia Pini1 , Francesca Rossi1 , K. Brent Venable1 , and Toby Walsh2 1

Universit` a di Padova, Italy {mpini,frossi,kvenable}@math.unipd.it 2 NICTA and UNSW, Sydney, Australia [email protected]

Abstract. The stable marriage problem has a wide variety of practical applications, including matching resident doctors to hospitals, and students to schools. In the classical stable marriage problem, both men and women express a strict order over the members of the other sex. Here we consider a more realistic case, where both men and women can express their preferences via partial orders, i.e., by allowing ties and incomparability. This may be useful, for example, when preferences are elicited via compact preference representations like soft constraint or CP-nets that produce partial orders, as well as when preferences are obtained via multi-criteria reasoning. We study male optimality and uniqueness of stable marriages in this setting. Male optimality gives priority to one gender over the other, while uniqueness means that the solution is optimal, since it is as good as possible for all the participating agents. Uniqueness of solution is also a barrier against manipulation. We give an algorithm to ﬁnd stable marriages that are male optimal. Moreover, we give suﬃcient conditions on the preferences (that are also necessary in some special case), that occur often in real-life scenarios, which guarantee the uniqueness of a stable marriage.

1

Introduction

The stable marriage problem (SM) [9] is a well-known collaboration problem. Given n men and n women, where each expresses a strict ordering over the members of the opposite sex, the problem is to match the men to the women so that there are no two people of opposite sex who would both rather be matched with each other than their current partners. In [6] Gale and Shapley proved that it is always possible to ﬁnd a matching that makes all marriages stable, and provided a quadratic time algorithm which can be used to ﬁnd one of two extreme stable marriages, the so-called male optimal or female optimal solutions. The Gale-Shapley algorithm has been used in many real-life scenarios, such as in matching hospitals to resident doctors, medical students to hospitals [10], sailors to ships, primary school students to secondary schools, as well as in market trading. C. Guttmann, F. Dignum, and M. Georgeﬀ (Eds.): CARE 2009/2010, LNAI 6066, pp. 44–55, 2011. c Springer-Verlag Berlin Heidelberg 2011

Male Optimal and Unique SMs with Partially Ordered Preferences

45

In the classical stable marriage problem, both men and women express a strict order over the members of the other sex. We consider a potentially more realistic case, where men and women express their preferences via partial orders, i.e., given a pair of men (resp., women), the women (resp., the men) can strictly order the elements of the pair, they may say that these elements are in a tie, or that they are incomparable. This is useful in practical applications when a person may not wish (or be able) to choose between alternatives, thus allowing ties in the preference list (or more generally, allowing each preference list to be a partial order) [11]. For example, in the context of centralized matching scheme, some participating hospitals with many applicants have found the task of producing a strictly ordered preference list diﬃcult, and have expressed a desire to use ties [12]. Ties also naturally occur when assigning students to schools, since many students are indistinguishable from the point of view of a given school. Another situation where partial orders are useful is when preferences are elicited with a compact preference representation formalism like soft constraints [1] or CP-nets [2] that give partial orders. Another context where partial orders naturally occur is when preferences are obtained via multi-criteria reasoning. We study male optimality and uniqueness of solution in this more general context. Male optimality can be a useful property since it allows us to give priority to one gender over the other. For example, in matching residents to hospitals in the US, priority is given to the residents. We present an algorithm, based on an extended version of the Gale-Shapely algorithm, to ﬁnd a male optimal solution in stable marriage problems with partially ordered preferences (SMPs). This algorithm is sound but not complete: it may fail to ﬁnd a male optimal solution even when one exists. We conjecture, however, that the incompleteness is rare. We also give a suﬃcient condition on the preference proﬁle that guarantees to ﬁnd a male optimal solution, and we show how to ﬁnd it. Uniqueness is another interesting concept. For instance, it guarantees that the solution is optimal since it is as good as possible for all the participating agents. Uniqueness is also a barrier against manipulation. This is important as all stable marriage procedures can be manipulated. In [5] suﬃcient conditions on the preferences are given, that guarantee uniqueness of stable marriages when only strictly ordered preferences are allowed. Such conditions identify classes of preferences that are broad and of particular interest in many real-life scenarios [4]. In particular, a class of preference orderings that satisfy one of these conditions requires that all the agents of the same sex have identical preferences over the mates of the opposite sex, i.e., there is a common ordering over the mates. Another class of preference orderings that satisfy one of these conditions of uniqueness requires that each agent has a diﬀerent most preferred mate, i.e., there is a subjective ranking over the mates. We show that it is possible to generalize these suﬃcient conditions for uniqueness to SMs with partially ordered preferences, by considering in some cases uniqueness up to indiﬀerence and incomparability. All the proofs have been omitted due to lack of space. They can be found in [8]. A brief overview of some of the theoretical results shown in this paper is contained in [7].

46

2 2.1

M. Gelain et al.

Background Stable Marriage Problems

Deﬁnition 1 (proﬁle). Given n men and n women, a proﬁle is a sequence of 2n strict total orders (i.e., transitive and complete binary relations), n over the men and n over the women. Given a proﬁle, the stable marriage problem (SM) [6] is the problem of ﬁnding a matching between men and women so that there are no two people of opposite sex who would both rather be married to each other than their current partners. If there are no such people, the marriage is said to be stable. Deﬁnition 2 (feasible partner). Given an SM P , a feasible partner for a man m (resp., a woman w) is a woman w (resp., a man m) such that there is a stable marriage for P where m and w are married. The set of the stable marriages for an SM forms a lattice w.r.t. the men’s (or women’s) preferences. This is a graph where vertices correspond bijectively to the stable marriages and a marriage is above another if every man (resp., every woman) is married with a woman (resp., man) is at least as happy with the ﬁrst marriage as with the second. The top of this lattice is the stable marriage, called male optimal (resp., female optimal), where men (resp., women) are mostly satisﬁed. Conversely, the bottom is the stable marriage where men’s (resp., women’s) preferences are least satisﬁed [9]. Deﬁnition 3 (male (resp., female) optimal marriage). Given an SM P , a marriage is male (resp., female) optimal iﬀ every man (resp., woman) is paired with his (resp., her) highest ranked feasible partner in P . 2.2

Gale-Shapley Algorithm

The Gale-Shapley (GS) algorithm [6] is a well-known algorithm to solve the SM problem. At the start of the algorithm, each person is free and becomes engaged during the execution of the algorithm. Once a woman is engaged, she never becomes free again (although to whom she is engaged may change), but men can alternate between being free and being engaged. The following step is iterated until all men are engaged: choose a free man m, and let m propose to the most preferred woman w on his preference list, such that w has not already rejected m. If w is free, then w and m become engaged. If w is engaged to man m’, then she rejects the man (m or m’) that she least prefers, and becomes, or remains, engaged to the other man. The rejected man becomes, or remains, free. When all men are engaged, the engaged pairs are a male optimal stable marriage. This algorithm needs a number of steps that is quadratic in n (that is, the number of men), and it guarantees that, if the number of men and women coincide, and all participants express a strict order over all the members of the other group, everyone gets married, and the returned marriage is stable. Since the input includes the proﬁles, the algorithm is linear in the size of the input.

Male Optimal and Unique SMs with Partially Ordered Preferences

47

Example 1. Assume n = 3. Let W = {w1 , w2 , w3 } and M = {m1 , m2 , m3 } be respectively the set of women and men. The following sequence of strict total orders deﬁnes a proﬁle: {m1 : w1 > w2 > w3 (i.e., man m1 prefers woman w1 to w2 to w3 ); m2 : w2 > w1 > w3 ; m3 : w3 > w2 > w1 } {w1 : m1 > m2 > m3 ; w2 : m3 > m1 > m2 ; w3 : m2 > m1 > m3 }. For this proﬁle, the Gale-Shapley algorithm returns the male optimal solution {(m1 , w1 ), (m2 , w2 ), (m3 , w3 )}. On the other hand, the female optimal solution is {(w1 , m1 ), (w2 , m3 ), (w3 , m2 )}. 2 The Extended Gale-Shapely algorithm [9] is the GS algorithm [6] where, whenever the proposal of a man m to a woman w is accepted, in w’s preference list all men less desirable than m are deleted, and w is deleted from the preference lists of all such men. This means that, every time that a woman receives a proposal from a man, she accepts since only most preferred men can propose to her.

3

Stable Marriage Problems with Partial Orders

We assume now that men and women express their preferences via partial orders. The notions given in Section 2 can be generalized as follows. Deﬁnition 4 (partially ordered proﬁle). Given n men and n women, a proﬁle is a sequence of 2n partial orders (i.e., reﬂexive, antisymmetric and transitive binary relations), n over the men and n over the women. Deﬁnition 5 (SMP). A stable marriage problem with partial orders (SMP) is just a SM where men’s preferences and women’s preference are partially ordered. Deﬁnition 6 (linearization of an SMP). A linearization of an SMP is an SM that is obtained by giving a strict ordering to all the pairs that are not strictly ordered such that the resulting ordering is transitive. Deﬁnition 7 (weakly stable marriage in SMP). A marriage in an SMP is weakly stable if there is no pair (x, y) such that each one strictly prefers the other to his/her current partner. Deﬁnition 8 (feasible partner in SMP). Given an SMP P , a feasible partner for a man m (resp., woman w) is a woman w (resp., man m) such that there is a weakly stable marriage for P where m and w are married. A weakly stable marriage is male optimal if there is no man that can get a strictly better partner in some other weakly stable marriage. Deﬁnition 9 (male optimal weakly stable marriage). Given an SMP P , a weakly stable marriage of P is male optimal iﬀ there is no man that prefers to be married with another feasible partner of P . In SMs there is always exactly one male optimal stable marriage. In SMPs, however, we can have zero or more male optimal weakly stable marriages. Moreover, given an SMP P , all the stable marriages of the linearizations of P are weakly stable marriages. However, not all these marriages are male optimal.

48

M. Gelain et al.

Example 2. In a setting with 2 men and 2 women, consider the proﬁle P : {m1 : w1 w2 ( means incomparable); m2 : w2 > w1 ; } {w1 : m1 m2 ; w2 : m1 m2 ; }. Then consider the following linearization of P , say Q: {m1 : w2 > w1 ; m2 : w2 > w1 ; } {w1 : m2 > m1 ; w2 : m1 > m2 ; }. If we apply the extended GS algorithm to Q, we obtain the weakly stable marriage μ1 where m1 marries w2 and m2 marries w1 . However, w1 is not the most preferred woman for m2 amongst all weakly stable marriages. In fact, if we consider the linearization Q , obtained from Q, by changing m1 ’s preferences as follows: m1 : w1 > w2 , and if we apply the extended GS algorithm, we obtain the weakly stable marriage μ2 , where m1 is married with w1 and m2 is married with w2 , i.e., m2 is married with a woman that m2 prefers more than w1 . Notice that μ2 is male optimal, while μ1 is not. Also, μ1 and μ2 are the only weakly stable marriages for this example. 2

4

Finding Male Optimal Weakly Stable Marriages

We now present an algorithm, called MaleWeaklyStable, that takes as input an SMP P and, either returns a male optimal weakly stable marriage for P , or the string ‘I don’t know’. This algorithm is sound but not complete: if the algorithm returns a marriage, then it is weakly stable and male optimal; however, it may fail to return a male optimal marriage even if there is one. We assume that the women express strict total orders over the men. If they don’t, we simply pick any linearization. The algorithm exploits the extended GS algorithm [9], and at every step orders the free men by increasing number of their current top choices (i.e., the alternatives that are undominated). List L contains the current ordered sequence of free men. More precisely, our algorithm works as follows. It takes in input an SMP P , and it computes the list L of free men. At the beginning all the men are unmarried, and thus L contains them all. Then, we continue to check the following cases on the man m which is the ﬁrst element of L, until they do not occur any longer: – If the set of top choices of m contains exactly one unmarried woman, say w, m proposes to w and, since we are using the extended GS algorithm, the proposal is accepted. Then, all men that are strictly worse than m in w’s preferences are removed from w’s preference list, and w is removed from the preference lists of these men. Then, m is removed from L and L is ordered again, since the top choices of some men may now be smaller. – If m has a single top choice, say w, that is already married, m proposes to w, w accepts the proposal, and she breaks the engagement with her current partner, say m . Then, m is removed from L, m becomes free and is put back in L, and L is ordered again. When we exit from this cycle, we check if L is empty or not:

Male Optimal and Unique SMs with Partially Ordered Preferences

49

– if L is empty, the algorithm returns the current marriage. Notice that the current marriage, say (mi , wi ), for i = 1, . . . , n, is weakly stable, since it is the solution of a linearization of P where, for every mi with ties or incomparability in current set of top choices, we have set wi strictly better than all the other women in the top choice. Also, the returned marriage is male optimal since we have applied the extended GS algorithm. – If L is not empty, it means that the next free man in L has several current top choices and more than one is unmarried. • If there is a way to assign to the men currently in L diﬀerent unmarried women from their current top choices then these men make these proposals, that are certainly accepted by the women, since every woman receives a proposal from a diﬀerent man. Therefore, we add to the current marriage these new pairs and we return the resulting marriage. Such a marriage is weakly stable and male optimal by construction. • If it is not possible to make the above assignment, the algorithm removes unfeasible women from the current top choices of the men until it is possible to make the assignment or until all unfeasible women have been removed. More precisely, if there is a set S of men in L with the same Top T and the cardinality of T is smaller than the cardinality of S, we check if there is a man m∗ such that, for every w ∈ T , m∗ is worse than mi for every mi in S − m∗ . If this is the case, for every woman w in T , we remove w from the preferences of m∗ in p and we apply again the algorithm MaleWeaklyStable to the proﬁle obtained so far. This could make now possible to make the assignment. If so, the algorithm adds to the current marriage these new pairs and returns the resulting marriage; otherwise, it performs the same reasoning for another pair of men that have some woman in common in their current top choices until all such pairs of men have been considered and no marriage has been returned. At this point the algorithm stops returning the string ‘I don’t know’. Example 3. Consider the proﬁle {m1 : w1 w2 > w3 ; m2 : w1 w2 > w3 ; m3 : w1 w2 > w3 ; } {w1 : m1 > m2 > m3 ; w2 : m1 > m2 > m3 ; w3 : m1 > m2 > m3 ; }. The algorithm ﬁrst computes the ordered list L = [m1 , m2 , m3 ]. The elements of L are men with more than one top choice and all these top choices are unmarried, but there is no way to assign them with diﬀerent women from their top choices, since they are three men and the union of their top choices contains only two women. However, in every linearization, m3 will not be matched with w1 or w2 , due to w1 and w2 ’s preferences. In fact, m1 and m2 will choose between {w1 , w2 }, while m3 will always propose to his next best choice, i.e., w3 . Hence, the considered proﬁle is one of the proﬁles where only two of the three men with multiple top choices are feasible with w1 and w2 , i.e. m1 and m2 , and there is a way to assign to these men diﬀerent unmarried women in their top choices. In such a case there are two male optimal weakly stable solutions, i.e., {(m1 , w1 )(m2 , w2 )(m3 , w3 )} and {(m1 , w2 )(m2 , w1 )(m3 , w3 )}. Our algorithm returns the ﬁrst one. 2

50

M. Gelain et al.

Algorithm 1. MaleWeaklyStable Input: p: a proﬁle; Output: µ: a weakly stable marriage or the string ‘I don’t know’; µ ← ∅; L ← list of the men of p; L ← ComputeOrderedList(L); while Top((ﬁrst(L)) contains exactly one unmarried woman) or (ﬁrst(L) has a single top choice already married) do m ← ﬁrst(L); if Top(m) contains exactly one unmarried woman then w ← UnmarriedTop(m); Add the pair (m, w) to µ; foreach strict successor m∗ of m on w’s preferences do delete m∗ from w’s preferences and w from m’s preferences ; L ← L \ {m}; L ← ComputeOrderedList(L); if m has a single top choice already married then w ← Top(m); m ← µ(w); Remove the pair (m , w) from µ; Add the pair (m, w) to µ; foreach strict successor m∗ of m on w’s preferences do delete m∗ from w’s preferences and w from m’s preferences; L ← L ∪ {m } \ {m}; L ← ComputeOrderedList(L); if (L = ∅) or (AllDiﬀUnmarried(L)=true) then Add to µ AllDiﬀUnmarriedMatching(L); return µ else if there is a set S of men in L with the same Top T and |T | < |S| then if there is a man m∗ ∈ S s.t., ∀w ∈ T , m∗ ≺w mi , ∀mi ∈ S − m∗ then ∀w ∈ T , remove w from the preferences of m∗ in p; MaleWeaklyStable(p);

Example 4. Consider the proﬁle obtained from the proﬁle shown in Example 3 by changing the preferences of w1 as follows: m1 > m3 > m2 . We now show that there is no male optimal solution. It is easy to see that in any weakly stable marriage m1 is married with w1 or w2 . In the weakly stable marriage where m1 is married with w1 , m2 must be married with w2 and m3 must be married with w3 , while in the weakly stable marriage where m1 is married with w2 , m2 must be married with w3 and m3 must be married with w1 . Therefore, in any weakly stable marriage, exactly one of these conditions holds: either m2 prefers to be married with w2 , or m3 prefers to be married with w2 . Therefore, there is no male optimal solution. Our algorithm works as follows. Since AllDiﬀUnmarried(L)=false and since we cannot remove any unfeasible woman

Male Optimal and Unique SMs with Partially Ordered Preferences

51

from the top choices of m1 , m2 , and m3 , the algorithm returns the string ‘I don’t know’. 2 5

The MaleWeaklyStable algorithm has a time complexity which is O(n 2 ). In fact, the ﬁrst part has the same complexity of the extended GS algorithm, which is O(n2 ). The second part requires performing an all-diﬀerent check between the current set of free men and the union of their top choices. Since there are at most n free men and n top choices for each man, we can build a bipartite graph where nodes are men and women, and each arc connects a man with one of his unmarried top choices. Performing the all-diﬀerent check means ﬁnding a subset of the arcs which√forms a matching in this graph and involves all men. This can be done in O(m n) where m is the number of edges, which is O(n2 ). The MaleWeaklyStable Algorithm is sound, but not complete, i.e., if it returns a marriage, then such a marriage is male optimal and weakly stable, but if it returns the string ’I don’t know’, we don’t know if there is a weakly stable marriage that is male optimal. A case where our algorithm returns the string ’I don’t know’ is when L is not empty and there is a free man with more than one top choice and all his top choices are already married. We conjecture that in this case there is a male optimal weakly stable marriage a few times, since it seems there are some very speciﬁc circumstances for our algorithm to mot return a male optimal weakly stable marriage (i.e., it has to pass through all the conditions we test) when one exists. As we noticed above, there are SMPs with no male optimal weakly stable marriages. We now want to identify a class of SMPs where it is always possible to ﬁnd a linearization which has a male optimal stable marriage. Deﬁnition 10 (male-alldiﬀerence property). An SMP P satisﬁes the malealldiﬀerence property iﬀ men’s preferences satisfy the following conditions: – all the men with a single top choice have top choices that are diﬀerent; – it is possible to assign to all men with multiple top choices an alternative in their top choices that is diﬀerent from the one of all the other men of P . Theorem 1. If an SMP is male-alldiﬀerent, then there is a weakly stable marriage that is male optimal and we can ﬁnd it in polynomial time. The MaleWeaklyStable Algorithm exploits this same suﬃcient condition, plus some other suﬃcient condition. Notice that if an SMP satisﬁes the male-alldiﬀerence property, then, not only is there at least one weakly stable marriage that is male optimal, but there is an unique stable marriage up to ties and incomparability.

5

On the Uniqueness of Weakly Stable Marriage in SMPs

For strict total orders, [5] gives suﬃcient conditions on preference for the uniqueness of the stable marriage. We now extend these results to partial orders. Notice that, if there is an unique stable marriage, then it is clearly male optimal. A class

52

M. Gelain et al.

of preference proﬁles in [5] giving an unique stable marriage, when the preferences are strict total orders, is deﬁned as follows. The set of the men and the set of the women are ordered sets, the preferences require that no man or woman prefers the mate of the opposite sex with the same rank order below his/her own order. Given such a preference ordering, by a recursive argument starting at the highest ranked mates, any other stable marriage would be blocked by the identity marriage, i.e., the marriage in which we match mates of the same rank. Theorem 2. [5] Consider two ordered sets M = (mi ) and W = (wi ). If the proﬁle satisﬁes the following conditions: ∀wi ∈ W : mi >wi mj , ∀j > i ∀mi ∈ M : wi >mi wj , ∀j > i

(1) (2)

then there is a unique stable marriage μ∗ (wi ) = mi , ∀i ∈ {1, 2, . . . , N2 }. Notice that the condition above is also necessary when the economies are small, i.e., N = 4 and N = 6. There are two particular classes of preference proﬁles that generate a unique stable marriage, and that are commonly assumed in economic applications [5]. The ﬁrst assumes that all the women have identical preferences over the men, and that all the men have identical preferences over the women. In such a case there is a common (objective) ranking over the other sex. Deﬁnition 11 (vertical heterogeneity). [5] Consider two ordered sets M = (mi ) and W = (wi ). A proﬁle satisﬁes the vertical heterogeneity property iﬀ it satisﬁes the following conditions: – ∀wi ∈ W : mk >wi mj , ∀k < j – ∀mi ∈ M : wk >mi wj , ∀k < j Example 5. An example of a proﬁle that satisﬁes vertical heterogeneity for N = 6 is the following. {m1 : w1 > w2 > w3 ; m2 : w1 > w2 > w3 ; m3 : w1 > w2 > w3 ; } {w1 : m2 > m3 > m1 ; w2 : m2 > m3 > m1 ; w3 : m2 > m3 > m1 .} 2 Corollary 1. [5] Consider two ordered sets M = (mi ) and W = (wi ) and a proﬁle P . If P satisﬁes the vertical heterogeneity property, then there is a unique stable marriage μ∗ (wi ) = mi . When agents have diﬀerent preferences over the other sex, but each agent has a diﬀerent most preferred mate and in addition is the most preferred by the mate, then the preference proﬁle satisﬁes horizontal heterogeneity. In this situation there is a subjective ranking over the other sex. Deﬁnition 12 (horizontal heterogeneity). [5] Consider two ordered sets M = (mi ) and W = (wi ). A proﬁle satisﬁes the horizontal heterogeneity property iﬀ it satisﬁes the following conditions:

Male Optimal and Unique SMs with Partially Ordered Preferences

53

– ∀wi ∈ W : mi >wi mj , ∀j – ∀mi ∈ M : wi >mi wj , ∀j Example 6. The following proﬁle over 3 men and 3 women satisﬁes horizontal heterogeneity. {m1 : w1 > . . . ; m2 : w2 > . . . ; m3 : w3 > . . . } {w1 : m1 > . . . ; w2 : m2 > . . . ; w3 : m3 > . . . .} 2 Corollary 2. [5] Consider two ordered sets M = (mi ) and W = (wi ) and a proﬁle P . If P satisﬁes the horizontal heterogeneity property, then there is a unique stable marriage μ∗ (wi ) = mi . We now check if the results given above for strictly ordered preferences can be generalized to the case of partially ordered preferences. Theorem 2 holds also when the men’s preferences and/or women’s preferences are partially ordered. Theorem 3. In SMPs, if there is an ordering of men and women such that the preference proﬁle satisﬁes the conditions described in Theorem 2, then there is a unique weakly stable marriage μ(wi ) = mi , ∀i ∈ {1, 2, . . . , n}. Notice that the condition above is also necessary when the economies are small. For example, this holds when N = 6 (that is, three men and three women). We now check if the vertical heterogeneity result (Corollary 1) holds also when the preferences are partially ordered. We recall that vertical heterogeneity assumes that all the agents of the same sex have the same strict preference ordering over the mates of the opposite sex. It is possible to see that, even if there is only one incomparable element in the ordering given by the men (or the women), then vertical heterogeneity does not hold and there may be more than one weakly stable marriage, as shown in the following example. Example 7. Consider the following proﬁle: {m1 : w1 > w2 w3 ; m2 : w1 > w2 w3 ; m3 : w1 > w2 w3 ; } {w1 : m1 > m2 > m3 ; w2 : m1 > m2 > m3 ; w3 : m1 > m2 > m3 }. In this proﬁle all the agents of the same sex have the same preference ordering over the mates of the opposite sex, however, there are two weakly stable marriages, i.e., μ1 = {(m1 , w1 ), (m2 , w2 ), (m3 , w3 )} and μ2 = {(m1 , w1 ), (m2 , w3 ), (m3 , w2 )}. Notice however that these two weakly stable marriages diﬀer only for incomparable or tied partners. 2 It is possible to show that if all the agents of the same sex have the same preference ordering over the mates of the opposite sex and there is at least one incomparable or tied pair, then there is a unique weakly stable marriage up to ties and incomparability. Let us consider now Corollary 2 regarding the horizontal heterogeneity property. From Theorem 3, it follows immediately that Corollary 2 holds also when partially ordered preferences are allowed. Corollary 3. In SMPs, if there is an ordering of men and women such that the preference proﬁle satisﬁes horizontal heterogeneity, there is a unique weakly stable marriage μ(wi ) = mi , ∀i ∈ {1, 2, . . . , n}.

54

M. Gelain et al.

For partially ordered preferences, we can also guarantee uniqueness of weakly stable marriages by relaxing the horizontal heterogeneity property as follows. Theorem 4. In an SMP, let us denote with mk is the ﬁrst man with more than one top choice, if he exists. If there is an ordering of men and women in increasing number of their top choices such that the preference proﬁle satisﬁes the following conditions: – ∀mi ∈ M with mi < mk , wi >mi wj , ∀j; – ∀mi ∈ M with mi ≥ mk , • (wi >mi (or mi ) wj ), ∀j < i, and • (wi >mi wj ), ∀j > i; – ∀wi ∈ W , with wi < wk , mi >wi mj , ∀j; – ∀wi ∈ W , with wi ≥ wk , • (mi >wi (or wi ) mj ), ∀j < i, and • (mi >wi mj ), ∀j > i, there is a unique weakly stable marriage μ(wi ) = mi , ∀i ∈ {1, 2, . . . , n}. In words, the conditions above require that every man mi (resp., woman wi ) with a single alternative, i.e., wi (resp., mi ) has as unique top choice wi (resp., mi ), and every mi (resp., wi ) with more than one top choice has exactly one alternative that must be chosen in every weakly stable marriage, that is, wi (resp., mi ).

6

Related Work

In this paper, as in [11,12], we permit non-strictly ordered preferences (i.e., preferences may contain ties and incomparable pairs) and we focus on weakly stable marriages. However, while in [11,12], an algorithm is given that ﬁnds a weakly stable marriage by solving a speciﬁc linearization obtained by breaking arbitrarily the ties, we present an algorithm that looks for weakly stable marriages that are male optimal, i.e., we look for those linearizations that favor one gender over the other one. Moreover, since there is no guarantee that a male optimal weakly stable marriage exists, we give a suﬃcient condition on the preference proﬁle that guarantees to ﬁnd a weakly stable marriage that is male optimal, and we show how to ﬁnd such a marriage. Other work focuses on providing suﬃcient conditions when a certain property is not assured for all marriages. For example, in [3] a suﬃcient condition is given for the existence of a stable roommate marriage when we have preferences with ties.

7

Conclusions

We have given an algorithm to ﬁnd male optimal weakly stable solutions when the men’s preferences are partially ordered. The algorithm is sound but not

Male Optimal and Unique SMs with Partially Ordered Preferences

55

complete. We conjecture, however, that incompleteness is rare since very speciﬁc circumstances are required for our algorithm not to return a male optimal weakly stable marriage when one exists. We have then provided a suﬃcient condition, which is polynomial to check, for the existence of male optimal weakly stable marriages. We have also analyzed the issue of uniqueness of weakly stable marriages, providing suﬃcient conditions, which are likely to occur in real life problems, that are also necessary in special cases.

References 1. Bistarelli, S., Montanari, U., Rossi, F.: Semiring-based constraint solving and optimization. JACM 44(2), 201–236 (1997) 2. Boutilier, C., Brafman, R.I., Domshlak, C., Hoos, H.H., Poole, D.: CP-nets: A Tool for Representing and Reasoning with Conditional Ceteris Paribus Preference Statements. J. Artif. Intell. Res (JAIR) 21, 135–191 (2004) 3. Chung, K.: On the existence of stable roommate matching. Games and Economic Behavior 33, 206–230 (2000) 4. Cole, H., Mailath, G., Postewaite, A.: Social norms, savings behavior and growth. Journal of Political Economy 100, 1092–1125 (1992) 5. Eeckhout, J.: On the uniqueness of stable marriage matchings. Economic Letters 69, 1–8 (2000) 6. Gale, D., Shapley, L.S.: College Admissions and the Stability of Marriage. Amer. Math. Monthly 69, 9–14 (1962) 7. Gelain, M., Pini, M.S., Rossi, F., Venable, K.B., Walsh, T.: Male optimality and uniqueness in stable marriage problems with partial orders (Extended Abstract). In: Proc. AAMAS 2010, vol. 1, pp. 1387–1388 (2010) 8. Gelain, M., Pini, M.S., Rossi, F., Venable, K.B., Walsh, T.: Male optimality and uniqueness in stable marriage problems with partial orders, http://www.math.unipd.it/~ mpini/ 9. Gusﬁeld, D., Irving, R.W.: The Stable Marriage Problem: Structure and Algorithms. MIT Press, Boston (1989) 10. Irving, R.W.: Matching medical students to pairs of hospitals: A new variation on a well-known theme. In: Bilardi, G., Pietracaprina, A., Italiano, G.F., Pucci, G. (eds.) ESA 1998. LNCS, vol. 1461, pp. 381–392. Springer, Heidelberg (1998) 11. Irving, R.W.: Stable marriage and indiﬀerence. Discrete Applied Mathematics 48, 261–272 (1994) 12. Irving, R.W., Manlove, D.F., Scott, S.: The hospitals/Residents problem with ties. In: Halld´ orsson, M.M. (ed.) SWAT 2000. LNCS, vol. 1851, pp. 259–271. Springer, Heidelberg (2000)

Agent-Based Simulation of Joint Fire Support Teams – Collaboration in Network-Centric Warfare Scenarios Christian Gerstner1 , Robert Siegfried1 , and Nane Kratzke2 1

Universität der Bundeswehr München Werner-Heisenberg-Weg 39 85577 Neubiberg, Germany {christian.gerstner,robert.siegfried}@unibw.de http://www.unibw.de/robert.siegfried/ 2 Lübeck University of Applied Sciences Mönkhofer Weg 239 23562 Lübeck, Germany [email protected]

Abstract. We present an agent-based model to compare diﬀerent coordination patterns in joint ﬁre support (JFS) scenarios. Modern warﬁghting approaches depend heavily on a separation of concerns (like reconnaissance, coordination and engagement) and therefore impose high requirements on the coordination of all involved parties. Following the General Reference Model for Agent-Based Modeling and Simulation (GRAMS), we present an agent-based model of this problem domain. Our simulations indicate that decentralized JFS coordination leads to smaller average times from identiﬁcation of a target to ﬁnal engagement, while at the same time requiring extensive resources. Central coordination is more eﬀective in terms of engaged units and reduced resource requirements, but tends to take more time.

1

Introduction

Joint Fire Support (JFS) is a military term for providing lethal engagements in an ad-hoc manner in highly dynamic warﬁghting scenarios. JFS requests are typically launched in tactical situations by military ground units confronted with non-predictable threats which can not be engaged by organic engagement means of these ground units. JFS is realized by military engagement, reconnaissance and on scene coordination means provided by army, air force and navy units. These functional nodes are assigned and combined ad-hoc. A typical JFS request shall be executed within few minutes including the following tasks: determine adequate reconnaissance and engagement assets, check rules of engagement, task and reposition assets, collect and provide adequate target data, conduct and assess the (lethal) engagement. A lot of military command nodes on diﬀerent command levels may be involved in processing JFS requests properly and in accordance with the given rules of engagement. As JFS requests C. Guttmann, F. Dignum, and M. Georgeﬀ (Eds.): CARE 2009/2010, LNAI 6066, pp. 56–67, 2011. c Springer-Verlag Berlin Heidelberg 2011

Agent-Based Simulation of Joint Fire Support Teams

57

can not be exactly forecasted in time, target location or class nearly everything has to be coordinated ad-hoc. A variety of national coordination patterns has evolved in western countries including israeli armed forces to handle this JFS problem domain. A coordination pattern is the command and control communication structure of command-, engagement- and reconnaissance-nodes in order to collectively provide a JFS service. None of the existing coordination patterns seems to be adequate in every situation. Each one has advantages as well as disadvantages. An optimal JFS coordination pattern has to consider the extent and landscape of the operational area of own forces, the amount of expected JFS requests, deﬁned areas of responsibilities of command nodes, the amount of engagement, reconnaissance and on scene coordination means capable to process JFS tasks as well as the applicable chain of command and the political rules of engagement. JFS may be performed by international forces (a so-called joint/combined setting) introducing additional decision making processes and aspects. The coordination patterns reach from strictly hierarchical to completely decentralized (in vision) as well as hybrid coordination patterns. Especially decentralized patterns are reﬂecting modern warﬁghting approaches like network centric warfare visions [1], power to the edge approaches [2], information age combat models [3,4] and resulting emergent behaviour models [6] which make agent-based simulation an obvious analysis approach. This paper presents the inital version of an agent-based model for analysing and comparing JFS scenarios as well as JFS coordination patterns. We present our JFS model (which is inspired by the Information Age Combat Model proposed by Cares et al. [3,4]) in section 2 and ﬁrst simulation results in section 3. Finally, we close with a conclusion and an outlook on our ongoing research in section 4.

2

Modeling Joint Fire Support Teams

The development of the JFS model [5] is inspired by the domain speciﬁc Information Age Combat Model [3,4] and follows closely the General Reference Model for Agent-Based Modeling and Simulation (GRAMS) [9,10]. Therefore, the description of the model presented here is structured according to the GRAMS reference model. To get acquainted to the problem domain, the model is restricted in many issues. Once the basics of the problem domain are well-understood, the restrictions may be relaxed and a larger parameter space will be covered (cp. [8]). Currently, three types of agents are distinguished: the Reconnaissance-Agent, the Coordinator-Agent and the Engagement-Agent. Each target is modelled as an object, which means that it is not able to plan or react to his environment like an agent. An action of a single agent is triggered by an event, which on the other hand is triggered by the action of another agent, the environment or the agent himself.

58

C. Gerstner, R. Siegfried, and N. Kratzke

2.1

Aims of the Model

The intention of the model is to evaluate diﬀerent coordination patterns. There are two possible directions: one is to maximize centralization and the other is to minimize it. Both coordination patterns have their own strengths and weaknesses. The idea is to ﬁnd the optimal pattern by analyzing diﬀerent parameters. These parameters may be the time needed by a coordinator for ﬁnding and assigning a required unit or the overall time needed until a speciﬁc target is fought. 2.2

Macro-Level: Time and Environment

Time is modeled as discrete time steps. The duration of each time step is not speciﬁed any further. This abstraction seems feasible as the comparison of the coordination patterns is purely qualitatively at the moment. Nevertheless, future calibration and validation activities will adress this issue. 1

2

3

4

5

6

7

8

9

10 11 12 13 14 15 16 17 18

1 E

2 3

T

R

4 5 6

E

R

7 8 9

R E

10 11 12

R R

13 Fig. 1. Schematics of the model environment (T = Target, R = Reconnaissance-Agent, E = Engagement-Agent)

As indicated in ﬁgure 1, the environment is modeled as a ﬂat 2-dimensional matrix. Six diﬀerent types of landscapes are distinguished, namely forest, mountain, plain, city, sea and inlandwater. The landscape determines the movement

Agent-Based Simulation of Joint Fire Support Teams

59

possibilities of diﬀerent unit types (army, air force, navy) as well as speciﬁc limitations (e. g. reduced speed of ground units in mountain areas). Currently the environment is assumed to be static, i. e. the model does not include any changes within the environment. According to the objective of analysing coordination patterns, this constraint is neglible at the moment. Nevertheless, future version of the model should account for a dynamic environment as this type of uncertainty is of course observed in the real world and regularly leads to reconsideration of earlier decision. 2.3

Micro-Level: Objects and Agents

At the micro-level, three diﬀerent types of agents are distinguished: – Reconnaissance-Agents – Coordinator-Agents – Engagement-Agents Furthermore, one object type is deﬁned at the moment, the Target. The Reconnaissance-Agents patrol along their routes, which are deﬁned by explicit waypoints (indicated by the colored paths in ﬁgure 1). As soon as a target is located, they stop their movement and report this target to their superior Coordinator-Agent. The Coordinator-Agents represent the command and control structure. These Coordinator-Agents control all their subordinates and evaluate their suitability of engaging this target. According to a pre-deﬁned prioritization method they choose one subordinate Engagement-Agent and order him to engage the target. If the kind of weapon ﬁre provided by the Engagement-Agent makes marking necessary, the reporting Reconnaissance-Agent or another available Engagement-Agent is ordered to serve this marking at the same timepoint as the weapon ﬁre from the executing combat unit. The result is controlled by the Reconnaissance-Agent and reported to the commanding CoordinatorAgent. The number of Coordinator-Agents as well as the actual process of coordination among them is inﬂuenced heavily by the number of coordinators and their hierarchy. For example, if there are no Coordinator-Agents, the whole process is coordinated by the Engagement-Agents themselves in a decentralized peer-topeer manner (cp. Information Age Combat Model [3,4]). Target. At the current state, targets are represented by immobile objects. Target-Objects appear according to a pre-deﬁned rate, and disappear according to some distribution (thereby imitating moving objects which leave the speciﬁed area of operations). For representing diﬀerent types of targets, each target is classiﬁed into one of three threat levels (low, medium and high). The threat level serves as selective criterion: a higher treat level requires a more speciﬁc unit for engaging this target.

60

C. Gerstner, R. Siegfried, and N. Kratzke

Visual perception

[has no superior Coordinator]

[has superior Coordinator]

Move

[Other unit ordered to marking]

Analyse effects of fire Send to Broadcastchannel

Submit request [No marking necessary]

[Marking necessary]

Acceptance of order

Mark the target

Key: Control-Flow Sensor

Effector

Data-Flow

Fig. 2. Behavior of the Reconnaissance-Agent (depicted as sensor-eﬀector-chains)

Reconnaissance. Reconnaissance units patrol on deﬁned waypoints and keep a given area under surveillance. They do not have the ability to ﬁght, but to analyse and mark a target if they detect one. A target may only be detected if it is within the range of sight of the reconnaissance unit (indicated in Fig. 1 by the surrounding colored areas). If the target is within the range of sight, the actual detection is determined stochastically, i. e. a target may be detected with a given probability p. If this event of detection happens, the reconnaissance unit initiates a request for ﬁre support and sends it to the command and control unit responsible for this part of the environment (see ﬁgure 2). Later on, after a unit is assigned to the target, the reconnaissance unit is capable of marking the target, if the combat unit needs marking (e. g. a ﬁghter bomber using guided bombs). After the combat unit has engaged the target, the reconnaissance unit analyses the impact on the target. If the target is succesfully destroyed, the reconnaissance unit continues with his patrol. Command and Control. The command and control units are represented by Coordinator-Agents which are not located at any speciﬁc position within the given environment. Each Coordinator-Agent controls a speciﬁc rectangular area of the environment. All targets which are detected within this area are reported to the responsible Command and Control unit. The Command and Control units have an amount of Engagement-Agents subjected to them. If a request for ﬁre support occurs, they evaluate the situation and check all subordinates if the request can be fulﬁlled. Each subjected Engagement-Agent is listed in a matrix

Agent-Based Simulation of Joint Fire Support Teams

61

6DYH5HTXHVW 3HUFHLYH5HTXHVWV

5HDG%URDGFDVWFKDQQHO

8SGDWH0DWUL[RI$YDLODELOLW\

*HW6WDWXV 8SGDWH0DWUL[RI$YDLODELOLW\

*HW6WDWXV 9DOXHRI$SSURSULDWHQHVV

6XEPLWUHTXHVW

6HQGWR%URDGFDVWFKDQQHO

6XEPLWRUGHU

.H\ &RQWURO)ORZ

6HQVRU

(IIHFWRU

'DWD)ORZ

Fig. 3. Behavior of the Coordinator-Agent (depicted as sensor-eﬀector-chains)

together with a value representing the suitability for the reported threat level of the target. This way the most appropriate Engagement-Agent is identiﬁed and ordered to engage the target. The Command and Control unit may come to the conclusion that no one of his subordinates can eﬀectively engage the target. In this case, the CoordinatorAgent ﬁrst tries to pass the request along to his superior Command and Control unit (if he has one) or to a neighbouring commander. If all this is not possible or this agent is on top of the hierarchy, he puts the request in a queue and checks the feasibility again later. Figure 3 illustrates this behavior of the CoordinatorAgent. One of the main objectives of this model is to analyse diﬀerent coordination pattern. In a ﬁrst step, we chose two very diﬀerent patterns: – Hierarchichal coordination Hierarchichal coordination reﬂects traditional military chains of command. Information is passed from the bottom to the top of the hierarchy, while decisions and orders are passed down from the top to the bottom. To reﬂect short decision processes, which are typical for JFS, only three levels of hierarchy are modeled. – Decentralized coordination In contrast to hierarchical coordination, we chose to model a coordination pattern without explicit coordinators. In other words, all agents are on the same level and have to make decisions in a peer-to-peer manner. Decentralized coordination does not involve any designated coordinators, instead all

62

C. Gerstner, R. Siegfried, and N. Kratzke [has no Superior]

[has Superior] Read Broadcastchannel Visual perception

Acceptance of order

Submit request

Move

Get status [Not Suitable] Analyse appropriateness [Order = Mark]

[Order = Analyse effects of fire] [Order = Fight]

[Suitable] Analyse effects of fire

Fight the target

Mark the target Calculate prioritization value [Platform = Ship]

[Platform = Army/ Infantry]

Send to Broadcastchannel

[Platform = Airplane] Check operating time [Not enough Time] Move [Enough Time] Key: Control-Flow Sensor

Effector

Data-Flow

Fig. 4. Behavior of the Engagement-Agent (depicted as sensor-eﬀector-chains)

Engagement-Agents available in the model coordinate themselves using a shared medium (in our case, the so-called broadcast channel ). We deliberately chose two very diﬀerent patterns to highlight the diﬀerent approaches. Of course, many nuances in between these two patterns may be conceived. Exploring more coordination patterns and their eﬀects on the chosen objectives is part of our future work. Engagement Unit. Figure 4 illustrates the behavior of the Engagement-Agent. The Engagement-Agent waits at his starting point until he is ordered to engage a target. As soon as he receives an order, he starts moving to the target until it gets in the range of his weapons and attempts to destroy the target. If the range of his weapons is higher then his line of sight he has a need for a marker to

Agent-Based Simulation of Joint Fire Support Teams

63

help him marking the target. This marker can be any other Engagement-Agent or the reporting Reconnaissance-Agent if they are capable of providing marking services. After the target is fought, the Engagement-Agent checks his operating time whether he can be assigned to another mission or if he has to move back to his starting point. Decentralized Coordination. Together with this agent the decentralized coordination shall be explained: If there are no superior Command and Control units the Engagement-Agents have to manage among themselves. Because there is no hierarchy between all the Engagement-Agents they have to communicate with each other to identify the most appropriate agent to engage a speciﬁc target. To do this, the agent reporting a new target publishes the request for ﬁre support into a so-called broadcast channel which is accessible by all free unbound agents. Each Engagement-Agent now evaluates the situation by calculating a speciﬁc value representing his appropriateness for engaging this target. This value is published by all agents into the public broadcast channel so that every agent gets to know the value of all agents. Each receiving agent can now check if his own value is the highest or if there is any higher value published by another agent. This way the most appropriate agent is identiﬁed and orders himself to fulﬁll the task of ﬁghting the target.

(1/2)

(1/7)

CoordinatorͲ Agent

ReconnaissanceͲ Agent

(4/8) (1) (1/2/3/7)

(1/2/4/5/6)

EngagementͲ Agent (1/2/3)

Fig. 5. Information exchange relations between the various agents

2.4

Coordination and Information Exchange

In order to coordinate themselves, the agents have to exchange a variety of information (regardless of the chosen coordination pattern). Figure 5 illustrates the information exchange relations between the agents (roughly in chronological order as they occur within a request for Joint Fire Support):

64

1. 2. 3. 4. 5. 6. 7. 8.

C. Gerstner, R. Siegfried, and N. Kratzke

Request for ﬁre support Request for marking Current status Order to mark a target Order to engage a target Order to analyse eﬀects of ﬁre Result of analysing eﬀect of ﬁre Task fulﬁlled

Although only three diﬀerent types of agents are involved in this model, eight information exchange relations are required to represent the basic activities of Joint Fire Support Teams. Diﬀerentiating more types of agents (e. g. diﬀerent types of Engagement-Agents requiring diﬀerent information and/or coordination) will lead to even more complex information exchange relations. Also, the chosen model of hierarchical coordination is simpliﬁed, as no diﬀerences between lower and higher levels of the chain of command are considered. Integrating these diﬀerences will, most probably, also increase the number of information exchange relations. Furthermore, two points are worth mentioning. First, it is obvious that the order of exchanging these information is important. Although this order is given roughly by the numbers (and slightly indicated in Figure 5), a more thorough description of the protocol (exchanged information including their order) is preferred. Second, the information exchange protocol deﬁned within this model does not account for any uncertainties nor wrong or missing information. Enabling the agents to cope with uncertainty is a crucial point in the further development of this model. As the real environment is highly uncertain (e. g. moving targets, incorrectly assessed targets, failing radio communication, etc.), the model has to reﬂect this uncertainty.

3 3.1

Results and Experiences Results

After successful implementation of the model, nine diﬀerent scenarios were simulated. In these scenarios we tested three diﬀerent ways to identify the most appropriate engagement unit while using diﬀerent approaches of prioritization. One way of prioritization was to command the unit with the shortest way to the target, which thus can engage the target the fastest. A second approach is to identify the engagement unit, which is just able to ﬁght the target. This way higher-ranked units are saved for more dangerous targets while assigning combat units according to their strengths. The third approach is a mixture of the ﬁrst two and should combine the advantages of both. Each of the nine scenarios was simulated 100 times to get average values. Figure 6 illustrates our main ﬁndings: Although basically ﬂat hierachies are aimed for, they reach their limit quite fast in the constraints of reality. First and foremost, a huge amount of combat units is needed for optimal coverage

Agent-Based Simulation of Joint Fire Support Teams

65

160 140 120 100 Timesteps

80 60 40 20 0 1

# Coordination 1 centralized 2 decentralized 3centralized 4decentralized d l d 5centralized

2

3

4

5

Shortdescription Ͳ 3coordinator,3recce,9engagement Ͳ 3recce,9engagement Ͳ 3coordinator,6recce,18engagement Ͳ 6recce,18engagement Ͳ 5coordination,3recce,9engagement

Fig. 6. Average time from identiﬁcation of a target to ﬁnal engagement in diﬀerent scenarios

in decentralized coordination. Also, perfect communication has to be ensured between all units to enable the necessary interaction and coordination as well as to avoid multiple ﬁghting of the same target. In summary, decentralized coordination leads to smaller average times from identiﬁcation of a target to ﬁnal engagement, while at the same time requiring extensive resources. Central coordination is more eﬀective in terms of engaged units and reduced resource requirements, but tends to take more time. 3.2

Experiences

The most diﬃcult part of the development was to ensure the correct coordination between the agents. Modeling the various information exchange relations and the subsequent activities to be carried out by the agents is a challenging task. Even though only three diﬀerent types of agents were considered, it is diﬃcult to keep track of the intricate interplay of mutiple agents. By following the GRAMS reference model to develop the agent-based model, we could focus purely on domain-speciﬁc issues. In this sense, the GRAMS reference model served very well as a guideline throughout the development process. The strict seperation of events and actions deﬁned by the GRAMS reference model turned out to be helpful also. This separation allowed us to construct complex event-action chains where each event could trigger diﬀerent actions at the same time, whereas these action could produce events as well.

66

C. Gerstner, R. Siegfried, and N. Kratzke

While being beneﬁcial, these event-action chains caused trouble at the same time. In fact, it turned out that they could hardly be analysed and debugged. This is not necessarily a drawback of the GRAMS reference model, but has at least two reasons: First, the tool chain currently available does not support all aspects of the GRAMS reference model very well and debugging features are far from complete. Second, and perhaps more notably, this complexity of modeling coordination patterns may be immanent to these kind of models.

4 4.1

Conclusion and Outlook Conclusion

We presented an agent-based model to analyse the inﬂuence of diﬀerent coordination patterns on so-called joint ﬁre support teams. In this (restricted) model, three diﬀerent types of agents had to coordinate themselves (ReconnaissanceAgents, Coordinator-Agents and Engagement-Agents). The coordination patterns investigated in a ﬁrst stage ranged from a typical hierarchical (centralized) coordination to completely decentralized coordination. A thorough model validation still outstanding, the ﬁrst simulation results indicate slight advantages of decentralized coordination about a centralized coordination. At the same time, our experience regarding the modeling of coordination is that the complexity increases very fast and even smaller scenarios (in our case, with three diﬀerent types of agents and less than 50 agents in total) are quickly hard to overlook. 4.2

Outlook

The future work will focus mainly on two topics: First, uncertainty has to be taken into account. This includes both modeling uncertainty as well as enabling the agents to cope with uncertainty and unexpected situations. Given the inherent uncertainty Joint Fire Support Teams are facing in the real world, incorporating this aspect into the model is a great step forward towards a more realistic model. Second, the trade-oﬀs between centralized and decentralized coordination in combination with the resource needs and utilization will be in the focus of future work. Furthermore, a lot of restrictions were made to reduce the complexity of the ﬁrst model version. This model was very helpful to get acquainted to the problem domain. With this background knowledge the restrictions may be relaxed and a larger parameter space will be covered (cp. [8]). A ﬁrst extension is to implement moving targets which will add a lot of complexity to the coordination of the agents and will naturally introduce a huge amount of uncertainty to be dealt with by the agents. Also, improved route ﬁnding algorithms for the recconnaisance agents are of interest (cp. [7]). Finally, we want to calibrate and validate the model as well as the parameters to move on from qualitative to quantitative investigations.

Agent-Based Simulation of Joint Fire Support Teams

67

References 1. Alberts, D., Garstka, J., Stein, F.: Network Centric Warfare. DoD Command and Control Research Program (CCRP) (1999), http://www.dodccrp.org/files/Alberts_NCW.pdf 2. Alberts, D., Hayes, R.: Power to the edge. DoD Command and Control Research Program (CCRP) (2003), http://www.dodccrp.org/files/Alberts_Power.pdf 3. Cares, J.: An Information Age Combat Model. Tech. rep., Director, Net Assessment, Oﬃce of the Secretary of Defense (2004), http://alidade.net/recent_research/IACM.pdf 4. Cares, J.: Distributed Networked Operations: The Foundations of Network Centric Warfare. iUniverse (2006) 5. Gerstner, C.: Erweiterung und Implementierung eines Modells zur Analyse von Fragestellungen zur Koordination verteilter Organisationsstrukturen. Bsc thesis, University of the Federal Armed Forces Munich (December 2009) 6. Newmann, M.E.J.: The mathematics of networks. In: Blume, L.E., Durlauf, S.N. (eds.) The New Palgrave Encyclopedia of Economics, Palgrave Macmillan, Basingstoke (2008), http://www-personal.umich.edu/~mejn/papers/palgrave.pdf 7. Paruchuri, P., Pearce, J., Marecki, J., Tambe, M., Ordonez, F., Kraus, S.: Coordinating randomized policies for increasing security of agent systems. Journal of Information Technology and Management (ITM) 10(1), 67–79 (2009) 8. Santamaria, A., Warwick, W.: Sailing to the Model’s Edge: Testing the Limits of Parameter Space and Scaling. In: Proceedings of the BRIMS 2010 (2010) 9. Siegfried, R.: A General Reference Model for Agent-Based Modeling and Simulation. In: 7th European Workshop on Multi-Agent Systems, EUMAS 2009 (December 2009) 10. Siegfried, R., Lehmann, A., El Abdouni Khayari, R., Kiesling, T.: A Reference Model for Agent-Based Modeling and Simulation. In: Proceedings of the Spring Simulation Multiconference, SCS (2009), Agent-Directed Simulation Symposium

Intelligent Collaborative Care Management (ICCM): Achieving and Managing Contractual Relationships and Service Delivery Christian Guttmann1,2, Ian Thomas2 , Kumari Wickramasinghe2, Michael Georgeff2, Hamid Gharib3 , Simon Thompson3, and Heinz W. Schmidt4 1

ETISALAT BT Innovation Centre (EBTIC) Khalifa University of Science, Technology and Research PO Box 127788, Abu Dhabi, UAE [email protected] 2 Faculty of Medicine, Nursing and Health Sciences Monash University, Melbourne, Australia {ian.thomas,kumari.wickramasinghe,michael.georgeff}@monash.edu 3 BT Research, Adastral Park, Ipswich, United Kingdom {hamid.gharib|simon.thompson}@bt.com 4 School of Computer Science and Information Technology, RMIT University, Australia [email protected]

Abstract. The Intelligent Collaborative Care Management (ICCM) project offers a comprehensive framework and architecture for understanding and helping to manage the complete life cycle of customer care. This framework formally captures the main components of providing a customer with a number of possibly interrelated services by various service providers. The customer has objectives expected to be fulfilled by these services and the service providers themselves may have objectives in delivering the services to the customer. These services are delivered over time and potentially the entire lifetime of the customer. The ICCM framework specifies two functionalities: a) basic functionalities to generate care plans, form contractual relationships and deliver services, including the specification of many complex interactions and constraints between service providers and customers, and b) extended functionalities to support adherence of contract formation and service delivery and variations of contractual commitments. We present first empirical results into the performance of a prototype system based on the ICCM framework. A key insight is that mental models of human agents (i.e., service providers and customers) are required to efficiently intervene when contractual commitment and service delivery processes go “off-track” and when the contract requires renewal and variation. Further, we offer guidelines to optimise the benefit/cost trade-offs of applying interventions in achieving and managing relationships and delivery.

This project is supported in part by ARC Linkage Grant LP0774944 and British Telecom Agreement CT1080050530. We wish to thank Dr Kay Jones, Professor Leon Piterman, Professor Peter Shattner and Mr Akuh Adaji for their advice and contribution to the project.

C. Guttmann, F. Dignum, and M. Georgeff (Eds.): CARE 2009/2010, LNAI 6066, pp. 68–84, 2011. c Springer-Verlag Berlin Heidelberg 2011

ICCM: Achieving and Managing Contractual Relationships and Service Delivery

69

1 Introduction Given the uncertain nature of the world and the possibly changing circumstances and objectives of the various participants, it is possible (or even likely) that the services do not get delivered in the contractually agreed manner. It may be possible to avoid such failures by monitoring the behaviour of the various participants and intervening to ensure that the contractual obligations are adhered to or at least their objectives realised. Similarly, the contractual relationship itself (or its renewal) may be subject to modification or abandonment as a result of the changing circumstances or objectives of the participants. It may be possible to monitor the behaviour of the participants and intervene to ensure that advantageous governance is maintained, in much the same way as delivery of the services themselves is monitored and managed. Our goal is to develop a framework that can manage the service delivery and contract variation under these changing circumstances and objectives. In developing this framework, we focus on the behaviour of human agents that are involved both as customers and as service providers. As well as changes in the environment of service delivery, certain changes, inconsistencies or deficits in the “mental attitudes” of these agents may affect service delivery or the existing contractual arrangements. By means of this framework, we aim to: 1. Develop and implement algorithms for generating service care plans given a specification of the problem domain and the objectives of the customer, including a specification of the service types available and possible constraints among them; 2. Develop and implement architectures for supporting the contractual commitment process and the ongoing monitoring and management of the contractual relationship (including contract renewal and extension), based on predefined and domainspecific governance principles and processes; 3. Develop and implement architectures for supporting the service delivery process and the ongoing monitoring and management of service delivery (including recognition of the need for contractual variation), based on predefined and possibly domain-specific service delivery principles or processes; 4. Model of a conceptual framework to demonstrate how explicit models of the mental attitudes of the human agents involved in the contractual and service delivery stages can be used in Items (2) and (3) above; 5. Develop a simulator for examining the behaviour of the total system and the component stages, both for specific scenarios and on a population (statistical) basis; 6. Apply and test the framework, algorithms, architectures, and simulations to the problem of chronic disease management in health care, generalising the findings to other problem domains; and 7. Relate this work to existing literature and research. The paper is organised as follows. Section 2 discusses the basic model of care that specifies a complete world and system with no uncertainty, incompleteness, and unbounded computational resources. This discussion is the basis to motivate our endeavour to build a comprehensive architecture for life cycle management suitable for domains associated with uncertain and incomplete information, Section 3 (one particular emphasis is

70

C. Guttmann et al.

&RRUGLQDWLRQ/HYHO 6SHFLILFDWLRQ

(QYLURQPHQW

&RPSRVLWLRQ

&RRUGLQDWLRQ 6SHFLILFDWLRQ

&RQWUDFWXDO &RPPLWPHQW

6HUYLFH 'HOLYHU\

,PSOHPHQWDWLRQ /HYHO

&RQWUDFWXDO7DVN/HYHO 6SHFLILFDWLRQ

6HUYLFH'HOLYHU\ 7DVN/HYHO6SHFLILFDWLRQ

7DVN/HYHO 6SHFLILFDWLRQ

Fig. 1. ICCM architecture: Basic Model of Customer Care

the uncertainty and incompleteness of human participants and their mental attitudes). Section 4 discusses the adherence and variation support required for the efficient and effective maintenance of agreements and delivery of services. Section 5 discusses related research. Section 6 summarises and outlines future directions of this research.

2 Basic Model of Customer Care The Intelligent Collaborative Care Management (ICCM) architecture consists of three stages to address the functionalities of customer life cycle management: 1. Composition: develops a service care plan by composing service types that realise the objectives of service providers and customers; 2. Contractual commitment: forms and manages contractual relationships between service providers and customers; and 3. Service delivery: manages the execution of the contractual obligations of the service providers and customers. Building an architecture with these stages is a complex task. We start by specifying a basic model of customer care and then highlight the need to extend this model to cope with greater complexities, Section 3. A basic model of customer care specifies the components and interactions required to achieve and manage composition, contractual commitments and service delivery. To build an architecture that provides the life cycle management functionalities, we need to specify the properties of a domain where customers and service providers operate. Some domain properties are used in all stages and some properties are used only in a particular stage. Hence, the domain specification is divided into two components, which we refer to as the coordination level specification and the task level specification. These specifications then need to be implemented in architectures that deliver the required functionalities, which we refer to as the implementation level. This framework is represented schematically in Figure 1.

ICCM: Achieving and Managing Contractual Relationships and Service Delivery

71

2.1 Coordination Level Specification: Sharing Domain Properties The coordination level specifies domain properties that are shared in all three stages. These properties include the representation of goals and constraints as well as data-flow or control-flow dependencies between services providers. They also include meta-level constraints between service providers to enable collaboration on the care plan. 2.2 Task Level Specification: Domain Processes and Information The task level specifies how to achieve and manage the tasks involved in achieving and managing contractual commitments and service delivery in a particular domain (Figure 1). The specified tasks are compliant with governance policies and partnership agreements. – Contractual Commitment: This stage is divided into two separate tasks: the achievement of a contractual agreement, and the management of a contractual relationship. During contract formation, the service providers and consumers enter into negotiation to achieve mutually acceptable terms of business. This is done through an interchange of requests and proposals describing aspects on an acceptable agreement. The outcome of this is a contract, specifying the terms that all contract parties consider acceptable. These terms could include the delivery date or content of the service being delivered. As renewals and modifications are often required during the life cycle of a customer, the processes to manage contractual relationships are also specified. – Service Delivery: In this stage, the contract parties carry out the agreed services within the parameters specified in the contract. The processes in the delivery stage aim to ensure the following. • Service providers deliver the services as committed, and • Customers perform the required activities of the contract. In the basic model of customer care the processes required to deliver services are specified in the task level specification. The contractual commitment and service delivery stages are associated with task level specifications, but not the composition stage. For example. in the life cycle management for patients with chronic disease, one element of reaching contractual commitment is to determine the distribution of tasks among care providers and to get their agreement to participate in the care of the patient. In the same application, waiting time for a care provider (which may vary for each provider) which affects the service delivery, is a parameter defined in the task-level specification. On the contrary, composition is a generic planning stage decomposing the coordination-level specification in a generic way. 2.3 Implementation Level: Three Stages for a Basic Model of Customer Care This section describes the three main procedures requires to design a basic model customer care: composition, contractual commitments and service delivery (Figure 1). The composition stage generates a service care plan by selecting services relevant to the characteristics and requirements of the customer. The appropriate services are selected from the service model (based on a coordination-level specification of the

72

C. Guttmann et al.

domain). The composition of services is a care plan for the customer. A care plan is a collection of services, including associated provider types and times for service execution. The composition stage offers two mechanisms that check constraints during the composition of a care plan. 1. Composition of service provider contracts and their provided and required operations within a component-based architecture [1–3], and 2. Composition algorithms that apply context-sensitive goal-directed service composition. The achievement of care plan objectives requires agents to agree formally to binding agreements. This is achieved by the contractual commitment stage characterised by two functionalities. – Achieving contractual commitments of service providers to deliver the services. – Managing contractual relationships among the service providers and the customer. In many applications, forming a contractual commitment involves the distribution of tasks defined in the service care plan to providers. The distribution often involves negotiation between service providers and may involve changes to the care plan itself. A negotiation phase achieves a contractual agreement between the customer and service providers (we refer to both types of agents, a customer and a service providers as a contract party). After the negotiation, a mutually acceptable agreement is reached. Customers and service providers are obliged to perform all the required activities as specified in the contract. This contract is to be performed in the service delivery stage and this contractual relationship can potentially change at any time during the lifetime of the customer. These changes could be recognised as part of managing the contractual relationship (the customer’s contract is about to expire, or the customer desires a change to his contract), or at the service delivery stage to address unresolvable circumstances (e.g., a significant change of the customer’s circumstances). In both these circumstances, the contractual commitment stage is required to manage potential modifications and renewals of contractual relationships. The task-level specification for the contractual relationship describes data, functional and processes used by the contractual commitment stage to perform the above functionalities. After the contractual commitment phase, service providers and customers are obliged to deliver the tasks as specified in the contract. The service delivery stage offers the following functionalities. – Service providers deliver the services as agreed in the contractual commitment stage, and – Customers perform the activities required to achieve the customer’s objectives. The task-level specification for service delivery defines data and processes to perform above functionalities.

3 Uncertainty, Incompleteness and Bounded Resources: Understanding the Need for Adherence and Variation Support A basic software architecture enables the specification and realisation of a basic model of customer care, including functional properties and business processes required for

ICCM: Achieving and Managing Contractual Relationships and Service Delivery

73

all aspects of the management of customer life cycle. The specification and realisation of this architecture is complex: they need to address the following. – composition of service plans (with various types and safety constraints), – achievement and management of contracts (including governance policies and partnership agreements), and – management of service delivery (with scheduling and management policies). If these specifications were all executed according to their design intentions, the system would always produce outcomes in time and at a quality level at which they were specified. However, many real world applications are associated with complexities that cannot be captured by a basic model of customer care. In many applications, the basic model does not adequately address the issues arising with bounded rational human and system agents that have incomplete and uncertain information about the world. Agents act in environments where agreements, requirements, plans and objectives change dynamically. Given these complexities, agents are likely to go off-track from their intended design and this off-track behaviour potentially jeopardises the execution of the entire system, and at worst fails the execution of the system leaving as the only options to recover or reset it.

&RPSRVLWLRQ

&RQWUDFWXDO &RPPLWPHQW

6HUYLFH 'HOLYHU\

$GKHUHQFH6XSSRUW &RQWUDFWXDO9DULDWLRQ 6XSSRUW

$GKHUHQFH6XSSRUW &RQWUDFWXDO9DULDWLRQ 5HFRJQLWLRQ

&RQWUDFW&RPPLWPHQW VXSSRUWVSHFLILFDWLRQ

6HUYLFH'HOLYHU\ VXSSRUWVSHFLILFDWLRQ

,PSOHPHQWDWLRQ /HYHO

(QYLURQPHQW ,QFRPSOHWH 8QFHUWDLQDQG %RXQGHG 5HVRXUFHV

$GKHUHQFH &RQWUDFWXDO9DULDWLRQ 6SHFLILFDWLRQ

Fig. 2. ICCM architecture: adherence and variation support to cope with uncertainty, incompleteness and bounded resources

Many exigencies are not foreseen or not included in the specification of a basic model of customer care. Hence, there could be many potential reasons for a failure during system execution, some are environmental, e.g., heat waves can compromise the distribution of electricity in metropolitan areas. Some reasons are related to agents having incomplete and uncertain knowledge of the domain. And yet others are related to malfunctioning components, e.g., when a wireless blood pressure machine is damaged and transmits wrong data to a central station.

74

C. Guttmann et al.

(QYLURQPHQW

&RPSRVLWLRQ

&RQWUDFWXDO &RPPLWPHQW

6HUYLFH 'HOLYHU\

$GKHUHQFH6XSSRUW &RQWUDFWXDO9DULDWLRQ 6XSSRUW

$GKHUHQFH6XSSRUW &RQWUDFWXDO9DULDWLRQ 5HFRJQLWLRQ

&RQWUDFW&RPPLWPHQW VXSSRUWVSHFLILFDWLRQ

6HUYLFH'HOLYHU\ VXSSRUWVSHFLILFDWLRQ

,PSOHPHQWDWLRQ /HYHO

%', $GKHUHQFH &RQWUDFWXDO9DULDWLRQ 6SHFLILFDWLRQ

Fig. 3. ICCM architecture: Adherence and variation support addressing the human mental deficits by means of the Belief-Desire-Intention model

3.1 Problem Description The complexity of effective monitoring-recognition-intervention processes in managing customers and providers includes the following issues. – What, when and how to monitor. Monitor events in the environment and agent behaviour. – What, when and how to recognise. Identify key events (or absence of key events) to decide on interventions. – What, when and how to intervene. Move processes back on-track such that a customer becomes loyal. – Monitoring-Recognition-Intervention Cost: Effort to achieve any of the above processes. No existing research has addressed these issues comprehensively [4–8]. 3.2 Human Participants and Mental Attitudes A special case of system components that exhibit unreliable behaviour and jeopardise system execution are human agents (Figure 3). Human agents are involved in achieving critical steps during the formation and management of a contractual commitment and service delivery. Human agents are bounded rational and make decisions with incomplete and uncertain information, thus performing activities (or indeed not performing activities) that are off-track and hence jeopardising the intended outcomes. Our interest is in modelling the behaviour of human participants as we are then able to more effectively recognise deviations or potential deviations from the agreed plan, and more effectively intervene to bring the plan back on-track. To do this, we model the mental attitudes of the human agents using the BDI paradigm.

ICCM: Achieving and Managing Contractual Relationships and Service Delivery

75

What is BDI? BDI stands for Belief-Desire-Intention and is a folk-psychological paradigm describing the practical reasoning of humans [9]. The agents are described using the Belief-Desire-Intention architecture [9, 10]. In short, each agent is associated with mental states: beliefs, desires (also called goals), and intentions. Agents follow their desires based on their beliefs (if a desire is believed to be achievable, it turns into intentions, or intended goals). Following Bratman, we consider “humans as planning agents” [9] in the context of Castelfranchi’s belief-based goal dynamics [11]. BDI uses three constructs of mental attitudes: – Beliefs – an agent’s factual model of the world. – Desires – states that an agent desires to be in, but not necessarily tries to achieve. – Intentions – states that an agent actively tries to achieve. As opposed to beliefs and desires, intentions are distinguished by the concept of commitment. Commitments are concerned with events that should happen in the future. They enable temporal persistence in plans and further plans being made based on already committed plans. We have chosen to use the BDI paradigm because it enables an intuitive description of human behaviour. The BDI framework offers a suitable approach to capture the mental attitudes of intelligent agents. This is partially because the BDI approach is a folk-psychological theory and enables the modelling of the behaviour of human and system agents intuitively. For example, as part of customer life cycle management, companies (1) develop models of customer types based on parameters: life time value, loyalty, propensity to change suppliers, and behaviours, (2) analyze customer behaviour for different customer types, (3) develop predictors which indicate possible migrations or switching, and (4) apply strategic interventions to retain customers based on the developed models and predictors [5]. What is a Mental Deficit? In BDI terms, a mental deficit can make an agent’s behaviour deviate from its specified behaviour. In other words, if a mental deficit exists then the mental state of an agent is not as it should be, and an agent does not do what it is supposed to do. A mental deficit is observed when an agent exhibits unreliable behaviour and it deviates from its expected behaviour. We use the BDI paradigm to describe the notion of three types of mental deficits. Bratman’s view state that humans frequently decide in advance a plan for the future, termed as future-directed intentions. Castelfranchi’s work on goal processing considers beliefs as the deciding factor for selecting and executing goals. Consolidating these view, we identify the main causes for a breach of contract from agent’s perspective (please note that we are not interested in lower-level platform failures) as follows. – changes to the beliefs on which commitments were made. – changes in agent’s expectation/priority from commitment time to execution time. The specific deficits are then the following three. – A belief deficit is an agent’s mental state that • does not include a belief B (but should be included) required to form an intention, or

76

C. Guttmann et al.

• includes a belief B’ instead of a belief B that represents a fact correctly. The model of belief-based goal dynamics [11] proposes that beliefs are at the core of selecting and executing goals: humans commit to obligations based on beliefs at commitment-time and follow guidelines based on beliefs at execution-time. A belief plays a critical role in forming an intention and in activating a goal of a BDI agent. If an agent has a belief deficit, an intended goal may not be pursued. – A plan deficit is an agent’s mental state that • does not include a plan P (but should be included) enabling an agent to pursue an intended goal, or • includes a plan P’ instead of a plan P enabling an agent to pursue an intended goal. With a non-existing/incorrect plan, a BDI agent is not able to pursue an intended goal. – An intention deficit is when an intention to achieve a specified goal is of lesser priority than intentions that achieve other non-specified goals. Given these uncertainty, incompleteness, and bounded resources, the next section offers recognition and intervention support specifications that are not captured by the basic model of care. A special case of this model is a world where humans are involved with their own mental attitudes.

4 An Extended Model of Collaborative Care Our research interest is to prevent failure, and in particular to offer efficient recognition and intervention support. One way to do this is to introduce domain specific recognisers for precursor events and domain specific intervention processes. In short, coping with uncertainty, incompleteness and bounded rationality in life cycle management requires the development of a system component that enables the execution to be ”‘on-track”’ according to the design intentions of the system. As seen in Figure 2, this requires an extra component to provide adherence and variation support. This component consists of monitoring, recognition and intervention processes. To monitor the behaviour of a contract party, we apply intrusive and overhearing methods [12]. The intrusive approach requests agents to communicate required information, and the overhearing approach monitors the message exchanges among agents passively. We monitor “precursor events” to detect whether events are favourable or unfavourable to the advancement of relationships. The detection of favourable events indicates that processes are on-track (the execution is as specified), and the absence of favourable events corresponds to processes being off-track. The reverse applies for the detection or absence of unfavourable events. Off-track execution can imply that an agent has a mental deficit. The mental-state recognition process associates a precursor event with a belief, intention or a plan deficit. This information is used to select a tailored intervention for an agent with a certain deficit. An intervention can prompt an agent to perform specified activities which in turn moves processes back on-track. Future directions include the investigation of issues other than the deviation of human behaviour that can threaten the advancements of relationships. The elements of an adherence and variation support framework include four components:

ICCM: Achieving and Managing Contractual Relationships and Service Delivery

77

1. Precursors: Generally a contract failure (failures in service delivery and/ or maintaining contractual relationships) does not occur instantly. Rather, it is a gradual failure that occurs due to: (1) non-occurrence of sequence of actions (steps or states) required for the successful execution of the contract and (2) occurrences of certain actions (steps or states), which may jeopardise the successful execution of the contract. If these a priori actions steps and or states are described as precursors, the (non-)occurrences of such precursors at run-time, indicate whether the contract execution is on-track (e.g., the patient has made an appointment to visit a service provider) or likely to go off-track (e.g., very low or high blood pressure). 2. Precursor detection strategies: Mechanisms to detect the occurrence or missing occurrence of such precursors at run-time. 3. Recognition processes: Processes to identify possible reasons which may have led to the (non-)occurrences of precursors (e.g., the reason for wrong data is either the device is malfunctioning or the patient is having a rare medical condition). 4. Intervention processes: Strategies to intervene with contract parties and system components to reduce the likelihood of the contract being violated (e.g., repair the device or refer the patient to a specialist). There are industry practices and research initiatives that consider some of these elements for adherence support. For example, as part of customer life cycle management, telecommunication companies: – develop predictors which indicate possible customer migration to other companies - precursors; – analyse customer behaviour (e.g., using service usage patterns) - precursor detection strategies; – develop models of customer types based on parameters such as lifetime value, loyalty, propensity to change suppliers, behaviour, and map them to predictors - recognition processes; and – apply strategic interventions to retain customers based on the developed models and predictors - intervention processes. From the perspective of electronic contract formation and management, in the CONTRACT project [13], each contract is associated with two types of states: critical states (CS) and danger of violation (DOV) states. CS define the states which are compulsory for the successful execution of the contract. The DOV states indicate a possible violation of the contract, but they are not explicit contract states. The precursors defined above is similar to the DOV states. The extension is that the recognition processes identify the reason which prevented the (non-)occurrence of the precursor, which help to select a tailored and more effective intervention. In addition to the uncertainty and inconsistencies caused by system components, when human contract parties are involved their bounded rational reasoning process may prevent them form executing committed obligations as discussed in Section 3.2. 4.1 Contractual Commitment This section describes initial concepts required to study the formation of contractual relationships using adherence support. Our contractual commitment domain consists of

78

C. Guttmann et al.

$$&3$ &3$

$$&3$ &3$

$$&3$ &3$ $$&3$

$$&3$ &3$

&3$ &RQWUDFW &RPPLWPHQW

$$&3$

$$&3$ &3$

&3$ $$&3$

$$&3$ &3$

&3$ &3$ $$&3$

Fig. 4. Conceptual framework for contractual formation and management

a set of software agents A, each having different responsibilities during the allocation process. We have the following agents: the Distribution Agent (DA), the Contract Party Agents (CPA) are the health care provider or negotiation agents, and Adherence Agents (AA). Note that we consider two specific CPAs, the CP AGP and CP AP atient which have different responsibilities to the other health care providers. One aim of contract formation is to assign a service provider to every service goal appearing in the careplan. Our interest is in the process of forming an agreement on this assignment and the associated tasks. This process is based on previously agreed procedures of negotiating the agreements. This “process-based” model of formation is the predominant way in which agreements are negotiated in practice (particularly among partner organisations). For example, if an agent requests information that is critical for the formation of an agreement, the requested agent is assumed to respond in time for the information to be useful. Regrettably, many agents do not act according to the design intentions of a system [14]. In our case, the “intended design” corresponds to the previously agreed procedures of negotiation in the formation of agreements. We treat the process of reaching and managing contractual commitments as a management process among collaborating agents, and develop processes that keep the execution “ontrack” (according to the “intended design”). This management process is also applied in the service delivery stage (described in the next section). The rules that govern the assignment process partially are defined in the domain model (e.g., in health care, many of these rules are imposed by the funder). These rules

ICCM: Achieving and Managing Contractual Relationships and Service Delivery 10

10

9

9

8

8

7

7

6

6

5

5

4

4

3

3

2

2

1

1

79

0

0

0

2

4

6

8

10

FF = Never Forget FF = Very Forgetful FF = Often Forgetful FF = Sometimes Forgetful FF = Very Forgetful FF = Always Forget

12

14

0

2

4

6

8

10

12

14

FF = Never Forget FF = Very Forgetful FF = Often Forgetful FF = Sometimes Forgetful FF = Very Forgetful FF = Always Forget

Fig. 5. Left figure (without adherence support): Number of confirmed services plotted against the number of days since an agreement request with the 6 forgetfulness settings. Right figure, with adherence support

are domain specific and defined and provided in the domain model, Section 2. In chronic disease management in Australia, Medicare imposes a rule on the contractual commitment process the rule to only sign off on a care plan if at least two health care providers have agreed to perform services defined in the plan. The GP can invoke rules when making negotiation the task allocation with other service providers. This information can be specified in Stage 2 by the composer. In chronic disease management, it is often the GP that imposes rules that regulate the interactions with the health care provider and patient. For example, the GP may stop assigning health care providers after a certain number of days. In this section, we use a subset of all rules are defined in the domain model [15]. In future research, we consider the use of the full specification of service providers, including types and interaction protocols. To illustrate the interaction of these agents, consider the following example involving the management of patients with a chronic disease. We use domain specific terminology – service providers are health care providers (HCPs) and customers are patients. The DA has a careplan (i.e., a care plan) in form of a loosely coupled set of services: S = {s1 , s2 , s3 }. This plan demands the participation of three Health Care Provider (HCP) types: a podiatrist for s1 , an optometrist for s2 and an educator for s3 . The DA now seeks to confirm participation and involvement (i.e., requestsAgreement) by CP Aj ’s that represent a podiatrist, optometrist, and educator. Assume that, as a response to DA’s requests, the podiatrist joins the care team (agree), the optometrist suggests changes (conditional agree) and the educator denies participation (disagree). The DA takes the following actions. It will assign the service s1 to the podiatrist. For the optometrist, the DA will seek approval by the GP (the supervisor of this distribution) regarding the suggested changes to the careplan. In this example, the GP approves the changes, and the optometrist is assigned to s2 . As the educator does not agree, the DA will send a new participation request to the next educator in a list of educators. The next educator then agrees to be assigned to service s3 . We now have a careplan where each service is assigned to a HCP.

80

C. Guttmann et al.

Initial Empirical Study of Contractual Commitments. As seen in Figure 5, we studied the efficiency of adherence support on contract formation under different simulation settings relating to the response behaviour of negotiation agents. The efficiency of the contract formation process is a representation of the quality of a solution. This quality is measured by the Number Of Confirmed Services (NOCS). This measure is represented by the value obtained from the service providerss simulated response behaviours. We now discuss agents that are forgetful and have belief deficits, and start with the results obtained without adherence support (left plots in Figure 5). As the NAs forget to respond to agreement requests (FF), the formation of agreements progressively becomes worse and NOCS decreases accordingly. In fact, if all agents forget to respond then no service is assigned to any service provider (FF=Always Forget). The reason is that NAs simply have no belief that would trigger them to respond to a request, and this constitutes a belief deficit that is permanent. This contrasts to contract formation with adherence support (right plots in Figure 5), where adherence agents remind negotiations agents on a regular basis (a day after an agreement request was sent by the AFA). As seen from the right plots of Figure 5, the forgetful NAs start off with a low level of NOCS, but recover quickly as the days pass. The reason is that our adherence support remedies the permanent belief deficit under the condition that each agent is willing and able to accept the interventions. Overall, our results (Figures 5) demonstrate that settings with adherence support reach a higher level of NOCS, than settings without adherence support. 4.2 Service Delivery The aim of the service delivery stage is to proactively assist unreliable and nonconformant contract parties to execute a plan as agreed. In our approach, this phase is achieved using a set of software agents, each of which is modelled as a BDI agent with particular goals and beliefs. However, while some of the goals and beliefs of these agents may reflect the service model defined in the design stage, the processes (and plan operators) used by these agents are quite different to the process model defined in the design stage. This is because the design stage processes are concerned with high level planning to produce a fixed careplan, whereas in the service delivery stage, we are interested in the timely execution of the plan. A service delivery domain consists of a set of software agents A, each having different responsibilities during the execution of the careplan. – Contract Party Agent (CPA). A CPA is assigned to each contract party to act as a personal assistant. Contract parties communicate with the system via the corresponding CPA. In the health care domain, a service provider CPA assists a service provider in organising his/her calendar, inform if changes to the careplan have occurred and a patient CPA communicates with the patient to inform the messages sent by the system. – Adherence Agent (AA). Each CPA is assigned with a corresponding adherence agent. An AA is able to monitor the interactions and intervene if the status of the patient or the activities of service providers could jeopardise a successful execution of the careplan. In the health care domain, an AA of a patient

ICCM: Achieving and Managing Contractual Relationships and Service Delivery

Fig. 6. Screenshot of prototype when agents in manage the execution of agreements

81

Fig. 7. Percentage of plan success vs responsiveness to incentives (RI) for intention deficit (ID) system

• • • • • •

monitors if the patient sets appointments with the service providers. reminds patients to set and attend appointments. monitors if the patient visits the providers on the appointment dates. inform failures to the corresponding service provider CPA. reminds the patient to purchase repeat medication. alerts a corresponding CPA, if the patient has taken too much or too little medication. • monitors if the measurements such as blood glucose, blood pressure and body weight are within acceptable limits. • alerts a corresponding CPA, if there is an abnormal deviation in the above measurements. Initial Empirical Study of Service Delivery. A screenshot from the prototype shows the agents of the service delivery stage, Figure 6. Figure 6 contains two types of CPAs: PatientAgents (PA) and ServiceProviderAgents (SPA) to support health care domain. Figure 6 shows a scenario where an AA advises the PA to set an appointment for a due visit. As defined in Section 3.2, the service delivery stage considers unreliable and nonconformant behaviour of contract parties as deficits in the mental attitudes; beliefs, intentions and plans, and formalises proactive strategies to intervene when a possible plan failure is detected. The following are examples of these deficits and interventions in the service delivery stage. – Belief deficit: The patient intends to visit the podiatrist in July. – Precursor: The occurrence of an event of the form setAppointment (PatientName, PodiatristName, Time, ’July 2009’) or a contract state of the form checkfeetScheduled(PatientName, PodiatristName, Time, ’July 2009’). – Intervention: Remind the patient that an appointment has to be set. – Intention deficit: The patient chooses to ”go to football’ rather than ’visit a provider“ on the day he has the appointment with a provider.

82

C. Guttmann et al.

– Precursor: The patient’s answer to the question ”whether he is still intending to visit the provider“ raised one day before the appointment date. – Intervention: Provide an incentive to the patient to visit the provider, e.g., send pictures of an amputated leg. – Plan deficit: The patient does not know how to proceed with setting an appointment. That is, a plan does not exist for setting an appointment. – Precursor: the occurrence of an event of the form setAppointment (PatientName, PodiatristName, Time, ’July 2009’) or a contract state of the form checkfeetScheduled(PatientName, PodiatristName, Time, ’July 2009’). Note: The same precursor is used for belief deficit previously. To distinguish from the precursor of the belief deficit, it will first use belief deficit interventions, and if these interventions fail, the system plan deficit interventions. – Intervention: Inform the plan for setting an appointment. As shown in Figure 7, a number of preliminary experiments were carried out considering two deficit related parameters: 1. percentage of belief deficit (BD) 2. percentage of intention deficit (ID), and two responsiveness to intervention related parameters: 1. percentage of responsiveness to reminders (RR) 2. percentage of responsiveness to incentives (RI). The percentage of plan success for varying values of ID and RI is shown in Figure 7.

5 Related Research Free markets are unrestricted and have no (or limited) regulation. Transactions are driven by supply and demand, and terms and conditions of agreements are defined only by the contract parties (with no governance and partnership interference). A problem of a free market is the opportunism and bounded rationality of human actors [16]. Opportunism and bounded rationality can incur a higher transaction cost because individual actors need to renegotiate agreements from scratch. Williamson [16] refers to a specification that constraints the behaviour of contract parties as a partial regulation of a market. Many real markets operate under partial regulation because they incur a transaction cost that is lower than in free markets. When human actors are not behaving as they are supposed to behave, related research considers the following approaches. If the execution of a plan has failed, [7] repair the plan, and then execute the repaired plan. However, in customer life cycle management, the problem does not lie with the existing plan, but with the human actors. Accommodating the behaviour of mentally deficient agents in plans may be a useful research direction, but [7] have not considered this research direction. Additionally, we cannot afford to fail a plan because it means to have lost a customer. Other frameworks consider the detection of norm violations in contracts followed by subsequent punishment [8]. However, there would be few, if any situations where the punishment of a customer is expected to result in an increased customer lifetime value. Our approach aims to prevent the failure of plans in the first place. Our interest is in managing an agent’s mental attitudes (based on the Belief-Desire-Intention formalism [9, 10]).

ICCM: Achieving and Managing Contractual Relationships and Service Delivery

83

6 Concluding Remarks Meeting the needs of consumers requires collaborative service provision and highly adaptive service management. Our project combines the paradigm of service oriented architectures and multi-agent systems into a four-stage framework. Using agents in health care domains is particularly promising as we can address key issues including planning, contract formation and service delivery of services. This paper outlines a conceptual framework, and a prototype system that demonstrates the benefits of agent based modelling. We have also shown how our system achieves careplan execution by means of adherence and variation support mechanisms. We are interested in future extensions on flexible workflows of creating and executing careplans. Due to the compositional nature of a careplan and the well-defined interfaces between service provider functions, there are opportunities for the system to do localised replanning of parts of the plan in the service delivery stage. For example, a part of the plan could be passed back to the composition or contract formation stage if the assessment of the adherence agents that the plan or assigned service providers is untenable. A localised subplan could be recreated or new service providers could be tasked to replace others. Therefore, decisions that are made in earlier stages have a chance to be revisited during the execution of the plan. For future work, more widely distributed systems in health, requirements for traceability of contractual performance are paramount, due to the high level of regulation and the high cost of failure (loss of life, injury, high litigation costs). This also requires reconciling the loose autonomy and uncertainty of agent-oriented approaches at the human-interface level with the tight control and determinacy of service-oriented approaches at the systems and software level. This work is also part of a larger industry and government supported project which is implementing and rolling out a careplan management service to patients with diabetes. The project, called CDM-Net, has already commenced rollout to patients in the Barwon South Western region of Victoria, Australia. We expect that results of the modelling reported here will contribute to the future design and implementation of CDM-Net and that the experience gained from that project will benefit the research work.

Reference 1. Reussner, R., Poernomo, I., Schmidt, H.: Reasoning about Software Architectures with Contractually Specified Components. In: Component-Based Software Quality, pp. 287–325 (2003) 2. Ling, S., Poernomo, I.H., Schmidt, H.: Describing web service architectures through designby-contract. In: Yazıcı, A., S¸ener, C. (eds.) ISCIS 2003. LNCS, vol. 2869, pp. 1008–1018. Springer, Heidelberg (2003) 3. Poernomo, I., Jayaputera, J., Schmidt, H.: Timed Probabilistic Constraints over the Distributed Management Taskforce Common Information Model. In: EDOC, pp. 261–272 (2005) 4. Jennings, N., Faratin, P., Johnson, M., Norman, T., O’brien, P., Wiegand, M.: Agentbased business process management. International Journal of Cooperative Information Systems 5(2), 105–130 (1996)

84

C. Guttmann et al.

5. Carlshamre, P., Regnell, B.: Requirements lifecycle management and release planning in market-driven requirements engineering processes, pp. 961–965 (2000) 6. Erl, T.: Service-Oriented Architecture: Concepts, Technology, and Design. Prentice-Hall, Englewood Cliffs (2005) 7. van der Krogt, R., de Weerdt, M.: Coordination through plan repair. In: Gelbukh, A., de ´ Terashima-Mar´ın, H. (eds.) MICAI 2005. LNCS (LNAI), vol. 3789, pp. 264– Albornoz, A., 274. Springer, Heidelberg (2005) 8. Jakob, M., Pˇechouˇcek, M., Miles, S., Luck, M.: Case studies for contract-based systems. In: Proceedings of the 7th International Joint Conference on Autonomous Agents and Multiagent Systems: Industrial Track, International Foundation for Autonomous Agents and Multiagent Systems, pp. 55–62 (2008) 9. Bratman, M.E.: Intentions, Plans, and Practical Reason. Harvard University Press, Cambridge (1987) 10. Rao, A.: BDI agents speak out in a logical computable language. In: Perram, J., Van de Velde, W. (eds.) MAAMAW 1996. LNCS, vol. 1038, pp. 42–55. Springer, Heidelberg (1996) 11. Castelfranchi, C., Paglieri, F.: The role of beliefs in goal dynamics: Prolegomena to a constructive theory of intentions. Synthese 155(2), 237–263 (2007) 12. Horling, B., Benyo, B., Lesser, V.: Using self-diagnosis to adapt organizational structures. In: Proceedings of the Fifth International Conference on Autonomous Agents, p. 536. ACM, New York (2001) 13. Oren, N., Miles, S., Luck, M., Modgil, S., Faci, N., Alvarez, S., Vazquez, J., Kollingbaum, M.: Contract based electronic business systems theoretical framework. Technical Report D2.2, King’s College London (2008) 14. Zambonelli, F., Jennings, N.R., Wooldridge, M.J.: Organizational abstractions for the analysis and design of multi-agent systems. In: Ciancarini, P., Wooldridge, M.J. (eds.) AOSE 2000. LNCS, vol. 1957, pp. 235–251. Springer, Heidelberg (2001) 15. Wickramasinghe, K., Georgeff, M., Schmidt, H., Thomas, I., Guttmann, C.: Goal-Directed Approach for Process Specification and Service Selection in Customer Life Cycle Management. LNBIP (2011) 16. Williamson, O.: The economics of organization: The transaction cost approach. American Journal of Sociology 87(3), 548 (1981)

A Centralized Framework to Multi-robots Formation Control: Theory and Application Soheil Keshmiri and Shahram Payandeh Experimental Robotics Laboratory, School of Engineering Science, Simon Fraser University, 8888 University Drive, Burnaby, B.C. Canada V5A 1S6 [email protected], [email protected] http://www.sfu.ca/~ ska61/index.html http://www2.ensc.sfu.ca/research/erl/index.html

Abstract. This paper presents a geometric approach to multi-robots group formation with connectivity preservation (from a graph-theoretic perspective) among group members. The controller demonstrates consistency among diﬀerent formations, as well as stability while performing dynamic switching between formations. Inter-robots collision avoidance is delivered through formation preservation, while permitting high degree of formation re-adjustability. It has been proven that such formation approach would result into complete, isomorphic formations (with regards to its first and second isogonic) with edge connectivity λ(G) = 14 n(n−1), and a unique, shortest connectivity link among group members. The complete connectivity along with the isomorphic property of the formations would, in essence, not only guarantee that the communication among the robotic agents will be preserved, but also relax the topological requirements for message passing among group members that might be needed while switching between diﬀerent formations. In addition, the existence of the inter-robot shortest connectivity link at the group level, would ease the message routing once the information sharing among all the members of the group is necessary. Keywords: Cooperative Formation.

1

Agents,

Multi-Robot

Systems,

Group

Introduction

Whether it is a search and rescue, agricultural coverage, security and surveillance, or a game scenario for entertainment purpose, formation coordination among agents plays a crucial role. ”Improved system performance, distributed action at a distance and fault tolerance”[1] are few beneﬁts that may result from one such formation. Area of group formation and formation coordination have been subject to an intensive research and study during last decade. [2] identiﬁes two main methods for formation control problems: Optimization-based method (e.g. [3], [4]) and potential ﬁelds method (e.g. [5] and [6]). A leader-following formation approach is presented in [7] in which every individual is required to C. Guttmann, F. Dignum, and M. Georgeﬀ (Eds.): CARE 2009/2010, LNAI 6066, pp. 85–98, 2011. c Springer-Verlag Berlin Heidelberg 2011

86

S. Keshmiri and S. Payandeh

maintain a speciﬁc position relative to the leader. Local sensing is used in [8] for achieving objectives such as coverage and formation maintenance. Global grouping through local communication is the approach presented in [9] and [10]. In [11] a set of behaviors such as avoidance, aggregation and dispersion are combined to achieve a global ﬂocking behavior. Global knowledge of robots’ positioning information is used in [12] for achieving circles and simple polygons formations approximations. Assuming the availability of global knowledge to every robot, algorithm in [19] achieves formation by ﬁtting the virtual structure (VS) to current robots’ positions. A behavior-based formation, based on three formation control principles namely, unit-centering, leader-referencing and neighbor-referencing is presented in [14]. In their work, authors demonstrate how an interaction law at sub-system level may lead to emergence of collective behavior. A soccer-playing team of three robots, exhibiting two diﬀerent behavior-based group formations is demonstrated in [15]. Some approaches like those of [11], [12], [15], [16], [17] and [18] provide no guarantee on achieving desired formation. Whereas, approaches presented in [19] and [14] require availability of global knowledge to every individual robots. Proposed formation method presented in this article addresses the issue of the group formation and the connectivity preservation among the robotic members. In order to instruct robots to fall into required formation and/or switching among diﬀerent formation conﬁgurations, the controller only needs position information of one single robot (referred to as group leader hereafter). All other group members’ locations would be then computed with regards to the group leader’s positioning information. This would, in turn, signiﬁcantly reduce the communication loads. In addition, no global knowledge of all robots’ positioning information is required for maintaining the formation. Further more, locating one robot at one of the isogonic points of the formation1 , not only assures the ﬁnal formation to remain a connected graph with a high degree of edge connectivity, but also provides a unique shortest communication link through which all necessary message passing can be routed. Even though the controller , to some extent, is based on leader-referencing approach, ﬂexibility and reconﬁgurability of the formation may provide the group with ability to readjust their positioning, while preserving the formation and connectivity among themselves, as per different environmental requirements and constraints. Further more, the proposed formation inherently prevents the inter-robots collisions while permitting a high degree of ﬂexibility and group reformability. The remaining of the paper is organized as follows: Section 2 introduces the nomenclature adapted in this work. The methodology along with series of proofs, demonstrating the properties of the approach are provided in section 3. Simulation result is presented in section 4. Section 5 provides conclusion and some insights on future work and direction.

1

Isogonic points of a triangle are the points that minimize the cumulative sum of the distances of the triangle’s vertices to it. There are two such points associated with every triangle.

A Centralized Framework to Multi-robots Formation Control

87

Fig. 1. First Isogonic Formation. riso i.e. blue-colored circle is located within the convex of r1 Rr2 .

2

Nomenclature Adapted

Referring to Fig. 1, following naming convention has been adapted throughout the paper while making references to the group’s individuals: – Group Leader, R: Other group members’ location information as well as their respective positions within the group will be calculated and maintained based on location information of R. (xR , yR ) has been used while referring to position information of group leader, R. – Side Robots, r1 and r2 : Robots that form left and right wings of the formation with respect to position of the group leader, R. – Isogonic Robot, riso : Robot whose position in the group adhere the isogonic points of the formation. – Central formation Angle, γ: Angle formed between side robots, r1 , r2 and the leader robot R. ∠R, ∠r1 Rr2 and γ are used interchangeably throughout the text while referring to the central formation angle. – Formation base length i.e. r1 r2 = 2x: The distance between side robots r1 and r2 . – Formation height h: It is the Euclidian distance between group leader R and the formation base r1 r2 . – Formation Side Length a i.e. length of the isosceles triangle r1 Rr2 .

3

Isogonic Formation: The Methodology

Group formation might be achieved in a top-down fashion. Another word, we may consider the ﬁnal, desired formation as a single object with certain level of ﬂexibility for reshaping and resizing, and try to ascertain individual’s location in one such formation. Fig.1 illustrates one such formation in which group of four robots are fallen into an isosceles triangle. riso is associated with the location of isogonic point (ﬁrst or second isogonic point) of r1 Rr2 . Considering the

88

S. Keshmiri and S. Payandeh

Fig. 2. Second Isogonic Formation. riso i.e. blue-colored circle is located outside the convex of r1 Rr2 .

formation in Fig. 1, and assuming zero degree of rotation for initial condition, location of every robot involved can be calculated based on group leader, R. Referring to r1 Rr2 , we have: γ γ h = a cos( ), x = a sin( ) 2 2

(1)

xr1 = xR − x, yr1 = yR − h xr2 = xR + x, yr2 = yR − h

(2) (3)

Using (1), we get

Before calculating the position information of riso i.e. isogonic robot, we prove the following theorem. Theorem 1. Location of isogonic robot riso would always lie along the normal from group leader, R to the base r1 r2 . Proof. As shown in Fig.1 and Fig. 2, isogonic points are the points that lie on the intersection of the lines, connecting each new vertex of the three equilateral triangles out of the three sides of the given triangle to the opposite triangle’s vertex. Such equilateral triangles would point outward in case of ﬁrst isogonic point and inward for second isogonic point. Referring to Fig. 1 and considering r1 S r2 , we have r1 S = r2 S & SS ⊥ r1 r2 ⇒ r1 S = r2 S

(4)

Similarly, in r1 Rr2 , we get r1 R = r2 R & RS ⊥ r1 r2 ⇒ r1 S = r2 S

(5)

A Centralized Framework to Multi-robots Formation Control

89

Equations (4) and (5) imply that RS and SS are aligned. Other word, RS ⊥ r1 r2 & r1 S = r2 S

(6)

Same result can be obtained for second isogonic formation where ∠r1 S r2 points upward and hence S lies above R instead. Using result obtained in Theorem 1, location of riso can be calculated as follow: 1. When ∠r1 Rr2 < 120◦ : When all of r1 Rr2 angles are less than 120◦, the isogonic point, riso would lie within the convex of r1 Rr2 , with side angles of 120◦ to every side of the triangle [20]. Fig. 1 illustrates one such scenario. In such a case, location of riso , with the assumption of zero rotation for initial condition, would be: xriso = xR a =

x , h = a × cos (α) sin (α) yriso = yR − (h − h )

(7)

(8)

2. When ∠r1 Rr2 ≥ 120◦ : Before calculating location of riso in such scenario, we prove the following theorem that demonstrates the necessity of introducing the second isogonic point into one such formation. Theorem 2. Leader R and isogonic riso robots’ Locations would coincide if ∠R ≥ 120◦ . Proof. Let riso be the isogonic point of r1 Rr2 with ∠R = 120◦ , r1 R = r2 R. Further more, let riso r2 riso r1 a1 + = (9) a2 riso r1 riso r2 Where (a1 , a2 )−1 represents summation result of left hand side normalized vectors. To ascertain riso so as to have r1 Rr2 as an isosceles triangle with riso being its isogonic point, it is required to satisfy the following [21]: riso r2 riso R riso r1 + + =0 riso r1 riso r2 riso R

(10)

Substituting (9) in (10), we get: riso R 0 a1 = ⇒ + 0 a2 riso R (1 − a21 )(xR − xriso )2 − a21 (yR − yriso )2 = 0

(11)

(1 − a22 )(yR − yriso )2 − a22 (xR − xriso )2 = 0

(12)

90

S. Keshmiri and S. Payandeh

Solving (11) and (12) for xriso we get: (1 − a22 )(1 − a22 )(xR − xriso )2 − a21 a22 (xR − xriso )2 = 0 ⇒ xriso = xR

(13)

(1 − a22 )(yR − yriso )2 − a22 (xR − xR )2 = 0 ⇒ yriso = yR

(14)

Substituting (13) in (12), we get

Due to the result obtained in Theorem 2, it’s necessary to relocate the riso so as to avoid collision while consistency of the approach is preserved. To do so, riso will be relocated to second isogonic point of the r1 Rr2 , once the ∠R ≥ 120◦. Fig. 2 illustrates one such situation. When ∠R = 120◦ , the riso R would be equal to formation side length a. Considering RR riso , we have:

sin(α) =

y RR = ⇒ y = a sin(α) riso R a

cos(α) =

riso R x = ⇒ x = a cos(α) riso R a

(15)

Using 15, coordinates of riso within the group and with regards to the group leader R can be calculated as: xriso = xR − x ⇒ xriso = xR − a cos(α) yriso = yR − y ⇒ yriso = yR − a sin(α)

(16)

With α and a being formation heading angle and formation side length, respectively. 3.1

Formation Preservation during the Rotation

Once all robots are in their designated locations, the entire robotic group might be treated as a single isosceles triangle whose rotational maneuvering can be delivered via the application of transformational matrix to its reference point. In our case, the reference point of the formation is in fact the group leader i.e. R. Having the rotational matrix around Z-axis of group leader been presented by ⎛ ⎞ cos (α) − sin (α) 0 Rz (α) = ⎝ sin (α) cos (α) 0 ⎠ (17) 0 0 1 with α being the heading angle i.e. the orientation angle of the group. Position information of r1 and r2 with regards to R and angle of rotation α i.e. Rz (α) is given by: ⎛ ⎞ −1/2 cos (α) × x + sin (α) × h ⎝ −1/2 sin (α) × x − cos (α) × h ⎠ (18) 1

A Centralized Framework to Multi-robots Formation Control

3.2

91

Computing Isogonic Position Information during the Rotation

There are two scenarios to be addressed: 1. When ∠R < 120◦: Recalling (7) and (8), position information of riso , during the rotation and with regards to (17) is given by: ⎛ √ ⎞ − sin (α) × (−h + x×√3 3 ) ⎜ ⎟ (19) ⎝ cos (α) × (−h + x×3 3 ) ⎠ 1 2. When ∠R ≥ 120◦ : Using (16), position information of riso , using the rotation matrix Rz (α), is given by: ⎛ ⎞ 2a × sin(α) × cos(α) ⎝ a × (sin(α)2 − cos(α)2 ) ⎠ (20) 1 3.3

Connectivity Preservation

Message passing, tasks delegation and assignment, workload scheduling, etc. are among the factors that may inﬂuence the robotic team’s performance in a crucial way. Other word, in order for a team of robots to be able to deliver the designated task(s) successfully and in a cooperative and collaborative manner, it is indispensable for the individual member to stay connected with the other group members throughout the course of the operation. The communication and connectivity issues among the members of a robotic team might be studied (analogously and with the assumption of no hardware failure) through the analysis of the properties of the group-level conﬁguration graph that they form while performing the delegated task(s) in their deployment unit. In this section, the connectivity preservation properties of the proposed controller will be studied through the following theorems. Theorem 3. Isogonic formations of group of robots ri , i ≤ n, would result into a unique, shortest connectivity link among group members. Proof. Refereing to Fig. 1 and Fig. 2, riso is the isogonic point of r1 Rr2 . This implies that riso is the point that minimizes the cumulative sum of the Euclidean distances to other members of formation. Other word, it satisﬁes min(R − riso +

2

ri − riso )

(21)

i=1

Theorem 4. First and second isogonic formations of group of robots ri , i ≤ n, are complete, isomorphic graphs with edge connectivity λ(G) = 14 n(n − 1) where G is the graph of one such formation.

92

S. Keshmiri and S. Payandeh

Proof. – G is a Complete Graph: Completeness of the formation is followed from the observation that every robot i, i ≤ n is connected to the remaining n − 1 members of the group. Other word: ∀(vi , vj ) ∈ V (G) i = j, ∃ei ∈ E(G)

(22)

With G, V(G) and E(G) being the formation and its vertex and edge sets respectively. This results into a formation in which every member is connected to every other n − 1 members and hence 1 n(n − 1) (23) 2 – Isomorphism: Since ﬁrst and second isogonic formations are both complete graphs of same number of vertices, it follows that any pairing oﬀ of vertices gives a corresponding pairing oﬀ of the edges and hence isomorphic to each other. – Connectivity: 1. Edge Connectivity: To have a disconnected component in one such formation, every robot ri , i ≤ n requires to be disconnected from every other remaining members of the formation i.e. from remaining n − 1 other robots. As a result, to have a robot disconnected from the formation, it requires to lose its edge-connectivity from the remaining n − 1 robots in the formation, hence E(G) =

λ(G) =

1 1 × n(n − 1) ⇒ 2 2 1 λ(G) = n(n − 1) 4

(24)

. 2. Vertex Connectivity: Such formation is maximally connected as the only vertex cut which disconnects the formation, is the complete set of robots in the formation.2 Proposition 1. Results obtained in Theorems 3 and 4 hold true for arbitrary robotic group size as long as the positioning information of all members adhere the isogonic formation methodology. Proof. Referring to Fig. 3, it is possible to consider every main robotic member of the group i.e. r1 , r2 and riso as the leaders of the robotic’s clusters whose formations comply with the methodology presented in section 3. Lets C1 , C2 , and Ciso represent clusters formed by r1 , r2 and riso , respectively. Then, 1. Leader Robot R-to-Clusters Communication: Since clusters are formed as per r1 , r2 and riso positioning information, results obtained in Theorems 3 and 4 hold true by considering every cluster, in its entirety, as a node of the main formation graph i.e. riso forms the shortest communication link between R and other clusters, including Ciso . 2

A complete graph with n vertices has no cuts at all, but by convention its vertex connectivity is n − 1.

A Centralized Framework to Multi-robots Formation Control

93

Fig. 3. Extendability of the formation to larger ﬂocks. Every agent of the main formation may form their corresponding cluster whose members positioning information are derived from the main agent of the respective cluster. C1 , C2 , and Ciso refers to cluster formed by r1 , r2 , and riso in main formation and as per Fig. 1 terminology.

2. Clusters’ Members Communication: Since every cluster is formed as per isogonic formation methodology, it is true that r 1 , r 2 , and r iso are the isogonic member of the corresponding cluster. This implies that for clusters C1 , C2 , and Ciso the shortest communication link is satisﬁed within the clusters’ members.

4

Simulation Result

In order to evaluate our method, we have tested our approach in simulation. For this purpose, the leader of a group of four holonomic, circular robots has been equipped with the proposed formation controller. To provide the group with a collision-free path, the controller is coupled with a non-parametric navigation controller [22] that uses the concept of kernel regression [23] for determining an obstacle-free path in a cluttered environment. The schematic of formation controller coupled with the navigation controller is given in Fig. 4. As it is shown, the navigation controller receives the locations information of the obstacles in the ﬁeld and produces an obstacle-free path that will be fed as input to the formation controller. Further more, it provides the controller with the position information of way points (i.e. points that are assigned to individual obstacles as per robot-obstacle safety distance criterion, while computing the path), based on which the collision-free path has been calculated. The controller uses these information to further calculates the group leader motion along the designated path and other group members locations in the formation in each time step. Group leader position information along with information available regarding the obstacles’ way point(s) are then used by the controller to evaluate the current formation conﬁguration validity (as per current environmental situation) and to determine if any readjustment (e.g. resizing such as shrinking or expanding the

94

S. Keshmiri and S. Payandeh

Fig. 4. Formation Controller Schematic Diagram

Fig. 5. Robotic group in their second isogonic formation with maximum permissible expansion. The start-to-goal path of the group is shown in red.

formation area, switching between ﬁrst and second isogonic formations, etc.) is needed. It is assumed that ﬁeld of operation is covered by a number of static obstacles that are randomly spread over the environment. The controller makes its decision for any contingent readjustment of the formation based on the position information of the previous and the next way points, with regards to current location of the group leader along the path. To avoid collision with the obstacles and in order to maintain the formation connectivity, the controller instructs the robots to ﬁrst switch into first isogonic formation, followed by shrinking to minimum permissible formation size, while performing collision avoidance maneuvers. The minimum permissible formation size takes into account the necessary interrobot safety distance requirement to avoid collision among group members. If there is no obstacle in vicinity, the robotic group is instructed to ﬁrst expand and then switch to second isogonic formation for maximum area coverage while

A Centralized Framework to Multi-robots Formation Control

95

Fig. 6. Robotic group switched to first isogonic formation followed by shrinking in order to avoid the obstacle. The start-to-goal path of the group is shown in red.

Fig. 7. Robotic group in their first isogonic formation while expanding since there is enough space available between current formation location and next obstacle to avoid. The start-to-goal path of the group is shown in red.

traversing the path. Fig. 5 through Fig. 11 demonstrates the formation preservation process by the controller during the simulation. In all the images, the straight, cyan-colored line shows the path that connects start and goal locations. The red-colored curve is the path calculated by the navigation controller. The robotic group has been presented using the coloring convention adapted in Fig. 1 i.e. green, red and blue colors are used to represent group leader, side and isogonic robots, respectively. As shown in Fig. 5, robots leave the start point while maintaining their formation based on second isogonic formation. Such formation would provide the group with maximum ﬁeld coverage while preserving the formation and connectivity among the robots. In the event of any obstacle encounter, the controller instructs the group to shrink into the formation size that is suitable for obstacle avoidance by ﬁrst switching into first isogonic formation (if the group is currently following the second isogonic formation) and then reducing the formation side length a, thereby shrinking the formation size. The group leader’s, and consequently, rest of the groups’ positioning information will be updated in each iteration of one such formation size reduction to preserve the formation and connectivity during the resizing process. As it is shown in Fig. 6, the formation size is reduced to the permissible size for maneuvering around the obstacle. Once the obstacle is avoided, the controller would decide if enough space is available for the group to expand. If positive, the expansion of the formation would take place (Fig. 7), otherwise robots would maintain their

96

S. Keshmiri and S. Payandeh

Fig. 8. Robotic group in their first isogonic formation and shrunk to avoid obstacle. The start-to-goal path of the group is shown in red.

Fig. 9. Robotic group in their first isogonic formation while maintaining their formation in shrinking mode. The start-to-goal path of the group is shown in red.

Fig. 10. Robotic group in their first isogonic formation while maintaining their formation in shrinking mode. The start-to-goal path of the group is shown in red.

Fig. 11. Robotic group in their second isogonic formation reaching to their maximum permissible expansion. The start-to-goal path of the group is shown in red.

A Centralized Framework to Multi-robots Formation Control

97

shrunk formation style until the ﬁeld is safe for switching to expansion (Fig. 8 through Fig. 10). Fig. 11 demonstrates the scenario in which all the obstacles have been avoided and the environment is safe, signaling the group to switch back into second isogonic formation.

5

Conclusion

A geometric approach to robotic agents’ group formation and connectivity preservation has been presented. The formation controller provides the robotic group with two main formation styles (determined by the ﬁrst and the second isogonic points of the formation) and demonstrates high degree of conﬁguration readjustability as well as consistency and stability while switching among diﬀerent formations. It has been shown that such approach is capable of maintaining the required formation as well as connectivity among the robotic agents in the group in a cluttered environment. Further more, some insight (through the proposition 1) with regards to scalability of the approach has been demonstrated. Despite the convincing results obtained during the experimentations conducted in simulation environment, there are still several important factors that are required to be addressed in the future. The current implementation of the controller is incapable of instructing the robotic group to perform line and column formation. In addition, introduction of moving obstacles into the ﬁeld of operation is an important issue that has not been addressed in the current implementation. Centralized property of the controller is another limitation of the present approach that requires special attention since its distributed-ness, would provide the resulting formation with higher degree of ﬂexibility and faulttolerance. Despite the scalability of the approach that has been proven at theoretical level, it would be interesting to formally deﬁne the upper limit of one such robotic group size, thereby formalizing the feasibility as well as real-life practicability of the technique.

References 1. Arkin, R.: Behavior-Based Robotics. The MIT Press, Cambridge (1998) 2. Murray, R.M.: Recent Research in Cooperative Control of Multi-Vehicle Systems. Dynamic Systems. Measurement and Control 129(5), 571–583 (2007) 3. Parker, L.E.: Designing Control Laws for Cooperative Agent Teams. In: IEEE International Conference on Robotics and Automation, pp. 582–587. IEEE Press, Los Alamitos (1993) 4. Dunbar, W.B., Murray, R.M.: Distributed Receding Horizon Control for MultiVehicle Formation Stabilization. Automatica 42(4), 549–558 (2006) 5. Leonard, N.E., Fiorilli, E.: Collective Motion: Bistability and Trajectory Tracking. In: IEEE International Conference on Decision and Control (1993) 6. Atkar, P.N., Choset, H., Rizzi, A.A.: Distributed Cooperative Control of Multiple Vehicle Formations Using Structural Potential Functions. In: 15th IFAC World Congress (2002)

98

S. Keshmiri and S. Payandeh

7. Wang, P.K.C.: Navigation Strategies for Multiple Autonomous Robots Moving in Fromation. Robotic Systems 8(2), 177–195 (1991) 8. Gage, D.W.: Command Control for Many-Robot Systems. Unmanned Systems Magazine. 10(4), 28–34 (1992) 9. Yoshida, E., Arai, T., Ota, J., Miki, T.: Eﬀect of Grouping in Local Communication System of Multiple Mobile Robots. In: IEEE International Conference on Intelligent Robots and Systems, pp. 808–815 (1994) 10. Yamaguchi, H.: Adaptive Formation Control for Distributed Autonomous Mobile Robot Groups. In: IEEE International Conference on Robotics and Automation, pp. 2300–2305 (1997) 11. Mataric, M.J.: Designing and Understanding Adaptive Group Behavior. Unmanned Systems Magazine 4(1), 50–81 (1995) 12. Sugihara, K., Suzuki, I.: Distributed Algorithms for Formation of Geometric Patterns with Many Mobile Robots. Robotic Systems 13(3), 127–139 (1995) 13. Lewis, M.A., Tan, K.H.: High Precision Fromation Control of Mobile Robots Using Virtual Structures. Autonomous Robots 4, 387–403 (1995) 14. Balch, T., Arkin, R.C.: Behavior-Based Formation Control for Multi-Robot Teams. IEEE Trans. on Rob. and Autom. 14(6), 926–939 (1998) 15. Werger, B.B.: Cooperation without Deliberation: A Minimal Behavior-Based Approach to Multi-Robot Teams. Artiﬁcial Intelligence 110, 293–320 (1999) 16. Kube, C.R., Zhang, H.: Collective robotic intelligenece. In: 2nd Conf. on Simulation of Adaptive Behavior, pp. 460–468 (1992) 17. Werger, B.B., Mataric, M.J.: Robotic ’Food’ Chains: Externalization of State and Program for Minimal-Agent Foraging, from Animals to Animats 4. In: 4th International Conference on Simulation of Adaptive Behavior, pp. 625–634 (1996) 18. Balch, T., Hybinette, M.: Social Potentials for Scalable Multirobot Formations. In: IEEE International Conference on Robotics and Automation, pp. 73–80. IEEE Press, New York (2000) 19. Lewis, M.A., Tan, K.H.: High Precision Fromation Control of Mobile Robots Using Virtual Structures. J. Autonomous Robots 4, 387–403 (1995) 20. Boltyanski, V., Martini, H., Soltan, V.: Geometric Methods and Optimization Problems. Kluwer Academic Publishers, Boston (1999) 21. Kupitz, Y., Martini, H.: Geometric Aspects of the generalized Fermat-Torricelli Problem. J. Bolyai Soc. Math. Stud. 6, 55–129 (1997) 22. Keshmiri, S., Payandeh, S.: Robot Navigation Controller: a Non-Parametric Regression Approach. In: Workshop on Intelligent Control Systems WICS 2010, Sinaia, Romania, September 29-October 02, pp. 27–32 (2010) 23. Nadaraya, E.: On Estimating Regression. Theory of Probability and Its Application. 9, 141–142 (1964)

Flexible Behaviour Regulation in Agent Based Systems Michael Luck, Lina Barakat, Jeroen Keppens, Samhar Mahmoud, Simon Miles, Nir Oren, Matthew Shaw, and Adel Taweel Department of Computer Science, King’s College London London WC2R 2LS, United Kingdom [email protected]

Abstract. Just as in human societies, for which we have developed reasonably effective systems to organise and manage interactions in such a way as to minimise the impact of erroneous or malicious behaviour, we also need to find ways to organise and manage computational entities in order to mitigate their potential deleterious effect on computational systems. In this paper, therefore, we analyse the role of trust, organisations and norms in a motivation-based view of agency that seeks to regulate behaviour, and illustrate some of these issues with aspects of several projects, including the CONTRACT project, concerned with electronic contract-based e-business systems.

1 Introduction As information and communications technologies have progressed, there has been a change of focus from individual standalone computers to large-scale interconnected and open distributed systems. In fact, to a large extent, this move has already occurred, with such interconnectedness and openness becoming increasingly prevalent. While the benefits are myriad, for example in enabling dynamic service composition and virtual organisations, and supporting developments contributing to the realisation of ambient intelligence and Grid computing, there are also some important potential problems. In particular, little consideration has been given to problems analogous to those in human societies, where we need to consider the issues surrounding the use of regulations and their absence, of opportunistic and malicious behaviour. Just as in human societies, for which we have developed reasonably effective systems to manage and organise interactions in such a way as to minimise the impact of erroneous or malicious behaviour, we also need to find ways to organise and manage computational entities in order to mitigate their potential deleterious effect on computational systems. While some work has been done on each of these concerns, their combination in large-scale open systems has not been addressed, yet this is a fundamental requirement if the visions of Grid computing and ambient intelligence, for example, are to be realised. In this paper, therefore, we focus on the need for flexible behaviour regulation for open agent-based systems, which must be designed with a focus on techniques that anticipate and respond to the potential for erroneous or malicious behaviour. This requires an ability to reason about complex overall system operation, resulting from individual agent interactions, in support of mechanisms for control and management of systems C. Guttmann, F. Dignum, and M. Georgeff (Eds.): CARE 2009/2010, LNAI 6066, pp. 99–113, 2011. c Springer-Verlag Berlin Heidelberg 2011

100

M. Luck et al.

as a whole. We argue that this, in turn, demands an understanding of how motivations, organisations, norms and trust (among others) relate to each other, and how, in combination, they may give rise to effective and efficient systems. The paper begins in Section 2 by reviewing a framework for characterising societies or systems along dimensions involved in flexible behaviour regulation.Then, in Sections 3, 4 and 5, it expands on the role of, and requirements for, organisations, norms and trust in such systems. Section 6 then introduces two distinct example applications, both driven by real-world needs, illustrating how the different aspects of the framework are relevant. Finally, the paper ends with broader conclusions.

2 Motivations, Organisations, Norms and Trust As has been articulated elsewhere [16], much of computing, especially AI, can be conceptualised as taking place at the knowledge level, with computational activity being defined in terms of what to do, or goals. In this view, an agent’s concern is taken to be the task of determining which actions to take to bring these goals about, yet the underlying motivation behind these goals, the why rather than the how, is not often considered. Despite being neglected, such motivation can have important and substantial influence over the way in which goals are achieved. As is generally accepted, goals specify what must be achieved but do not specify how; in this sense, they allow individual agents to decide for themselves how best to achieve them. Although this provides freedom in problem-solving, it provides little by way of direction or meta-level control that may be valuable in determining how best to achieve overarching aims. Motivations address this both by providing the reasons for the goal, and by offering constraints on how the goal might best be achieved when faced with alternative courses of action [15]. In this context, motivations are the starting point for considering flexible behaviour regulation, since they characterise the nature of the agents involved: at extreme points, whether they are malevolent or benign. Thus agents may be well integrated members of a system or society, cooperating with others when requested to do so, participating effectively in joint ventures, and contributing to the good of the whole. Alternatively, they may be malicious, seeking to take advantage of others, by requesting cooperation but not providing it, by taking the benefits provided by a society without contributing to its success, or they may simply be incompetent or unable to deliver. Depending on the nature of such motivations, various mechanisms may be required to ensure effective system operation. For example, in cases where there is a prevalence of benign behaviour from individual agents, resulting from their motivations, there is less risk in interacting with agents because they will generally seek to cooperate without malicious intention. There is thus less risk in trusting others, since defection (which occurs when agents renege on their agreements with others) is unlikely; indeed, agents that are unwilling to trust others may miss opportunities for cooperation because of this. Moreover, in these situations, the use of excessive regulation through strict enforcement of system or societal rules (or norms) may hinder agent interactions to such a degree that cooperation is ineffective. Here, it is most appropriate to allow free reign to the full range of behaviours with little constraint, since malicious actions are unlikely, and the effort that might be expended on introducing stricter regulation may be wasted.

Flexible Behaviour Regulation in Agent Based Systems

101

However, in cases where agents are less likely to be benign, some form of behaviour regulation is needed, either through constraints imposed by organisational structure and norms (limiting what is possible for agents to do) or through careful analysis of potential cooperation partners through an analysis of trust and reputation. (Here, trust is taken to be an individual’s direct assessment of the reliability of another based on direct interactions with it, while reputation is an indirect assessment relying on the assessment of third parties, based on their interactions.) In the former case (when constraints are imposed by the organisational structure and norms), trust may be less important, since the system is heavily regulated through strict norms and enforcement. This is characteristic of the electronic institutions approach of several researchers (e.g., [6,8]) in which agents do not have the possibility of violating norms. However, despite this, if agents are less willing to trust others, then the possibility for taking advantage of opportunities in terms of cooperation may be ruled out. In the latter case, (when constraints are imposed by placing less trust in agents with poor reputation), we have a prevalence of agents with malicious motivations but their effectiveness is curtailed because of the care taken in determining cooperation partners. Here, if there is little organisational structure and lax enforcement of norms, there is a high likelihood that agents may defect, and since there is little protection from societal or system regulation, the role of trust is vital. Typically, agents should place very little trust in others in these situations. In essence, this discussion characterises the axes of a three-dimensional space, described in [16], and illustrated in Figure 1, that identifies different types of systems or societies. More specifically, the x-axis represents motivations, with an increase in the value of x representing a prevalence of malicious motivations, indicating that agents are more likely to defect if they see more utility in alternative interactions. The y-axis represents organisational constraints, norms, and their enforcement, with an increase in value indicating the prevalence of stricter organisational structure, norms and enforcement. This can constrain the motivations of agents and prevent them from acting maliciously if they intend to do so. Finally, the z-axis represents trust, with an increase indicating an increase in the trust that agents place in others and, therefore, an increase in willingness to cooperate with others. In the figure, eight points in the space (at the vertices) are labelled by a circled letter, indicating different types of society or system. As indicated above, societies A, B, E and F largely involve agents with benign motivations, so that the levels of organisational structure, norm enforcement and trust are less important, but societies C, D, G and H are more interesting. For example, C involves agents with malicious motivations, a lack of trust and lax organisational constraints, so that it is likely to be very inefficient. Society G uses strict organisational constraints to mitigate against these malicious motivations, society D uses a lack of trust to avoid potentially problematic interactions, and society H uses both. Each of these cases has been discussed in more detail above, but we have not yet provided a substantial consideration of the different axes of this space, which we do next.

3 Organisations As computational systems increasingly comprise many tens or even hundreds of interconnected and interacting components, whether they be in a single physical location or

102

M. Luck et al.

F

Strict organisations, norms and enforcement

E H

G B Lax organisations, norms and A enforcement Benign motivations

D Reliable partners Malicious motivations

C

Unreliable partners

Fig. 1. The interplay between norms, motivations and trust (adapted from [16])

distributed across geographically diverse areas, they are increasingly difficult to manage in a manual fashion. One approach to overcoming these difficulties lies in the application of organisational models to structure the interactions between the components. Such organisations can be as simple as a hierarchy of authority or as complex as a society containing sub-society structures. One of the most basic forms of organisational structure is a hierarchy [9], which naturally reflects structured relationships of authority or other subordinate relationships, for example. Hierarchies are very simple organisations, and are commonly applied in many fields because of their rigid structure and ease of understanding and use. In their most basic form, hierarchies are similar to trees, or directed graphs, in which there are nodes and edges connecting these nodes, but no cycles. Here, each node has an arbitrary number of child nodes connected to it via edges. Although the obvious way to structure a hierarchy is by means of authority (with managers and employees or contractors in the business domain, and with commanders, sub-commanders and regular troops in the military), in fact this attribute could be something entirely different. In particular, it is this, whatever it may be, that determines the ordering of the hierarchy. For example, instead of authority, we might use time of execution as the ordering attribute in the case of a workflow hierarchy (e.g., [11]), which is a decomposition of a task into smaller sub-tasks, each node being a different sub-task. Such a hierarchy is needed because some of the sub-tasks may need to be executed before others can be started. Taking Figure 2 as an illustration of this type of hierarchy, if sub-task A must be executed before sub-task B, then A is the parent node of B. If sub-task C also cannot be executed until A is completed but it does not matter if B and C are executed sequentially or in parallel, then they may be siblings (that is to say that they are both children of the same parent). The direction of the edges shows the order in which the tasks must be completed, so in our particular application the root node must be executed first, followed by its children, and so on.

Flexible Behaviour Regulation in Agent Based Systems

103

$

&

%

'

Fig. 2. A simple hierarchy

Similarly, as indicated above, we can have hierarchies that represent the authority of some entities over others, for example in the case of an employer’s authority over an employee or a sergeant’s authority over a private in the military. In this case, therefore, assuming the same graph from Figure 2, the direction of the edges reflects the order of importance or seniority in the hierarchy. Importantly, such structures may provide constraints over interaction or communication. For example, one node in a hierarchy can only communicate with, and can only receive messages from, nodes to which it is connected. In Figure 2, node A can send and receive messages to and from node B, and vice versa. Conversely, A is not permitted to send and receive direct messages to node D, since they are not directly connected, and all communication must take place through the intermediary, node C. Such structures can also aid control of a system. For example, in our hierarchy, the apex (root) node is the ultimate controller of the system, delegating tasks down the hierarchy. Nodes lower in the hierarchy send data produced lower down the hierarchy to inform subsequent decisions about the control of the system. In this way, the nodes in the system act as filters for communication: as data is passed up the hierarchy, each agent filters the data that is propagated upwards to ensure that no single node is overwhelmed with information. When decisions need to be made concerning information passed up the hierarchy, they are made at the lowest level possible, by the node that knows enough to be able to make the decision and has sufficient authority to execute it. In this way, hierarchies provide simple but effective structures with clear and simple protocols for communication. Perhaps more interesting than simple static structures, however, are those that can change or adapt in response to prevailing circumstances. The goal of self-organising systems is to automatically form new organisation structures in situations when the existing structure is no longer optimal. However, systems of the complexity we are considering, comprising multiple interacting entities, can sometimes produce complex and unexpected global behaviour. This phenomenon is known as emergence, which refers to a novel macro-level situation that arises dynamically from the interactions between parts of the system at the micro-level. Here, the behaviours of the individual parts of the system can be fully specified or observed, but the interactions between these parts give rise to an unexpected and unpredictable macro-level behaviour.

104

M. Luck et al.

4 Norms More generally, the notion of a society can be taken to cover a group of individual agents, bound together in some fashion, through the adoption of an organisational structure, or rules to provide some structure arising from adherence to them. Such rules (also labelled conventions, social laws or norms) impose constraints on a population so that agents know both how they should act and what behaviour to expect from others. However, they are only valuable if used effectively; Fitoussi and Tennenholtz [7], for example, suggests that the balance between individual objectives and norms is critical, because norms must be sufficiently restrictive to have the desired effect for which they are applied, but must also be sufficiently flexible so that all objectives are equally feasible. Along these lines, L´opez y L´opez and Luck [13] provide a more specific definition of norms within their normative framework. According to them, norms facilitate mechanisms to drive the behaviour of agents, especially in those cases where their behaviour affects other agents. Norms can be characterised by their prescriptiveness, sociality, and social pressure. In other words, a norm tells an agent how to behave (prescriptiveness) in situations where more than one agent is involved (sociality) and since it is expected that norms may often conflict with the personal interests of some agents, socially acceptable mechanisms to force agents to comply with norms are needed (social pressure). 4.1 Norm Enforcement Since agents are autonomous, compliance with norms is not guaranteed, but violation, or non-compliance, of norms, can have negative effects on a society as a whole. As a result, there is a need to provide some means of encouraging compliance, which is typically achieved through the use of sanctions. In this view, the potential for sanctions to be imposed on a norm violator can be seen as norm enforcement, since agents must take into consideration the possibility of receiving some punishment if they violate norms. The issue of using sanctions to enforce norms has already been addressed, for example, by L´opez y L´opez et al. [14], who define the notion of interlocking norms. Here, two norms are interlocking if satisfying or violating one triggers the activation of the other. The first norm is the primary norm, and the second is the secondary norm. L´opez y L´opez et al. suggest applying the notion of interlocking norms to norm enforcement by specifying that reward or punishment norms are considered as secondary norms for the primary norms that the rewards or punishments are assigned to. In this view, de Pinninck et al. [5] suggest the use of sanctions as a means of discouraging norm violation in Gnutella, a peer-to-peer file sharing application. Here, the system operates by different peers searching for files hosted by others and downloading them, and relies on the assumption that peers will both share and download. However, some may join the network and download files without contributing to the society by not sharing their own files. To prevent this problem of allowing peers to consume services without providing any of their own, de Pinninck et al. suggest adding a norm specifying that any agent that needs to download a file should share some files with others. If agents violate this norm, the violator is ostracised so that no other member of the society interacts with it and, as a result, it is denied access to the network’s resources. In this

Flexible Behaviour Regulation in Agent Based Systems

105

case, ostracising the violator is accomplished through the spread of negative reputation across the network. 4.2 Norm Emergence From the description of de Pinninck’s model, it is not clear if the norm itself is explicitly represented or merely implicit in the behaviour of agents as a result of attempting to generate the desired results. Along similar lines, Axelrod [3] proposes a game and undertakes experiments to illuminate the process of norm establishment (and norm collapse), when a norm arises through the guided behaviour of a group of agents. In his norms game and metanorms game, punishments are applied to agents who do not comply with norms, thus reducing their utility. However, because these punishments alone turn out not to be adequate, metanorms are introduced as secondary norms that help to enforce compliance of primary norms by applying punishments to agents that do not punish norm violators. Having integrated metanorms in his experiments, the results show that all runs end with a population that always complies with norms, so that norms always emerge when metanorms are used. 4.3 Norm Processing In addition to these aspects, there are several other processes involved in the use of norms in open systems. Here, for completeness, we briefly outline the most important, including norm recognition, norm adoption and decision making. Norm recognition refers to the process by which an agent identifies a norm, or whether what might be taken to be a norm is, in fact, some other form of social input. For example, Andrighetto et al. [1] claim that the norm recogniser plays a very important role within the agent reasoning cycle in their EMIL - A architecture. When an agent recognises a new norm, it accepts it if it believes the norm concerns, or is directed towards, its behaviour. This is known as norm adoption. Conte et al. [4] state that an agent adopts a norm only if it believes that this norm helps, either in a direct or indirect way, to achieve one of the agent’s goals. Based on this, the agent forms a normative goal that results from its decision to adopt the norm, but it does not make the decision to comply with this norm. Decision making is a critical phase of normative reasoning, as an agent decides within this phase if it is going to comply with a norm. Whatever the decision, it might have a major impact on behaviour. If the agent complies with the norm, some of its goals might conflict with the norm and, as a result, the agent will not be able to achieve any of these conflicting goals. On the other hand, if the agent violates the norm, then some punishments may be applied to the agent, which in turn can affect the achievement of some of its goals.

5 Trust The use of norms in open computational systems can be extremely valuable, but without appropriate mechanisms to encourage compliance with them, they can become useless. This raises two important issues. The first, which is concerned with the nature of regulation in a society — how agents take decisions about norm compliance, and efforts

106

M. Luck et al.

to encourage compliance, through enforcement and the severity of sanctions — is one we have largely outlined above. The second, which relies on the general levels of compliance in a society — in determining to what extent agents should be concerned that others may cooperate with them or defect (either because agents are more willing to defect in pursuit of more utility somewhere else or because there is uncertainty about whether agents can achieve the task) — requires a consideration of the third dimension of behaviour regulation, trust and reputation. In particular, open multi-agent systems consist of a large number of interacting autonomous agents; each may have its own goals and may act to maximise its own benefit. Thus, in such environments, there is a challenge for agents to choose the most reliable interaction partner among many possible available. To cope with this challenge, many trust and reputation models have been introduced, enabling agents to calculate the trustworthiness of their potential partners, and then to choose the most trustworthy partner to interact with. By doing so, agents can maximise the chance that the interaction will achieve its potential benefits. For example, TRAVOS [23] is a trust and reputation model for agent-based virtual organisations that computes the level of trust an agent (the truster) has in another agent (the trustee). More precisely, the trustworthiness of the trustee from the truster’s perspective is the expected probability that the former will fulfil its obligations towards the latter in an interaction. The estimation of this expected probability is based on the outcomes of the previous direct interactions of the truster with the trustee, with each interaction being evaluated as either successful (the trustee fulfilled the agreed obligations) or unsuccessful (the trustee did not do so). Each calculated trust value is associated with a confidence level, which increases with the increased number of the observed outcomes. When the confidence in the calculated trust value is below a pre-specified threshold, which means that there is a lack of personal experience between the evaluating agent and the agent being evaluated, the former depends on the latter’s reputation to evaluate its trustworthiness. This reputation is obtained by combining the experiences (provided as outcome observations) the other agents had with the trustee and, as a result, the evaluating agent will have a larger number of observations with which to assess the trustee. Moreover, to cope with the problem of inaccurate third-party experiences (provided, for example, by misleading reputation sources), the truster estimates the reliability of each reputation source depending on the accuracy of its past opinions, and then uses this reliability value to decrease the role of unreliable opinions in the calculation of the trustee’s reputation.

6 Case Studies Given the above view of how we can understand the need for behaviour regulation in open agent systems, and the analysis of the relevant characteristics or dimensions that make up the space of such systems, we can move to consider how they are relevant in real cases. In this section, therefore, we introduce two case studies, one addressing a fully developed system, and the other addressing an application that would benefit from an analysis in terms of the concepts described in this paper. The first is concerned with

Flexible Behaviour Regulation in Agent Based Systems

107

the need to provide contract-enabled e-business systems, to provide guarantees over service delivery between partners, based on several different motivating use cases [10], and illustrating the value of flexible behaviour regulation in normative settings. The second is focussed on a healthcare application for linking primary care providers to clinical researchers, showing the need for trust mechanisms and robust organisations in this domain. 6.1 Electronic Contracting The CONTRACT project, created to explore multiple aspects of contract-based systems, developed a comprehensive framework for the creation, management [19] and monitoring [18] of electronic contracts. Here, electronic contracts are viewed as comprising sets of norms, i.e. statements of expected behaviour, usually expressed in terms of deontic concepts such as obligations, permissions and prohibitions. Development of the framework was informed by a series of case studies concerned with insurance settlement, aerospace aftermarkets [17] and certification testing, described by Jakob et al. [10]. The CONTRACT framework is a conceptual model for specifying applications using electronic contracting. The general architecture that was built using this framework provides an instantiation of relevant aspects of contract administration, through serviceoriented middleware and multi-agent design patterns. Figure 3 illustrates the overall structure of the framework and architecture, which can be seen as a set of models and specifications, and comprising a methodology for adapting application designs to utilise electronic contracts. The primary component is the framework, at the top, which is the conceptual structure used to describe a contract-based system, including the contracts themselves and the agents to which they apply. Each level in the figure provides support for the components below it. Arrows indicate where one model influences or provides input to another. Given a particular application, its framework specification enables other important information to be derived. For example, off-line verification mechanisms can check whether the contracts to be established satisfy particular properties, such as whether they can be achieved given the possible reachable world states. In turn, those states that are critical to observe during execution to ensure appropriate behaviour can be determined. In this context, critical states indicate whether, for example, an obligation is fulfilled or fulfillable, (e.g., achieved, failed, in danger of not being fulfilled, etc.). A state-based description, along with the deontic and epistemic implications of the specified contracts, can then be used to verify a system either off-line, using a modelchecking approach [12], or with run-time monitoring [18]. Using the CONTRACT framework as a starting point, it is then possible to determine suitable processes for administration of the electronic contracts, including establishment, updating, termination, renewal, and so on. Such processes may include observation of the system, so that contractual obligations can be enforced or otherwise effectively managed, and these processes depend on the critical states identified above. Once suitable application processes are identified, we can also specify the roles that agents play within them, the components that should be part of agents to allow them to manage their contracts, and the contract documents themselves. An XML-based

108

M. Luck et al.

Framework (applied to application)

Architecture

C Critical ritical ap n application s states

Agent a architecture c component erfaces interfaces

Contract ontract ad administration processes

Contra ra act Contract administration roles

C Contract documents

Supporting Methodology

V Verifi Verification ification ti mechanisms

T ch o d lo Technology-specific deployment

Fig. 3. The CONTRACT framework

language was developed to provide a concrete realisation of contract specifications using the framework, allowing contracts to be communicated and negotiated over [20]. All this provides the basic infrastructure required for establishing flexible behaviour regulation among agents or service providers in open systems, through a normative framework for agreeing contracts that impose structural relationships on the agents in the system, and through mechanisms for providing guarantees over their behaviour. In this context, their is an implicit suggestion that agents may not be benign, which is why contractual agreements are required. Contracts. More specifically, the agreements between agents are formally described in electronic contracts, which document obligations, permissions and prohibitions (collectively clauses) on agents. Agents bound by contract clauses are said to be contract parties, and a contract specifies contract roles, which are fulfilled by contract parties, so that clauses apply to specific contract roles. Importantly, each of these clauses represents a norm, which can be interpreted here as socially derived prescriptions specifying that some set of agents (the norms targets) may, or must, or may not, perform some action, or see that some state of affairs occurs. As discussed earlier, norms can be understood as regulating the behaviour of agents: this is their role when encoded in contracts. As indicated earlier, norms are social constructs, and it is meaningless to consider norms independently of their social aspect. This is because norms are imposed on the target by some other entity (the imposer) which must be granted, via the society, some power to impose the norm. Without this power, the norms target is free to ignore the norms prescriptions. With the presence of this power, a penalty may be imposed on an agent violating a norm. These penalties take the form of additional norms, giving

Flexible Behaviour Regulation in Agent Based Systems

109

certain agents within a society permission to impose penalties (or obliging them to do so). This provides the means to establish the strict regulation that may be required without guarantees of benign behaviour. We assume that a contract is also made up of various descriptive elements, for example, stating which ontologies may be used to explain the terms found within it. Since norms may have normative force only in certain situations, we associate norms with an activation condition. Norms are thus typically abstract, and are instantiated when the norms activation condition holds. Once a norm has been instantiated, it remains active, irrespective of its activation condition, until a specific expiration condition holds. When it occurs, the norm is assumed to no longer have normative force. Finally, independent of these two conditions is the norm’s normative goal, which is used to identify when the norm is violated (in the case of an obligation), or what the agent is actually allowed to do (in the case of a permission). Obligations and permissions are the two norm types on which our framework focuses. Like others (e.g., [24]), we assume that additional norm types may be constructed from these basic types (e.g. a prohibition could be seen as an obligation with a negated normative goal). Norms may be activated, satisfied and discharged based on a number of factors including the status of other norms, the state of the environment (and the actions performed by other agents therein), and the status of contracts. Contract Parties. Contracts in our system are agreed by agents, which are assumed to be autonomous, pro-active, flexible (decision-making) and social, and agents engage in contract-directed interactions to fulfil the clauses specified in a contract. Contract interactions require a minimum of two agents fulfilling the role of participants. Some applications may require contract-related processes to have certain properties, e.g. that violations are acted on, or that the integrity of the contract documents is maintained. These requirements lead to obligations on (and the creation and use of) administrative parties, and contracts may document their required behaviour. We thus have two distinct kinds of contract parties. – Business Contract Party Agents for whom the contract is created: the obligations on the business contract parties are largely concerned with the business of the application. – Administrative Contract Party Agents are required to ensure that the contract is accessible, retains integrity and legitimacy, is monitored and enforced, and other such administrative functions that ensure the contract has force. The obligations on these agents relate to their administrative roles. Enforcement. Two particular administrative contract party roles are those of observer and manager [17]. The former detects whether the system enters a critical state (success, violation, in danger of violation) with regard to a particular clause. A manager reacts on the basis of observation, e.g. to inform a user of the problem, penalise a contract party in some way, and so on. There may be several observers and managers for an application, for example checking compliance on behalf of different users, and handling violations in different ways. One of the important aspects of the CONTRACT architecture in ensuring and encouraging compliance with a contract is the process of run-time monitoring [18]. Here,

110

M. Luck et al.

the key information available to third parties for this purpose consists of the messages exchanged between agents, and these messages are first gathered by observer agents. Monitors then receive the observations from observers that are explicitly entrusted by all contract parties to accurately report on the state of the world, and determine their status. The use of trusted observers ensures some degree of certainty that a norm is reported as violated if and only if it has in actuality been violated, and thus provides assurance that sanctions are only applied as and when appropriate. Once the status of a norm is ascertained through the monitoring process, the decision of what actions are to be taken is delegated to manager agents, which might apply sanctions for violations and rewards for fulfilment, as appropriate. 6.2 ePCRN The ePCRN (electronic Primary Care Research Network)1 is an infrastructure project that seeks to connect healthcare (primary care) with clinical research by facilitating the management of primary care clinical trials [2,22]. Clinical research is a vital resource for continually improving healthcare [21]. The issues involved relate to patient privacy and confidentiality, access to patient data across geographically distributed primary care clinics, heterogeneity of electronic health record (EHR) systems and interfaces, and so on, providing significant syntactic and semantic interoperability problems. The idea is that clinical researchers should be able to generate interoperable queries that can be submitted to all available clinics within the ePCRN network by adopting a Grid-based framework for distributed information access. Clinical patient information is stored in EHRs in individual clinics or repositories (the record from several clinics combined into one repository), with other information potentially coming directly from hospital information systems (e.g., laboratory results, prescriptions, etc.). Each may have its own data format and interface. Researchers undertaking clinical trials need to find eligible patients by searching through this data, but preserving confidentiality and anonymity. As a result, they do not access the data directly at this point, and send requests to local clinics to invite patients to participate in studies, with only the local clinic being able to identify patients. Once patients are recruited, data for each participant is collected; if the EHR data is needed, it can be provided at this point, via secure or authorised access mechanisms. Finally, when the results of the trials are available, relevant data may be fed back into the patient record, but without repudiating the original unmodified record. A key challenge here is to enable access to patient data and maintain patient confidentiality; this requires a robust yet flexible confidentiality and security framework that enables healthcare providers to control their data sharing policy and benefit from clinical research output. Importantly, such a framework must both conform to organisational data and process flow and enforce system and local policy measures. In the context of the three-dimensional space, it is clear that the organisational structure of, the relationships between, and the rules governing the researchers, the primary clinics, the practitioners and the patients, imposes significant constraints on what is possible. This is not simply a matter of efficiency and effectiveness, but is a requirement 1

http://www.epcrn.bham.ac,uk

Flexible Behaviour Regulation in Agent Based Systems

111

placed on the handling of data both legally and ethically. From a different perspective, the individuals involved must have strong trust in others to ensure that data is correct, that data will not be mis-used, and that patient records will not be modified inappropriately. At the same time, since all parties can be assumed to have benign motivations, and since the system is designed to impose the organisational constraints described, the levels of trust required are likely to be relatively high. Nevertheless, the relationship between the different axes described earlier in the paper should be clear.

7 Conclusions In open agent systems comprising multiple autonomous agents, agent behaviour cannot be guaranteed, resulting in the possibility that overall system operation may be ineffective or inefficient. For this reason, there has been an increasing amount of research devoted to establishing appropriate mechanisms to encourage or enforce socially acceptable (and even valuable) behaviour on the part of individual agents. Drawing on analogous mechanisms in human societies, such mechanisms include those to establish, monitor and manage trust among agents that need to work together, based on experience of their previous interactions. However, in the absence of strong trust relationships to enable effective interaction in dynamic and open environments, some form of societal regulation must also be considered. In this respect, organisations and norms have been proposed as a means of mitigating the consequences of action when these trust relationships are absent. In relation to these concerns, this paper has sought to elaborate on an earlier framework for characterising and contrasting different types of systems and societies as a means for understanding the requirements for behaviour regulation of the component agents. It considered systems with a prevalence of stricter organisations, norms and enforcement, those with agents inclined to behave maliciously, and those in which trust in agents is higher or lower. Given this analysis, the different dimensions of the framework were considered in more detail, and illustrated in relation to two distinct realworld case studies in which the issues raised are important factors in successful system design and development. The key message is that while the identified characteristics and techniques have been considered across a range of different research efforts, the relationship between them must also be analysed and understood in order to determine which techniques are appropriate in different circumstances.

References 1. Andrighetto, G., Campenni, M., Conte, R., Paolucci, M.: On the immergence of norms: a normative agent architecture. In: Proceedings of the AAAI Symposium on Social and Organizational Aspects of Intelligence (2007) 2. Arvanitis, T.N., Taweel, A., Zhao, L., Delaney, B.C., Peterson, K.A., Speedie, S.M., Sim, I., Weissman, J., Fontaine, P., Lange, C., Janowiec, M., Stone, J.: Supporting e-trials over distributed networks: A tool for capturing randomised control trials (RCT) eligibility criteria using the national cancer institutes (NCI) enterprise vocabulary services (EVS). Technology and Health Care 15(5), 298–299 (2007)

112

M. Luck et al.

3. Axelrod, R.: An evolutionary approach to norms. The American Political Science Review 80(4), 1095–1111 (1986) 4. Conte, R., Castelfranchi, C., Dignum, F.P.M.: Autonomous norm acceptance. In: Papadimitriou, C., Singh, M.P., M¨uller, J.P. (eds.) ATAL 1998. LNCS (LNAI), vol. 1555, pp. 99–112. Springer, Heidelberg (1999) 5. Perreau de Pinninck, A., Sierra, C., Schorlemmer, W.M.: Distributed norm enforcement: Ostracism in open multi-agent systems. In: Casanovas, P., Sartor, G., Casellas, N., Rubino, R. (eds.) Computable Models of the Law. LNCS (LNAI), vol. 4884, pp. 275–290. Springer, Heidelberg (2008) 6. Esteva, M., Rodr´ıguez-Aguilar, J.-A., Sierra, C., Garcia, P., Arcos, J.-L.: On the formal specification of electronic institutions. In: Sierra, C., Dignum, F.P.M. (eds.) AgentLink 2000. LNCS (LNAI), vol. 1991, pp. 126–147. Springer, Heidelberg (2001) 7. Fitoussi, D., Tennenholtz, M.: Choosing social laws for multi-agent systems: Minimality and simplicity. Artificial Intelligence 119(1-2), 61–101 (2000) 8. Garc´ıa-Camino, A., Noriega, P., Rodr´ıguez-Aguilar, J.A.: Implementing norms in electronic institutions. In: Thompson, S., Pechoucek, M., Steiner, D. (eds.) Proceedings of the Fourth International Joint Conference on Autonomous Agents and Multiagent Systems, pp. 667– 673. ACM Press, New York (2005) 9. Horling, B., Lesser, V.: A survey of multi-agent organizational paradigms. The Knowledge Engineering Review 19(4), 281–316 (2005) 10. Jakob, M., Pechoucek, M., Ch´abera, J., Miles, S., Luck, M., Oren, N., Kollingbaum, M., Holt, C., Vazquez-Salceda, J., Storms, P., Dehn, M.: Case studies for contract-based systems. In: Proceedings of the 7th International Conference on Autonomous Agents and Multiagent Systems, pp. 55–62 (2008) 11. Kota, R., Gibbins, N., Jennings, N.: Self-organising agent organisations. In: Proceedings of the Eighth International Conference on Autonomous Agents and Multiagent Systems, pp. 797–804 (2009) 12. Lomuscio, A., Qu, H., Solanki, M.: Towards verifying contract regulated service composition. In: Proceedings of the 8th International Conference on Web Services (ICWS 2008), Beijing, China, pp. 255–261 (2008) 13. L´opez y L´opez, F., Luck, M.: Modelling norms for autonomous agents. In: Ch´avez, E., Favela, J., Mej´ıa, M., Oliart, A. (eds.) Proceedings of The Fourth Mexican Conference on Computer Science, pp. 238–245. IEEE Computer Society, Los Alamitos (2003) 14. L´opez y L´opez, F., Luck, M., d’Inverno, M.: A normative framework for agent-based systems. Computational and Mathematical Organization Theory 12(2-3), 227–250 (2006) 15. Luck, M., d’Inverno, M.: Motivated behaviour for goal adoption. In: Zhang, C., Lukose, D. (eds.) DAI 1998. LNCS (LNAI), vol. 1544, pp. 58–73. Springer, Heidelberg (1998) 16. Luck, M., Munroe, S., Lopez y Lopez, F., Ashri, R.: Trust and norms for interaction. In: Proceedings of the IEEE International Conference on Systems, Man & Cybernetics, pp. 1944– 1949. IEEE, Los Alamitos (2004) 17. Meneguzzi, F.R., Miles, S., Luck, M., Holt, C., Smith, M., Oren, N., Faci, N., Kollingbaum, M., Modgil, S.: Electronic contracting in aircraft aftercare: A case study. In: Berger, M., Burg, B., Nishiyama, S. (eds.) Proceedings of the Seventh International Joint Conference on Autonomous Agents and Multiagent Systems, Industry and Applications Track, pp. 63–70 (2008) 18. Modgil, S., Faci, N., Meneguzzi, F., Oren, N., Miles, S., Luck, M.: A Framework for Monitoring Agent-Based Normative Systems. In: Proceedings of the 8th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2009), Budapest, Hungary, pp. 153–160. IFAAMAS (May 2009)

Flexible Behaviour Regulation in Agent Based Systems

113

19. Oren, N., Panagiotidi, S., V´azquez-Salceda, J., Modgil, S., Luck, M., Miles, S.: Towards a formalisation of electronic contracting environments. In: H¨ubner, J.F., Matson, E., Boissier, O., Dignum, V. (eds.) COIN@AAMAS 2008. LNCS, vol. 5428, pp. 156–171. Springer, Heidelberg (2009) 20. Panagiotidi, S., Vazquez-Salceda, J., Alvarez-Napagao, S., Ortega-Martorell, S., Willmott, S., Confalonieri, R., Storms, P.: Intelligent contracting agents language. In: Behaviour Regulation in MAS, AISB 2008 Convention Communication, Interaction and Social Intelligence, pp. 49–55 (2008) 21. Sim, I., Olasov, B., Carini, S.: An ontology of randomized controlled trials for evidencebased practice: content specification and evaluation using the competency decomposition method. Journal of Biomedical Informatics 37(2), 108–119 (2004) 22. Speedie, S.M., Taweel, A., Sim, I., Arvanitis, T., Delaney, B., Peterson, K.: The primary care research object model (PCROM): A computable information model for practice-based primary care research. Journal of the American Medical Informatics Association 15(5), 661–670 (2008) 23. Teacy, W.T.L., Patel, J., Jennings, N.R., Luck, M.: Travos: Trust and reputation in the context of inaccurate information sources. Autonomous Agents and Multi-Agent Systems 12(2), 183–198 (2006) 24. von Wright, G.H.: Deontic logic. Mind 60, 1–15 (1951)

Eﬀect of Network Topology in Opinion Formation Models Brenton J. Prettejohn and Mark D. McDonnell Institute for Telecommunications Research University of South Australia Mawson Lakes, SA 5095, Australia {Brenton.Prettejohn,mark.mcdonnell}@unisa.edu.au

Abstract. Simulations of consensus formation in networks of interacting agents have recently demonstrated that convergence to a small number of opinion clusters is more likely when the network is adaptive rather than static. In order to better model realistic social networks, we have extended an existing model of such a process, by the introduction of a parameter representing each agent’s level of ‘authority,’ based on their opinion relative to the overall opinion distribution. Here we aim to determine the importance of initial network topology for opinion formation in this model, using two distinct initial network topologies: an Erd˝ osR´enyi random network, and the Watts-Strogatz small-world network. It is shown that marked diﬀerences exist in statistics of the model after opinion convergence. These include the number of interactions between agents needed to reach consensus, as well as a clear inﬂuence of opinion tolerance on the network’s clustering coeﬃcient, mean shortest path, and degree distribution after convergence. This latter eﬀect suggests some interesting possibilities regarding the topology of ‘converged’ networks. Keywords: Opinion Formation, Consensus Formation, Adaptive Network, Social Network Analysis, Agent-based Model.

1

Introduction

Recent sociophysics simulations, aimed at investigating opinion formation and consensus in terms of the emergence of global phenomena based solely on local interactions, have illustrated the importance of adaptability in the connections between agents in social networks [1,2]. In this work a random graph network [3] is used to simulate interactions between individuals, to map the diﬀerence between social groups that are able to dynamically reform, and those which are not. As described in Section 2, nodes in the network represent agents and edges in the network represent a relationship between pairs of agents who are neighbours that may potentially communicate. Each agent has a continuously valued opinion that can be altered after an interaction between pairs of neighbours, provided the two neighbours’ opinions are within some tolerance. Investigations within this model included varying population sizes, as well as levels of adaptability, i.e. the likelihood of an agent cutting a link to a neighbour and creating a C. Guttmann, F. Dignum, and M. Georgeﬀ (Eds.): CARE 2009/2010, LNAI 6066, pp. 114–124, 2011. c Springer-Verlag Berlin Heidelberg 2011

Eﬀect of Network Topology in Opinion Formation Models

115

new one, as well as the number of interactions required to achieve the network’s ﬁnal state [1]. Simulation results indicated that global consensus can only be formed in a static model when agents have relatively high tolerance for opinion diﬀerence, whereas when the ability for nodes to break old relationships and form new ones is introduced, consensus with relatively small tolerances becomes likely. Consideration of this model from a sociological view raised two interesting questions. Firstly, the aﬀect on consensus of a more realistic ‘variable’ opinionconvergence factor, and secondly that of external determinants such as perceived authority disparity between nodes. In considering the ﬁrst point, there is general acceptance [4,5] that when a consensus is reached between individuals (or indeed groups), as often as not that consensus is not the result of an absolute meeting of the minds. It is rather a working agreement, which originates from, as well as facilitating, a more subtle convergence of actual opinion. These adaptations were shown to provide notable, and arguably more realistic outcomes for opinion convergence modeling [6]. A further factor examined in this paper is the aﬀect of network topology on the process of convergence, as well as any eﬀect that process has on topology. Precisely how to model local and universal social networks has been a topic of research for some time. From Milgram, in the 60s [7], through Strogatz and Watts’ seminal work [8] and to this day, it has generally been agreed that social networks are not entirely unstructured by nature. Exactly what form such networks take, and precisely how the formation paradigm can be deﬁned, is as yet not clear. The original topology in [1] (described in detail in the next section) was an Erd˝ os-R´enyi network model, which produces an artiﬁcial random network, in the sense that all nodes are likely to have the same number of neighbours, and there is no local clustering. In this paper we compare and contrast consensus formation in both the original Erd˝ os-R´enyi and the Watts-Strogatz [8] network models, using their network creation methods. We note diﬀerences in simulation data for each, both during the process of opinion convergence, and in the ﬁnal state of the network after convergence. Additionally we consider the eﬀect of initial opinion distribution. In the previous model opinion was uniformly distributed. Here we compare this with a more polarized opinion distribution, whereby opinions are initially distributed with a preference for either extreme. It is assumed that this model will better represent more divisive issues faced by the population. With the inclusion of these factors, the model considered has potentially greater scope for usefulness in a broad range of applications. This might include task allocation problems, where it has been shown that employment of an always optimistic method of estimating team eﬃciency will still result in an optimal solution for task assignment [9]. Another example is semi-autonomous sensor arrays with adaptable positioning and team membership capability, where tight cordon array patterns could enhance target acquisition and lock.

116

2

B.J. Prettejohn and M.D. McDonnell

Model Formulation and Extensions

The consensus formation model of [1] is predicated on a simple and virtually axiomatic principle. If two agents (network nodes) each have opinions that diﬀer only within a tolerance that would make communication productive, and they are oﬀered the opportunity to communicate, the disparity between the opinions will be reduced. Several variables are integral to the model of [1], which is initially an Erd˝ osR´enyi random graph. We assume the network has size N , and initially has average degree k¯ = 10. In each interaction, a randomly chosen pair of neighbours is selected. After the t-th interaction of the entire network, agent i has opinion, o(i, t) (i ∈ 1, .., N ), which is an initially uniformly random value between 0 and 1, representing each node’s opinion. In this paper we consider only a single opinion. Tolerance, d, is an arbitrarily set value used to determine if opinions are close enough to make communication between neighbours productive, or far enough apart to require dynamically altering the network, by breaking a link between those two neighbours, and the formation of a new one. Communication factor w is an arbitrarily chosen constant between 0 and 1, such that with probability w there will be an opportunity for communication between nodes in any given interaction and with probability 1 − w the connection will be broken and a new one created, if the opinion diﬀerence is larger than d. In [1] the opinions of interacting nodes i and j update whenever |o(i, t) − o(j, t)| ≤ d according to o(i, t + 1) = o(i, t) + μ [o(j, t) − o(i, t)] o(j, t + 1) = o(j, t) − μ [o(j, t) − o(i, t)] ,

(1)

where μ is an opinion-convergence factor. Interactions continue until the ﬁnal state of the network is reached, which is deﬁned as the point at which further interactions will not result in any change to the current status of the network. Throughout [1], the convergence factor is set at μ = 0.5, and therefore a complete local agreement is reached every time two interacting agents alter their opinions. When discussing this approach, [1] make it clear that use of this complete agreement model, rather than one which makes room for variable convergence, was chosen primarily to simplify the process. However as mentioned in the introduction, we aim to adapt the model to be more analogous to ‘real life’, by employing a randomly chosen value μ ∈ [0, 0.5]. While the adoption of a variable opinion-convergence factor provides a further measure of realism, it is argued that this does not go far enough, as it only provides a representation of internal sociological factors. As far back as Aristotle, it has been recognised that there are three modes of persuasion: pathos (appealing to the target’s emotions), logos (applying a logical argument), and ethos (aﬀect of the ‘persuader’s’ perceived authority) [10]. Indeed [11] argues that all three are present in any eﬀective form of persuasive discourse. The ﬁrst two are catered for in the model of [1], as it is argued that they can be considered internal in nature, provided a random opinion-convergence factor is used, but not the eﬀect

Eﬀect of Network Topology in Opinion Formation Models

117

of authority, which is a determinant derived, at least in part, from external factors. There is a considerable body of evidence to support the argument that the relative authority of the agents that are communicating would have an impact on the convergence of opinions. In order to model this, we introduce a new variable to represent the ‘authority level’ of the i–th node, a(i, t) ∈ [0, 1]—see [6] for more details on how a(i, t) is calculated. Authority is determined by the relative popularity of a node’s opinion within its peer group, and the size of that peer group in relation to the total population [6]. The magnitude of the diﬀerence in authority between two communicating nodes, |Ai,j | := |a(i, t) − a(j, t)| ∈ [0, 1], is used to skew the altered opinions toward the more authoritative node’s opinion, via the following extension of Eqn. (1). When two neighbours interact, they alter their opinions only if |o(i, t) − o(j, t)| ≤ d + |Ai,j /3|, in which case o(i, t + 1) = o(i, t) + μ(1 − Ai,j ) [o(j, t) − o(i, t)] o(j, t + 1) = o(j, t) − μ(1 + Ai,j ) [o(j, t) − o(i, t)] .

(2)

In our deﬁnition of authority diﬀerence, if node i has more authority, then Ai,j is positive, while if node j has more authority, Ai,j is negative. Hence, in the extreme case where Ai,j = 1, node i’s opinion is unchanged, while node j’s is changed to o(j, t + 1) = o(j, t)(1 − 2μ) + 2μo(i, t)). Thus, when μ = 0.5, node j takes on node i’s opinion exactly. Otherwise, node j’s opinion change is doubled relative to the model of [1] with the same μ. By the criteria for convergence, if both nodes have the same relative authority, then tolerance is simply d. Otherwise, note that authority disparity is also used to aﬀect the level of tolerance, which is increased by one third of the authority diﬀerence. Our reﬁned model of opinion convergence is applied in two distinct initial network topologies. We then analyze changes in network topology caused by opinion convergence, by considering clustering coeﬃcient, as deﬁned by Watts and Strogatz [8], mean shortest path, and degree distribution. We consider undirected graphs. The ﬁrst topology we consider is created by the Erd˝ os-R´enyi method where n nodes are randomly connected with a probability k. The resultant average degree of the network (the average number of ¯ The second topology connections from a node to any other node) is denoted as k. is the Watts-Strogatz ‘small world’ (SW) model [8]. Creation of this topology begins with a lattice network of n nodes, where each node i is connected to k nodes i1 , i2 ... i k and i−1 , i−2 ... i− k . Random re-wirings are then applied to 2 2 the network with a constant probability of b. Higher b produces a ﬁnal network topology that is less clustered and more like the Erd˝ os-R´enyi model. This process creates a so-called ‘small world’ network, because it results in a network with a higher clustering coeﬃcient than an Erd˝ os-R´enyi random network model, and an average shortest path only slightly longer than the Erd˝ os-R´enyi random network model. Our initial results indicated a profound eﬀect on topology by the consensus formation process, and thus a further extension to the model was deemed necessary. The ability to repeat the process of consensus formation—without

118

B.J. Prettejohn and M.D. McDonnell Table 1. Quick reference guide for variables

Variables/Acronyms Opinion, o(i, t)

Definitions Continuous variable between 0 and 1, representing the opinion of node i after t interactions, on one speciﬁc issue. The initial opinions, o(i, 0) may be uniformly or Beta distributed. Tolerance, d The maximum diﬀerence between node opinions before communication becomes disallowed. Convergence factor, µ A uniform continuous variable between 0 and 0.5. Determines the degree of convergence prior to adjustment for authority (that is, how each node’s opinion changes), when communication has occurred. ¯ Average degree, k Average number of connections from a node to other nodes in the network. Communication factor, w Determines the probability with which a link between nodes is broken, and a new link formed to another randomly chosen node when two nodes’ opinions are not within tolerance. Authority, a(i, t) Node i’s authority in relation to all other nodes, after t interactions. Derived from the popularity of a node’s opinion within its peer group, and the size of that peer group in relation to the sample population. MSP Mean shortest path: the average of all shortest paths between all node pairs in a network . CC Clustering coeﬃcient: the average of all local clustering coeﬃcients within a network as deﬁned by [8]. DD Degree distribution: the frequency of node degree occurrences within a network, where the degree for a single node is the total number of connections from that node to other nodes. Time slice, t Term introduced by [1] to refer to a single iteration of the potential communication process. Topology Refers to either the Erd˝ os-R´enyi random network model [1] (ER), or the Watts-Strogatz small-world model (SW) [8].

resetting to the original topologies—was added. This involves the replacement of ‘converged’ opinions with a new set of randomly chosen opinions. This models the eﬀect of introducing a new topic for consideration, while maintaining the topology changes which occurred in the original process of opinion convergence. A further factor is introduced to oﬀer a comparison with more polarized opinion distributions. We achieve this by use of the transformation o = 0.5 + 0.5 sin((r − 0.5)π), where r ∈ [0, 1] is a uniformly distributed variable. This results in opinions that have a Beta distribution, β(0.5, 0.5). Table 1 provides a quick reference guide for terms and variables used in this model.

3

Results

This section contains simulation results that illustrate the eﬀect of altering the various parameters described in Section 2. All results are for a network with N = 1000 agents, w = 0.25 and k¯ = 10, and ensemble-averaged over 50 diﬀerent runs.

Eﬀect of Network Topology in Opinion Formation Models

119

6

10

Number of interactions

SW ER

−2

10

−1

10 Tolerance, d

Fig. 1. Comparison of total number of required interactions for both topologies as a function of tolerance, d

Our ﬁrst ﬁnding, as illustrated in Fig. 1, is that more time slices (node interactions) are required for the SW network to reach convergence than the ER network, for d larger than about d = 0.12. For d smaller than this, there is negligible diﬀerence. We now compare the clustering coeﬃcient, mean shortest path, and degree distribution before and after the opinion convergence process, for the two diﬀerent initial network topologies and initial opinion distributions. 3.1

Topology Before and After the Opinion Formation Process

Figs. 2, 3 and 4 show that the clustering coeﬃcient (CC), degree distribution (DD) and mean shortest path (MSP) all are markedly changed by the opinion convergence process, with ﬁnal values highly dependent on the tolerance, d, for both initial topologies. As d increases, at d 0.035 the ﬁnal clustering coeﬃcient, as shown in Fig. 2— see also Fig. 5(a)—for both topologies drops below the Erd˝ os-R´enyi starting point. This reduction continues with increasing d, until d 0.12 when the random network maintains a rough parity with its initial level, with the SW slightly higher. When the opinion convergence process is run twice, the ﬁnal clustering coefﬁcient for both topologies is higher than when the process is run once. However, this diﬀerence decreases as d increases, and at d 0.03 the diﬀerence all but disappears. Fig. 3 shows the degree distribution for the network before and after opinion formation, for each topology, for several values of d. As is expected, the degree distribution in the initial state is spread wider for the ER topology (most node degrees between 0 and 20), than it is for a SW network (between 5 and 15). We ﬁnd that the ﬁnal degree distribution (after opinion formation) for both initial topologies has an inverse relationship to tolerance (in the sense that the

120

B.J. Prettejohn and M.D. McDonnell

ER− Start ER x 1− Stop ER x 2− Stop

0.3 Clustering Coefficient

Clustering Coefficient

0.25 0.2 0.15 0.1 0.05

0.25 0.2 0.15

SW− Start SW x 1− Stop SW x 2− Stop

0.1 0.05

0 −2 10

0 −2 10

−1

10 Tolerance, d

(a) ER network

−1

10 Tolerance, d

(b) SW network

Fig. 2. Initial and ﬁnal clustering coeﬃcients for single and double consensus process runs, as a function of tolerance, d

distribution is skewed towards lower degrees for smaller d), resulting in a more ‘power-law-like’ distribution with very small d. With d = 0.01, less than one node (on average) has a degree higher than 100, and most nodes have a degree within the range 0–20. Both topologies also display a rise and fall in the number of nodes with a degree approximately in the range 38 – 82, peaking at approximately 20 nodes with degree of of 60. For a larger tolerance (d = 0.2), a similar skewing of the degree distribution can be seen. However for larger tolerance d = 0.4, there is almost no change from the initial network, which is to be expected, since for large tolerance there are very few chances for nodes to alter their neighbours. Fig. 4 shows that the eﬀect of tolerance is more dramatic for MSP than it is for CC and DD. For a SW topology MSP is decreased by the opinion convergence process at low tolerance, as it is to a lesser extent for an ER topology. However, there is a sharp rise in MSP in comparison with the initial MSP for an ER 200 Start Finish d = .01 Finish d = .2 Finish d = .4

150

Start Finish d = .01 Finish d = .2 Finish d = .4

200

Frequency

Frequency

150 100

50

0 0

100

50

20

40

60 Degree

(a) ER network

80

100

0 0

20

40

60

80

100

Degree

(b) SW network

Fig. 3. Comparison of ﬁnal degree distributions, as a function of tolerance, d

Eﬀect of Network Topology in Opinion Formation Models 6

6.5 ER− Start ER x 1− Stop ER x 2− Stop

6

Mean Shortest Path

Mean Shortest Path

5.5

5

4.5

4

3.5

3 −2 10

121

5.5

SW− Start SW x 1− Stop SW x 2− Stop

5 4.5 4 3.5

−1

10

3 −2 10

Tolerance, d

(a) ER network

−1

10

Tolerance, d

(b) SW network

Fig. 4. Initial and ﬁnal mean shortest paths for single and double consensus process runs, as a function of tolerance, d

network, and a fall in MSP for a SW network, as d increases from its smallest value. Then for larger d, the MSP becomes larger than the initial value for both topologies, before dropping oﬀ sharply at the point of phase transition near d = 0.3. This results in similar MSPs after convergence for all d for both topologies. The value d 0.3 corresponds to a point of phase transition, above which the ﬁnal network usually reaches a global consensus, as noted in [6]. It is also notable that the ﬁnal MSP for the both topologies are both slightly lower when the process is run twice, compared to when it is run once. 3.2

Erd˝ os-R´ enyi (ER) Network versus Small World (SW) Network

Fig. 5 shows the ﬁnal CC and MSP after the opinion formation process, with both initial topologies on the same axes. It is clear that the ﬁnal topology is almost the same for each initial topology, although for small d the ﬁnal MSP appears to be slightly smaller for the initially ER network than it is for the initially SW network, and a similar result is apparent for CC. These results indicate that the ﬁnal network topology due to opinion formation may be nearly independent of the initial network topology. 3.3

Eﬀect of Initial Opinion Distribution

We found that changing the distribution of initial opinions resulted in a significant change in the value of tolerance above which the network always reaches global convergence. This is illustrated in Fig. 6, which shows (following [1]) the fraction of the nodes in the network contained in the two largest opinion clusters in the ﬁnal state of the network, for an ER network. An almost identical result (not shown) was obtained for an initially SW network.

122

B.J. Prettejohn and M.D. McDonnell 6.5 6

Mean Shortest Path

Clustering Coefficient

0.25 0.2 ER− Start SW− Start ER− Stop SW− Stop

0.15 0.1 0.05

ER− Start SW− Start ER− Stop SW− Stop

5.5 5 4.5 4 3.5

0 −2 10

3 −2 10

−1

10 Tolerance, d

−1

10

Tolerance, d

(a) Clustering coeﬃcient

(b) Mean shortest path

Fraction of network in each of top two clusters

Fig. 5. Comparison of the eﬀect of initial topology on mean shortest path and clustering coeﬃcient, with double opinion convergence, as a function of tolerance, d

1

0.8

Beta o Uniform o

0.6

0.4

0.2

0

0.1

0.2

0.3

0.4

d

Fig. 6. The eﬀect of initial opinion distribution on the top two opinion cluster sizes, as a function of d, for an ER network

It is clear that introducing a more polarized initial opinion distribution results in a markedly larger tolerance level being required before complete convergence of opinion is reached for both initial topologies. Complete opinion convergence is achieved for d 0.24 for uniformly distributed initial opinions, and at d 0.36 using a beta distribution.

4

Discussion

It is clear from our results that initial network topology may have only a limited role to play in the consensus formation process. It is equally clear however that the process itself aﬀects the ﬁnal topology. We begin by considering the impact of

Eﬀect of Network Topology in Opinion Formation Models

123

tolerance level. In our original paper [6], and in [1], the primary aspect of study was the formation of consensus. It was shown that a larger tolerance was able to compensate for the restrictions of a static network, and consensus required less iterations in an adaptive network. In this paper we are concerned with topology, and consider results across a range of metrics, i.e. clustering coeﬃcient (Fig. 2), degree distribution (Fig 3) and mean shortest path (Fig. 4). These have shown that the lower the tolerance level the more impact there is both by and on topology. To this end it is considered advantageous to maintain a low tolerance when deliberately trying to manipulate network topology. This makes sense from a sociological perspective; if tolerance is generally low within a given population, it follows that they tend to form relatively tight cliques within the broader population, maintaining rather narrow opinion divergency. Another outcome of note is that for a low tolerance, the ER topology requires only one consensus process to reach a ‘real world’ like outcome. This may be deﬁned loosely as a network with a small world topology (high clustering coeﬃcient compared to the initial topology) and a ‘power-law-like’ degree distribution. In contrast, a small world topology requires two iterations of the process to get this eﬀect at small tolerance—see Fig. 2(b). The ﬁrst opinion formation process results in a shorter MSP, but it is only the second process that results in a higher clustering coeﬃcient than the initial SW network. This suggests the possibility that the process of convergence, if run enough times and with an appropriately low tolerance, results in a speciﬁc topology type, which is robust in terms of initial topology type. Much research has been carried out into methods for ‘growing’ network topologies which have a power law degree distribution. Obviously there are beneﬁts in such networks, particularly in terms of path length. Also such networks are statistically unlikely to be overly aﬀected by removal of a randomly chosen node. The one major drawback is that in the extreme case of loss of ‘super-nodes’ (nodes with very high degree), the results can be catastrophic. If a network was able to form or re-form such a topology, without adding or removing nodes—an outcome the process of convergence seems to provide—networks required to display the properties inherent with this type of topology would be considerably more robust. The eﬀect of initial opinion distribution is not surprising from a sociological viewpoint. What the results show is that if a network has two polarized opinion groups, then more interactions and a higher tolerance will be required to reach a complete consensus. However, it is an interesting phenomena from a topology manipulation perspective, suggesting that manipulation of speciﬁc variables can be used to change or maintain the network architecture. While more extensions to our work are required, our results suggest that precise topologies may be created by introduction of speciﬁc types of manipulation algorithms to existing node variables in adaptive networks.

124

B.J. Prettejohn and M.D. McDonnell

Acknowledgment Mark D. McDonnell is funded by an Australian Research Council Post Doctoral Fellowship, DP0770747.

References 1. Kozma, B., Barrat, A.: Consensus formation on adaptive networks. Physical Review E 77 (2008), Art. No. 016102 2. Kozma, B., Barrat, A.: Consensus formation on coevolving networks: Groups’ formation and structure. Journal of Physics A: Mathematical and Theoretical 41 (2008), Art. No. 224020 3. Barab´ asi, A.L., Crandall, R.E.: Linked: the new science of networks. American Journal of Physics 71, 409–410 (2003) 4. Ben-Naim: Opinion dynamics: Rise and fall of political parties. Europhysics Letters 69, 671–677 (2005) 5. Baron, R.A., Byrne, D.E., Branscombe, N.R.: Social psychology, 11th edn. Pearson, Allyn-Bacon, Boston (2006) 6. Prettejohn, B.J., McDonnell, M.D.: Eﬀects of authority on consensus formation for continuous opinions in an adaptive network. In: Proc. European Conference on Complex Systems (2009), Art. No. 145 7. Milgram, S.: The small world problem. Psychology Today 2, 60–67 (1967) 8. Watts, D.J., Strogatz, S.H.: Collective dynamics of ‘small-world’ networks. Nature 393, 440–442 (1998) 9. Guttmann, C., Rahwan, I., Georgeﬀ, M.: A study of an approach to the collective iterative task allocation problem. In: Proc. International Conference on Intelligent Agent Technology (2007) 10. Braet, A.C.: Ethos, pathos and logos in aristotle’s rhetoric: A re-examination. Argumentation 6, 307–320 (1992) 11. Poggi, I.: The goals of persuasion. Pragmatics and Cognition 13, 297–336 (2005)

Behavior Learning in Minority Games Guanyi Li, Ying Ma, Yingsai Dong, and Zengchang Qin Intelligent Computing and Machine Learning Lab School of Automation Science and Electrical Engineering Beihang University, Beijing, 100191, China [email protected]

Abstract. The Minority Game (MG) is a simple model for the collective behavior of agents in an idealized situation where they have to compete through adaptation for a ﬁnite resource. It has been regarded as an interesting complex dynamical disordered system from a statistical mechanics point of view. In this paper we have investigated the problem of learning the agent behaviors in the minority game. We assume the existence of one “intelligent agent” who can learn from other agent behaviors. We consider two scenarios in this research: (1) Given an environment with complete information, i.e., all records of agents’ choices are known to public. The intelligent agent can use a Decision Tree to learn the patterns of other agents and make predictions. (2) If we only know the data of collective behaviors, we assume the data are generated from combining the behaviors of variant groups of agents. The intelligent agent can use a Genetic Algorithm to optimize these group parameters in order to get the best guess of the original system. The experimental results show that, in this conﬁguration of MG in both environments with complete information and incomplete information, the intelligent agent can learn from history data and predict which side is the minority.

1

Introduction

Agent-based experimental games have attracted much attention among scientists in diﬀerent research areas, particularly, psychology [13], economics [10,14] and ﬁnancial market modeling [5,7,10]. Agent-based models of complex adaptive systems (CAS) provide invaluable insight into the highly non-trivial collective behavior of a population of competing agents. These systems are universal and we aim to model the systems where involving agents are with similar capability competing for a limited resource. Agents share global information, which is in turn generated by the action of the agents themselves, and they learn from past experience. The ﬁeld of econophysics [9] is an area that such complex system models may be applicable: every agent knows the history data in a stock market and must decide how to trade based on this global information. Among these models, minority game (MG) [4] is an important model in which an odd number N of agents successively compete to be in the minority. It can be regarded as a simpliﬁed version

Corresponding author.

C. Guttmann, F. Dignum, and M. Georgeﬀ (Eds.): CARE 2009/2010, LNAI 6066, pp. 125–136, 2011. c Springer-Verlag Berlin Heidelberg 2011

126

G. Li et al.

of EI Farol bar problem [1], in which a number of people decide weekly whether go to the EI Farol bar to enjoy live music in the risk of staying in a crowd place or stay at home. As a new tool for learning complex adaptive systems, the minority game has been applied to variety areas [7,8] especially in ﬁnancial market modeling [2,3]. But this model has limitations when describing the real case where people try to make a better choice. The ﬁrst thing we notice is that not all agents use the same pattern to select strategy. In real-life scenarios, the complexity of marketing world is embodied in existence of varieties types of agents using strategies based on their own rules. Several patterns of agents should have existed such as agents who make random decisions. Also it is unrealistic to have a population with all agents following the rules of a game. There must be some more intelligent ones who make a more wise decision by learning from others. The previous unrealistic assumptions are not enough for us to have a good understanding of the dynamics of an adaptive system (e.g., a market). In this paper, we propose a model that eﬃciently ﬁgures out limitations of previous models. Through observing the dynamics, we continuously increase complexity of our model by adding diverse types of agents in order to simulate and analyze intricate systems. We also introduce intelligent agents that can learn adaptively by using machine learning and data mining. We consider two learning scenarios in this paper: (1) learning with complete information that includes all records of other agents’ choices and winning outcomes at each time and (2) learning with incomplete information that only contains winning outcomes. Eventually, the intelligent agent can analyze and estimate the environment to maximize own proﬁts in terms of winning probability. It is a new way to understand the relationship of micro-behaviors and macro-behaviors by utilizing minority game model and machine learning. This paper is organized as the following: section 2 introduces the minority game and propose the framework in which an intelligent agent can learn the patterns of other agents by using complete information. In section 3, we assume an environment in which the intelligent agent does not have the complete information. We propose a method of using a genetic algorithm for discovering the composition of agents in order to predict the macro-behavior of the MG. By conducting a series of experiments, we verify the usefulness of this learning mechanism and conclusions are given at the end.

2

Learning in the Minority Games

In the minority game, there is an odd number of players and each must choose one of two choices independently at each turn. The players who end up on the minority side win. It is originated from the El Farol Bar problem that was originally formulated to analyze a decision-making method other than deductive rationality, the minority game examines the characteristic of the game that no single deterministic strategy may be adopted by all participants in equilibrium [15]. In this section, we will set an environment that is populated by a few diverse groups of agents. We aim to design one type of intelligent agents that

Behavior Learning in Minority Games

127

can learn from the patterns of other agents and make choices in order to be on the minority side. 2.1

Strategies of Agents

Formally, the minority game consists of N (an odd number) agents, at time t (t = 1, . . . , T ), each agent need to take an action ai (t) for i = 1, · · · , N , to attend room A or B. A Agent i choose room A (1) ai (t) = B Agent i choose room B In each round of the game, the agents belonging to the minority group are the winners. The winning outcome can be considered as a binary function w(t). Without loss of generality (w.l.o.g), if A is the minority side, we deﬁne that the winning outcome is 0, otherwise, it is 1. In this paper, the winning outcomes are known to all agents. 0 #(ai (t) = A)|i=1,...,N ≤ (N − 1)/2 w(t) = (2) 1 otherwise where #() is the function for counting numbers: for all the agents (i runs from 1 to N ), if the number of agents that satisfy the condition ai (t) = A is less than or equal to (N − 1)/2, then w(t) = 0; otherwise, it is w(t) = 1. We assume that agents make choices based on the most recent m winning outcomes h(t), which is called memory and m is called the length of memory. h(t) = [w(t − m), . . . , w(t − 2), w(t − 1)]

(3)

Given the outcome w(t) at the moment t, agent i keeps a record ri (t) that tells whether it has won the game or not. W in Agent i wins at time t ri (t) = (4) Loss Agent i loses at time t In minority game, we usually assume that each agent’s reaction based on the previous data is governed by a “strategy” [2,3,4]. Each strategy is based on the past m-bit memory which are described as a binary sequence. Every possible m-bit memory are mapped in correspond to a prediction of choosing m room A or B in the next round. Therefore, there are 22 possible strategies in the strategy space. Agents in the same ﬁxed strategy group share one strategy randomly selected from the strategy space. Given the memory h(t), the choice for the agent i guided by the strategy S is denoted by S(h(t)). Table 1 shows one possible strategy with m = 3. For example, h(t) = [000] represents that if the agent who choose A in the latest three time steps win, the next round (at time t) choice for this agent will be S([000]) = A. A strategy can be regarded as a particular set of decisions on the permutations of previous winning outcomes. Giving the history data of all winning outcomes, training data for an agent can be sampled by a sliding window with size m on each time step. At time t,

128

G. Li et al.

Table 1. One possible strategy with m = 3, the current choice of agent is decided by its previous 3 step memory h(t) 000 001 010 011 100 101 110 111 S(h(t)) A B B A B A B B

the target value (either A or B) is the agent’s actual choice at the current time. Therefore, training set for agent i can be formally deﬁned as: Di = {(h(t), ai (t))}

f or

t = 1, . . . , T

(5)

The left-hand side ﬁgure of Fig. 1 shows a schematic illustration of how to sample the training data with m = 3. 2.2

Decision Tree Learning

Decision tree induction is one of the simplest and yet most successful learning algorithms. It has been widely used in numerous machine applications for its simplicity and eﬀectiveness [12]. A decision tree can be regarded as a set of rules which are interpreted from branches of the tree. In this paper, the tree model we use is actually probability estimation tree [11] because we are considering the probability distribution over two possible choices of A and B.

Fig. 1. Left-hand side: training data D is obtained from a sliding window of size m. Right-hand side: A schematic decision tree trained on the database given on the left.

The right-hand side ﬁgure of Fig. 1 shows a decision tree for modeling behaviors of agent i based on the training date given on the left. For each branch W = [w(t − m), . . . , w(t − 2), w(t − 1)], there is an associated probability distribution on possible agent’s choices (i.e., A or B) that is calculated based on the proportion of these two kinds of data falling through the branch. Or formally: #(h(t) = W ∧ ai (t) = A) (6) P (A|W ) = #(h(t) = W ) (h(t),ai (t))∈Di

Behavior Learning in Minority Games

129

where #() is the counting function in eq. 2 and eq. 6 means that: given a branch, it looks for all matching string from the training data and checks how many of them chose A. If agent i follows a particular ﬁxed strategy and game is repeated for n (n 0) times. We then can get a fair good estimation of the behaviors of agent i given the 3-step memory training data. 2.3

Learning with Fixed Strategies

Some previous studies on MG or other multi-agent systems usually assume agents are homogeneous [4,5,10]. The ﬁrst thing we notice is that not all agents use the same pattern to select strategy. In real-life scenarios, the complexity of marketing world is embodied in existence of varieties types of agents using strategies based on their own rules. In the following experiment, N agents are divided into four groups. The agents in ﬁrst group make random choices between A and B using an uniform distribution, which are called random agents. This is a realistic assumption in the real-world: there is always something without patterns that we cannot control or even understand. The second and the third group of agents follow two particular strategies S1 and S2 , respectively (see table 2). The remaining one agent is a special agent called the “intelligent agent” who can observe not only the history of winning, but also all the records of each agent’s choices. Based on these data, the intelligent agent can use decision tree (or other machine learning models) to learn the patterns of each agent. His decision making is then based on its predictions on the other agents. Table 2. Two ﬁxed strategies with m = 4 are used in experiments h(t) 0000 0001 0010 0011 0100 0101 0110 0111 1000 1001 1010 1011 1100 1101 1110 1111 S1 (h(t)) A A B B B A B B B A A A A A B B S2 (h(t)) A B B B A A A B A A B A B B A B

For each agent i, its current choice ai (t) can be predicted by the decision tree learning based on the training data (see eq. 5). At time t, the probability that the intelligent agent choose A, PI (A), is calculated based on its estimation of other agents’ choices ai (t) where i = 1, . . . , N . #(ai (t) = A) (7) PI (A) = 1 − #(ai (t) = A) + #(ai (t) = B) i=1,...,N and PI (B) = 1 − PI (A). The above equation can be interpreted that the intelligent agent will go to the room that most of agents won’t go based on its predictions. Simply, the intelligent agent makes choice based on its estimation PI (A) and PI (B), or formally: A PI (A) > PI (B) aI (t) = (8) B otherwise

G. Li et al. 30

20

10

0 0

100

200 300 Number of Games

400

500

Accuracy of Intelligent Agent

Number of Agents in Room A

130

1 0.8 0.6 0.4 0.2 0 0

100

200 300 Number of Games

400

500

Fig. 2. Left-hand side: the number of agents in room A in 500 runs of ﬁxed strategies MG experiments. Right-hand side: the accuracy of the intelligent agent.

The accuracy of wining for the intelligent agent I can be obtained by: #(rI (t) = W in) ACI (t) = #(rI (t) = W in) + #(rI (t) = Loss) t=1,...,T

(9)

In the following experiment, we set the total number of agents N = 31, the number of random agents Nr = 12, the number of agents using S1 and S2 are Ns1 = 6 and Ns2 = 12, respectively. The right-hand side ﬁgure of Fig. 2 shows the performance of the intelligent agent by running the minority game for 500 times. The results show that the intelligent agent can take advantages by observing and learning from other agents’ behaviors. He can predict which room is going to be the minority side with accuracy of 90% after 500 runs. However, in the macro-level (see the left-hand side ﬁgure of Fig. 2), the system still looks very random and shows no patterns. Table 3. Impact of random agents: results are obtained by using diﬀerent percentage of random agents in 500 runs of the minority game Percentage of Random Agents 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Accuracy of Intelligent Agent (ACI ) 0.96 0.96 0.92 0.88 0.82 0.75 0.66 0.65 0.53 0.45 0.44

With higher accuracy, the intelligent agent is more likely to predict right in the next round. From the above experimental results, we can conclude that intelligent agent works almost perfectly with ﬁxed strategies although over 30% of agents are unpredictable random agents. To study the inﬂuence of the random agents, we tested diﬀerent percentage of random agents ranging from 0 to 100% and the winning accuracy of the intelligent agent is shown in table 3. Seen from the table that the accuracy of intelligent agent is almost monotonically decreasing as the percentage of random agent increases. When the percentage of random agents becomes larger than 80%, the predictive accuracy for the intelligent is around 50%. It is understandable because the dominance of random agents makes the system become totally random and unpredictable. When the

Behavior Learning in Minority Games

131

accuracy is less than 50%, it means that the learning method is ineﬀective because even the random choice can achieve the accuracy around 50%. 2.4

Adaptive Learning

Fixed strategies for agents are not realistic assumptions in the real-world. Agents should have the ability to change their strategies adaptively based on their own records. For example, based on strategy S1 that given h(t) = [0011] the agent should take action of S1 (h(t)) = B (see table 2). However, if it fails for most of time in experiments, the agent may need to adjust its strategy to choose A at the next time when see the same h(t). Given a particular memory sequence h (t), agent i’s losing probability for this sequence is calculated as the following: #(h (t) = h(t) ∧ ri (t) = Loss) (10) li (h (t)) = #h (t) = h(t) (h(t),ai (t))∈Di The above equation can be interpreted as the following: given a memory string (say, [0110]), it looks for all the matching strings from the training data and see how many times the corresponding strategy S([0110]) will fail. If it is lower than a given threshold L, we need to adjust the strategy from A to B or vice versa. Formally, for the agent i, its strategy S can be adjusted by: ¬S(h(t)) if li (h(t)) > Li (11) S(h(t)) = S(h(t)) otherwise

30

20

10

0 0

500

1000 1500 Number of Games

2000

Accuracy of Intelligent Agent

Number of Agents in Room A

where the negative function gives ¬A = B and ¬B = A. The probability threshold for each agent i is sampled from a normal distribution: Li ∼ N (0.7, 0.12) in the following experiments. The reason of using a normal distribution is to increase the heterogeneous behaviors of agents in order to avoid “sudden changes” at some critical points. In order to get suﬃcient statistics, the changing of strategy can only be considered after 200 runs of MGs. If the losing probability for one memory string is greater than the given threshold, we will update the strategy before continuing 1 0.8 0.6 0.4 0.2 0 0

500

1000 1500 Number of Games

2000

Fig. 3. Left-hand side: the number of agents in room A in 2000 runs of MG experiments. Right-hand side: the accuracy of the intelligent agent in the environment populated by a number of adaptive agents. Adaptive change of strategies starts after 200 runs.

132

G. Li et al.

Table 4. Impact of adaptive agents: results are obtained from 2000 runs of the minority games Percentage of Adaptive Agents in S1 and S2 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 Accuracy of Intelligent Agent (ACI ) 0.86 0.78 0.73 0.65 0.59 0.58 0.55 0.51

the MG experiments. The right-hand side ﬁgure of Fig. 3 shows the accuracy of the intelligent agent with the following parameters: Nr = 6, NS1 = 9 where 4 of them use adaptive strategies. NS2 = 15 where 5 of them are adaptive agents. The intelligent agent performs very well in the ﬁrst 200 runs because the strategies are ﬁxed. After that, the learning ability has a sharp drop because of the sudden increase of the complexity. Even under these conditions that contains much uncertainties making the system so unpredictable, the accuracy of intelligent agent can still slowly stabilizing to a certain value that is around 60%. This proves the superiority of the learning model. We further test the impact of the percentage of adaptive agents and the results are shown in table 4: the accuracy of the intelligent agent ACI is decreasing with the increasing percentage of the adaptive agents. Even in the condition that the system is populated with nearly 20% random agents (i.e. Nr = 6 of total N = 31 agents) and 70% of adaptive agents, the accuracy ACI is still a little bit larger than 50%.

3

Learning Collective Behaviors

In previous sections, we designed an intelligent agent that uses machine learning method to learn the patterns of other agents in the MG. The experimental results show that it can predict the minority side in diﬀerent scenarios. However, it assumes that the intelligent agent knows the complete information about other agents, i.e., who went to which room in which round of the game. That is not a realistic assumption. In this section, we aim to let the intelligent learn from the macro-level data w(t) without knowing the records of each agent ri (t). In this research, we still assume that N agents are divided into a few groups. One group of agents is random agents, and several groups of agents have ﬁxed strategies of their own. However, we have no idea how many agents in each group and what strategies this group of agents employ. We only know the history of winning outcomes w(t) and a guessed maximum number of groups K. In this section, we use a vector of parameters to represent the number of agents in each group and the strategy they use, then use a Genetic Algorithm [6] to optimize these parameters in order to obtain the most similar history of winning sequence. For such a problem with a large parameter space, using the Genetic Algorithm is always a good way for ﬁnding the most suitable parameters for its simpleness and eﬀectiveness. 3.1

Fitness Function of MG

Genetic algorithms, developed by Holland [6], comprise a class of search, adaptation, and optimization techniques based on the principles of natural evolution.

Behavior Learning in Minority Games

133

Fig. 4. The process for calculating the ﬁtness function for a chromosome at time t. A chromosome is consisted by numbers of agents in each group and the strategies this group of agents employ. For each chromosome xj , we can obtain a sequence of winning outcomes yj (t) by running MGs based on the given parameters. The ﬁtness function is calculated based on the comparisons between yj (t) and the actual sequence of winning outcomes w(t).

Possible solutions are coded as chromosomes that evolves through generations based on a fitness function which evaluates the quality of the solution. Chromosomes with higher ﬁtness are more likely to be selected to be used to reproduce oﬀ-springs. New generations of chromosomes are reproduced by crossover and mutation [10]. Successive generations are created in the same way until ultimately the ﬁnal population satisﬁes certain criterion or reach a speciﬁed number of runs. In our model, intelligent agent only use the information of winning outcomes w(t) and a guessed maximum number of groups using ﬁxed strategies K, such that the agents can be divided into K +1 groups: {Gr , G1 ,. . ., GK }, where group Gr is the group of random agents and Gk for k = 1, . . . , K employs the strategy Sk . We use the following parameters to deﬁne one MG: percentage of random agents Pr , corresponding percentage of agents with one certain ﬁxed strategy PSk . Therefore, each chromosome x is composed by these parameters: x = {Pr , PS1 , S1 , . . . , PSK , SK } The calculation of the ﬁtness function f (x) is illustrated in ﬁgure 4. At time t of the game, in order to evaluate one chromosome xj (j = 1, . . . , J where J is the population size in the GA), we run the MG with the parameter setting given by xj to obtain the history of winning outcomes yj (t). Comparing y(t) with the actual sequence w(t): for t runs from 1 to a speciﬁed time T , once yj (t) = w(t), we add one point to f (xj ), formally: f (xj (t)) + 1 if: yj (t) = w(t) f (xj (t)) ← (12) otherwise f (xj (t))

134

G. Li et al.

At each time t, the best chromosome xj is selected from the pool: x∗ (t) = arg max f (xj (t)) j

f or

j = 1, . . . , J

Given the best chromosome x∗ (t), its parameters can give the best possible complete information of a MG so that the intelligent agent can learn with decision trees discussed in section 2. 3.2

Evolutionary Experiments

In the following experiments: we deﬁne K = 4, strategies Sk are generated with memory length m = 3. Other parameters for the MG are: N = 81, Pr = 0.04, PS1 = 0.27, PS2 = 0.31, PS3 = 0.16, PS4 = 0.21. For the genetic algorithm: crossover rate Pc = 0.7, mutation rate Pm = 0.01 and population size J = 50. The upper ﬁgure of Fig. 5 shows the time-series data of how many agents went to room A in 1000 runs. This macro-data has the empirical mean nearly 41 and with a small variance. This property makes the prediction even harder because a small change of agent’s choice may lead to a diﬀerent winning outcome. The overall performance looks random and unpredictable. By using the method discussed in the previous section, we use a genetic algorithm with the above parameters setting to run 1000 generations and the results are shown in the lower ﬁgure of Fig. 5. As we can see from the ﬁgure, by the increase of generations, the accuracy of the intelligent agent increase monotonically before reaching a stabilizing stage which is around 0.78. The signiﬁcance of these results is that, the collective behaviors of MGs can be decomposed into several micro-behaviors controlled by diﬀerent group of agents. The behaviors of each group may follow certain simple strategies.

Number of Agents in Room A

46 44 42 40 38 0

100

200

300

400 500 600 Number of Games

700

800

900

1000

100

200

300

400 500 600 Number of Games

700

800

900

1000

Accuracy of Intelligent Agent

0.8 0.6 0.4 0.2 0 0

Fig. 5. Upper: Macro-level data: the number of agents in room A. Below: Accuracy of the intelligent agent by using genetic algorithm.

Behavior Learning in Minority Games

135

Combining these simple behaviors may generate complex and seemingly unpredictable macro-behaviors. Given a sequence of history wining outcomes, by using genetic algorithm, we can ﬁnd the most likely combinations of single behaviors that could generate this sequence. Many real-world complex phenomena are related to the minority game [7,8,4] such as the ﬂuctuations of stocks and currency exchange rates, although the macro-level data seemingly are random and unpredictable. Researchers could use the history of macro-level data and build models with this MG+GA framework to reconstruct the mechanism for generating these data in order to predict possible future results. This research points a new way of using the minority game model and evolutionary optimization in understanding the relationship between micro-data and macro-data. Our future work is to use real-world data to study some complex real-world phenomena that could be decomposed and explained by using this MG+GA framework.

4

Conclusions

In this paper, we proposed a method of designing intelligent agents which can learn from other agents’ patterns in the minority game. Two learning scenarios are considered and tested with experiments. In the environment of complete information, a simple decision tree algorithm is used for making decisions by the intelligent agent. Experiments are conducted with ﬁxed strategies and adaptive strategies. The results show that the intelligent agent can capture the trend and make wise decisions. However, if all agents adaptively adjust their strategies, the system becomes unpredictable again. In the environment of incomplete information, only the history of winning outcomes are known. We assume that the agents can be decomposed into a few groups of agents that may follow diﬀerent strategies. We hope to ﬁnd out a combination of group behaviors that most likely to generate the given collective behaviors. A genetic algorithm can be used to ﬁnd such solutions and experimental results show that the intelligent agent can predict the minority side with a probability much larger than 0.5. Our future work aims to use the MG+GA framework proposed in this research to study some real-world complex adaptive systems.

Acknowledgment This work is partially funded by NCET of Chinese Ministry of Education and Aerospace 100 Project of Beihang University.

References 1. Arthur, W.B.: Bounded rationality and inductive behavior (the El Farol problem). American Economic Review 84, 406 (1994) 2. Challet, D., Marsili, M., Zhang, Y.C.: Stylized facts of ﬁnancial markets and market crashes in minority games. Physica A 294, 514 (2001)

136

G. Li et al.

3. Challet, D., Marsili, M., Zecchina, R.: Statistical mechanics of systems with heterogeneous agents: Minority Games. Phys. Rev. Lett. 84, 1824 (2000) 4. Challet, D., Zhang, Y.C.: Emergence of cooperation in an evolutionary game. Physica A 246, 407 (1997) 5. Gode, D.K., Sunder, S.: Allocative eﬃciency of markets with zero-intelligence traders: Market as a partial substitute for individual rationality. Journal of Political Economy 101(1), 119–137 (1993) 6. Holland, J.H.: Emergence: From Chaos to Order (1998) 7. Johnson, N.F., Jeﬀeries, P., Hui, P.M.: Financial Market Complexity. Oxford University Press, Oxford (2003) 8. Lo, T.S., Hui, P.M., Johnson, N.F.: Theory of the evolutionary minority game. Phys. Rev. E 62, 4393 (2000) 9. Mantegna, R.N., Stanley, H.E.: An Introduction to Econophysics: Correlations and Complexity in Finance. Cambridge University Press, Cambridge (1999) 10. Qin, Z.: Market mechanism designs with heterogeneous trading agents. In: Proceedings of Fifth International Conference on Machine Learning and Applications (ICMLA 2006), Orlando, Florida, USA, pp. 69–74 (2006) 11. Qin, Z.: Naive Bayes classiﬁcation given probability estimation trees. In: The Proceedings of ICMLA 2006, pp. 34–39 (2006) 12. Qin, Z., Lawry, J.: Decision tree learning with fuzzy labels. Information Sciences 172/1-2, 91–129 (2005) 13. Rapoport, A., Chammah, A.M., Orwant, C.J.: Prsoner’s Dilemma: A Study in Conﬂict and Cooperation. Uni. of Michigan Press, Ann Arbor (1965) 14. Smith, V.L.: An experimental study of competitive market behavior. Journal of Political Economy 70, 111–137 (1962) 15. http://en.wikipedia.org/wiki/El_Farol_Bar_problem

Two Decades of Multiagent Teamwork Research: Past, Present, and Future Matthew E. Taylor1 , Manish Jain2 , Christopher Kiekintveld3, Jun-young Kwak2 , Rong Yang2 , Zhengyu Yin2 , and Milind Tambe2 1

2

Lafayette College, Easton, PA University of Southern California, Los Angeles, CA 3 University of Texas at El Paso, El Paso, TX

Abstract. This paper discusses some of the recent cooperative multiagent systems work in the TEAMCORE lab at the University of Southern California. Based in part on an invited talk at the CARE 2010 workshop, we highlight how and why execution-time reasoning has been supplementing, or replacing, planning-time reasoning in such systems.

1 Introduction There have been over two decades of work in computer science focusing on cooperative multiagent systems and teamwork [6,15,28], much of it in the context of planning algorithms. In addition to the problems encountered in single-agent scenarios, multi-agent problems have a number of significant additional challenges, such as how agents should share knowledge, assist each other, coordinate their actions, etc. These extra considerations often make multi-agent problems exponentially more difficult, relative to single agent tasks, in terms of both the computational complexity and the amount of memory required for a planner. As discussed in the following section, the BDI (Belief-Desire-Intention) framework was one of the first to directly addresses multi-agent problems with significant theoretical and experimental success. In addition to formalizing teamwork relationships, BDI became popular because of its ability to reduce computation. Rather than requiring agents to laboriously plan for all possible outcomes, or expect a centralized planner to account for a state space exponential in the number of agents, the BDI approach allowed agents to reason about their plans at execution-time and adapt to information gathered about the environment and teammates. Later techniques focused more on preemptive planning, requiring a computationally intensive planning phase up front, but allowed the agents to execute their joint plan with few requirements at execution time. Two particular approaches, DCOPs and DECPOMDPs, will be discussed in later sections of this chapter. The DCOP framework allows agents to explicitly reason about their coordination in a network structure in order to achieve a joint goal. DEC-POMDPs use centralized planning to reasoning about uncertainty, both in the sensors and actuators of the agents, producing provably (near-) optimal plans for the multi-agent team. While the current generation of multi-agent techniques, including DCOPs and DECPOMDPs, have been successful in a number of impressive contexts, they often fail C. Guttmann, F. Dignum, and M. Georgeff (Eds.): CARE 2009/2010, LNAI 6066, pp. 137–151, 2011. c Springer-Verlag Berlin Heidelberg 2011

138

M.E. Taylor et al.

to scale up to large numbers of agents. More important, they ignore the power of execution-time reasoning and focus on planning-time reasoning. In this chapter, we argue that the multi-agent community would do well to focus on incorporating more execution-time reasoning, possibly inspired by past BDI methods, in order to 1) reduce planning time, 2) reduce the amount of required coordination, and/or 3) allow agents to gracefully handle unforeseen circumstances. After we give a brief introduction to the BDI framework, we will discuss some of our own work in the DCOP and DEC-POMDP frameworks, highlighting the benefits of integrating execution-time and planning-time reasoning.

2 The Past: BDI The Belief-Desires-Intention (BDI) formalism was the dominant approach to multiagent teamwork in the mid-90’s, spurred on in large measure from the work on SharedPlans [6] and joint-intentions [15]. The key idea behind BDI was to capture some of the “common-sense” ideas of teamwork and address questions like: “why does communication arise in teamwork,” “why do teammates help each other,” and “how can a teammate best help another teammate?” Answers to these questions were captured in a logic-based domain-independent form, allowing for the same types of team-level reasoning in disparate domains (e.g., a team of airplane pilots or a team of personal office assistants). One important contribution of BDI was that this domain independence allowed programmers to reason about teams at very high levels of abstraction. BDI teamwork libraries could be responsible for the low-level control of coordinating the team, handling failures, assigning agent roles, etc., allowing the programmer to instead focus on coding at the team level of abstraction. BDI proved useful in (at least) three distinct ways: 1. through direct implementation of the logic as agent decision-making code, 2. as inspiration for operationalization in other languages, and 3. for the rational reconstruction of implemented systems. Benefit #2 in particular has been useful in that it has allowed for the development and deployment of large-scale teams (c.f., [9,28]). A second important contribution of BDI was to focus on execution time reasoning. As discussed in the previous section, a set of pre-defined rules could be used at execution time, allowing agents to react to their environment without needing to plan for all possible team contingencies ahead of time.

3 DCOPs This section briefly introduces the DCOP framework and then discusses recent advancements in multi-agent asynchronous reasoning and multi-agent exploration. 3.1 Background Distributed Constraint Optimization Problems (DCOP) [18,19] are cooperative multiagent problems where all agents are part of a single team; they share a common

Two Decades of Multiagent Teamwork Research: Past, Present, and Future

x2 x1

0 1 0 7 2 1 2 3

R 1,2

R 2,3 1

2

3

x2

139

x3 0 1 0 15 2 1 2 9

Fig. 1. This figure depicts a three agent DCOP

reward function. DCOPs have emerged as a key technique for distributed reasoning in multiagent domains, given their ability to optimize over a set of distributed constraints, while keeping agents’ information private. They have been used for meeting scheduling problems [17], for allocating tasks (e.g., allocating sensors to targets [14]) and for coordinating teams of agents (e.g., coordinating unmanned vehicles [27] and coordinating low-power embedded devices [5]). Formally, a DCOP consists of a set V of n variables, {x1 , x2 , . . . , xn }, assigned to a set of agents, where each agent controls one variable’s assignment. Variable xi can take on any value from the discrete finite domain Di . The goal is to choose values for the variables such that the sum over a set of binary constraints and associated payoff or reward functions, fij : Di × Dj → N , is maximized. More specifically, find an assignment, A, s.t. F(A) is maximized: F (A) = xi ,xj ∈V fij (di , dj ), where di ∈ Di , dj ∈ Dj and xi ← di , xj ← dj ∈ A. For example, in Figure 1, x1 , x2 , and x3 are variables, each with a domain of {0,1} and the reward function as shown. If agents 2 and 3 choose the value 1, the agent pair gets a reward of 9. If agent 1 now chooses value 1 as well, the total solution quality of this complete assignment is 12, which is locally optimal as no single agent can change its value to improve its own reward (and that of the entire DCOP). F ((x1 ← 0), (x2 ← 0), (x3 ← 0)) = 22 and is globally optimal. The agents in a DCOP are traditionally assumed to have a priori knowledge of the corresponding reward functions. 3.2 k-OPT and t-OPT: Algorithms and Results When moving to large-scale applications, it is critical to have algorithms that scale well. This is a significant challenge for DCOP, since the problem is known to be NP-hard. Recent work has focused on incomplete algorithms that do not guarantee optimal solutions, but require dramatically less computation and communication to achieve good solutions. Most of the incomplete algorithms in the literature provide no guarantees on solution quality, but two new methods based on local optimality criteria, k-size optimality [23] and t-distance optimality [11], offer both fast solutions and bounds on solution quality. The key idea of k-size optimality is to define optimality based on a local criteria: if no subset of k agents can improve the solution by changing their assignment, an assignment is said to be k-size optimal. Using a larger group size gives better solutions (and bounds), but requires additional computational effort. A variation on this idea, tdistance optimality, uses distance in the graph from a central node to define the groups

140

M.E. Taylor et al.

of agents that can change assignment. Formally, we define these optimality conditions as follows. Definition 1. Let D(A, A ) denote the set of nodes with a different assignment in A and A . A DCOP assignment A is k-size optimal if R(A) ≥ R(A ) for all A for which |D(A, A )| ≤ k. Consider the DCOP in Figure 1. The assignment {1, 1, 1} is a k-size optimal solution for k = 1 (with reward of 12), but not k = 2 and k = 3. It is 1-size optimal because the reward is reduced if any single variable changes assignment. However, if x2 and x3 change to 0 the reward increases to 17 from 12, so {1, 1, 1} is not 2-size optimal. Definition 2. Let T (vi , vj ) be the distance between two variables in the constraint graph. We denote by Ωt (v) = {u|T (u, v) ≤ t} the t-group centered on v. A DCOP assignment A is t-distance optimal if R(A) ≥ R(A ) for all A , where D(A, A ) ⊆ Ωt (v) for some v ∈ V . There are at most n distinct t-groups in the constraint graph, centered on the n variables. There may be fewer than n distinct groups if some Ωt (v) comprise identical sets of nodes. Consider again the DCOP in Figure 1. Assignment {1, 1, 1} is 0-distance optimal, because each t-group contains a single node, equivalent to k = 1. However, {1, 1, 1} is not 1-distance optimal. The t = 1 group for x2 includes both other variables, so all three can change to assignment 0 and improve the reward to 22. Both k-size optimal solution and t-distance optimal solution have proven quality bounds that improve with larger value of k or t. However, there is a distinct tradeoff between k-size and t-distance optimality. In k-size optimality, the number of nodes in each individual group is strictly bounded, but the number of distinct k-groups may be very large, especially in dense graphs. For t-distance optimality the situation is reversed; the number of groups is bounded by the number of variables, but the size of an individual t-group is unbounded and may be large in dense graphs. Empirically, this has significant implications for the speed of solution algorithms for computing the two types of local optima. Algorithms. One advantage of k-size and t-distance optimality is that they can be computed using local search methods. DALO (Distributed Asynchronous Local Optimization) (DALO) is an algorithmic framework for computing either k-size or t-distance optimal solutions for any setting of k or t. DALO is fast, and supports anytime, asynchronous execution. This makes it ideal for dynamic environments that require significant execution-time reasoning. At a high level, DALO executes in three phases:1 1. Initialization. Agents send initialization messages to nearby agents, which are used to find all of the k or t groups in the constraint graph and assign each group a unique leader. 2. Optimization. Each group leader computes a new optimal assignment for the group, assuming that all fringe nodes maintain their current assignment, where fringe nodes of a group are directly connected to a group member, but are not members themselves. 1

More details about the algorithm can be found elsewhere [11].

Two Decades of Multiagent Teamwork Research: Past, Present, and Future

141

3. Implementation. The group leader implements the new assignment if it is an improvement, using an asynchronous locking/commitment protocol. DALO is particularly useful in execution time reasoning of large agent teams for the following reasons. First, DALO allows agents to reason and act asynchronously by following the locking/commitment protocol, avoiding expensive global synchronization in execution. Second, as a locally optimal algorithm, DALO requires much smaller amount of computation and communication on each individual agent as opposed to a globally optimal algorithm, leading to efficient execution in dynamic environments. Third, as verified by our simulations, the convergence speed of DALO remains almost constant with increasing number of agents, demonstrating its high scalability. Experimental Evaluation. Here, we present an abbreviated set of results showing some of the advantages of local optimality criteria and the DALO algorithm. We test k-size optimality and t-distance optimality using a novel asynchronous testbed and performance metrics.2 In our experiments, we vary the setting of computation / communication ratio (CCR) to test algorithms across a broad range of possible settings with different relative cost for sending messages and computation. Katagishi and Pearce’s KOPT [10], the only existing algorithm for computing k-size optima for arbitrary k, is used as a benchmark algorithm. In addition, we examine tradeoffs between k-size and t-distance optimality. We show results for: 1) random graphs where nodes have similar degrees, and 2) NLPA (Nonlinear preferential attachment) graphs in which there are large hub nodes. Figure 2 shows a few experimental results. As shown in Figures 2, both DALO-k and DALO-t substantially outperform KOPT, converging both more quickly and to a higher final solution quality.3 In general, DALO-t converges to a higher final solution quality, though in some cases, the difference is small. Convergence speed depends on both the graph properties and the CCR setting. DALO-k tends to converge faster in random graphs (Figure 2(a)) while DALO-t converges faster in NLPA graphs (Figure 2(b)). Figure 2(c) shows the scalability of DALO-t and DALO-k as we increase the number of nodes tenfold from 100 to 1000 for random graphs. The time necessary for both DALO-k and DALO-t to converge is nearly constant across this range of problem size, demonstrating the high scalability of local optimal algorithms. The asynchronous DALO algorithm provides a general framework for computing both k-size and t-distance optimality, significantly outperforming the best existing algorithm, KOPT, in our experiments and making applications of high values of t and k viable. DALO allows us to investigate tradeoffs: DALO-t consistently converges to better solutions in practice than DALO-k. DALO-t also converges more quickly that DALOk in many settings, particularly when computation is costly and the constraint graph 2

3

Code for the DALO algorithm, the testbed framework, and random problem instance generators are posted online in the USC DCOP repository at http://teamcore.usc.edu/dcop The settings t=1 and k=3 are the most closely comparable; they are identical in some special cases (e.g., ring graphs), and require the same maximum communication distance between nodes. Empirically, these settings are also the most comparable in terms of the tradeoff between solution quality and computational effort.

Random Graphs, CCR 0.1 100 80 60

T1 K3 KOPT 3

40 20 0 0

200

400

600

Global Time

800

1000

Normalized Quality

M.E. Taylor et al.

Normalized Quality

142

NLPA Graphs, CCR 0.1 100 80 60

T1 K3 KOPT 3

40 20 0 0

100

(a)

Convergence Time

200

300

Global Time

400

500

(b)

Scaling to Large Graphs 350 300 250 200 150 100

100% Quality K3 95% Quality K3 100% Quality T1 95% Quality T1

50

0 100 200 300 400 500 600 700 800 900 1000

Number of Nodes (c)

Fig. 2. Experimental results comparing DALO-k, DALO-t, and KOPT

has large hub nodes. However, DALO-k converges more quickly on random graphs with low computation costs. Investigating additional criteria for group selection (e.g., hybrids of k-size and t-distance) can be a key avenue for future work. 3.3 DCEE: Algorithms and the Team Uncertainty Penalty Three novel challenges must be addressed while applying DCOPs to many real-world scenarios. First, agents in these domains may not know the initial payoff matrix and must explore the environment to determine rewards associated with different variable settings. All payoffs are dependent on agents’ joint actions, requiring them to coordinate in their exploration. Second, the agents may need to maximize the total accumulated reward rather than the instantaneous reward at the end of the run. Third, agents could face a limited task-time horizon, requiring efficient exploration. These challenges disallow direct application of current DCOP algorithms which implicitly assume that all agents have knowledge of the full payoff matrix. Furthermore, we assume that agents cannot fully explore their environment to learn the full payoff matrices due to the task-time horizon, preventing an agent from simply exploring and then using a globally optimal algorithm. Indeed, interleaving an exploration and exploitation phase may improve accumulated reward during exploration.

Two Decades of Multiagent Teamwork Research: Past, Present, and Future

143

Such problems are referred to as DCEE (Distributed Coordination or Exploration and Exploitation) [7], since these algorithms must simultaneously explore the domain and exploit the explored information. An example of such a domain would be a mobile sensor network where each agent (mobile sensor) would explore new values (move to new locations) with the objective of maximizing the overall cumulative reward (link quality, as measured through signal strength) within a given amount of time (e.g., within 30 minutes). We here discuss both k=1 and k=2 based solution techniques for DCEE problems. Most previous work in teamwork, including previous results in k-optimal algorithms, caused us to expect that increasing the level of teamwork in decision making would lead to improved final solution quality in our results. In direct contradiction with these expectations, we show that blindly increasing the level of teamwork may actually decrease the final solution quality in DCEE problems. We call this phenomenon the teamwork uncertainty penalty [29], and isolate situations where this phenomenon occurs. We also introduce two extensions of DCEE algorithms to help ameliorate this penalty: the first improves performance by disallowing teamwork in certain settings, and the second by discounting actions that have uncertainty. Solution Techniques. This section describes the DCEE algorithms. Given the inapplicability of globally optimal algorithms, these algorithms build on locally optimal DCOP algorithms. While all the algorithms presented are in the framework of MGM [22], a k-optimal algorithm for a fixed k, the key ideas can be embedded in any locally optimal DCOP framework. In k=1 algorithms, every agent on every round: (1) communicates its current value to all its neighbors, (2) calculates and communicates its bid (the maximum gain in its local reward if it is allowed to change values) to all its neighbors, and (3) changes its value (if allowed). An agent is allowed to move its value if its bid is larger than all the bids it receives from its neighbors. At quiescence, no single agent will attempt to move as it does not expect to increase the net reward. k=2 algorithms are “natural extensions” of k=1 algorithms. In these algorithms, each agent on each round: (1) selects a neighbor and sends an Offer for a joint variable change, based on its estimate of the maximal gain from a joint action with this neighbor; (2) for each offer, sends an Accept or Reject message reflecting the agent’s decision to pair with the offering agent. Agents accept the offer with the maximum gain. (3) calculates the bid or the joint gain of the pair if an offer is accepted, and otherwise calculates the bid of an individual change (i.e. reverts to k=1 if its offer is rejected). (4) If the bid of the agent is highest in the agent’s neighborhood, a confirmation message is sent to the partnering agent in case of joint move, following which (5) the joint / individual variable change is executed. The computation of the offer per agent in a k=2 DCEE algorithms is as in k=1, the offer for a team of two agents is the sum of individual offers for the two agents without double counting the gain on the shared constraint. k=2 algorithms require more communication than k=1 variants, however, have been shown to reach higher or similar solution quality in traditional DCOP domains [16]. Static Estimation (SE) algorithms calculate an estimate of the reward that would be obtained if the agent explored a new value. SE-Optimistic assumes the maximum reward on each constraint for all unexplored values for agents. Thus, in the mobile sensor

144

M.E. Taylor et al.

network domain, it assumes that if it moved to a new location, the signal strength between a mobile sensor and every neighbor would be maximized. On every round, each agent bids its expected gain: NumberLinks × MaximumReward − Rc where Rc is the current reward. The algorithm then proceeds as a normal k=1 algorithm, as discussed above. SE-Optimistic is similar to a 1-step greedy approach where agents with the lowest rewards have the highest bid and are allowed to move. Agents typically explore on every round for the entire experiment. On the other hand, SE-Mean assumes that visiting an unexplored value will result in the average reward to all neighbors (denoted μ) instead of the maximum. Agents have an expected gain of: NumberLinks × μ − Rc , causing the agents to greedily explore until they achieve the average reward (averaged over all neighbors), allowing them to converge on an assignment. Thus, SE-Mean does not explore as many values as SE-Optimistic, and is thus more conservative. Similarly, Balanced Exploration (BE) algorithms allow agents to estimate the maximum expected utility of exploration given a time horizon by executing move, as well as precisely when to stop exploring within this time horizon. The utility of exploration is compared with the utility of returning to a previous variable setting (by executing backtrack) or by keeping the current variable setting (executing stay). The gain for the action with the highest expected reward is bid to neighbors. This gain from exploration depends on: (1) the number of timesteps T left in the trial, (2) the distribution of rewards, and (3) the current reward Rc of the agent, or the best explored reward Rb if the agent can backtrack to a previously explored state. The agent with the highest bid (gain) per neighborhood wins the ability to move. BE-Rebid computes this expected utility of move given that an agent can, at any time, backtrack to the best explored value, Rb , in the future. On the other hand, BE-Stay assumes that an agent is not allowed to backtrack, and thus decides between to move to a new value or stay in the current value until the end of the experiment. Thus, BE-Stay is more conservative than BE-Rebid and explores fewer values. Results. The DCEE algorithms were tested on physical robots and in simulation.4 A set of Creates (mobile robots from iRobot, shown in Figure 3(a)) were used. Each Create has a wireless CenGen radio card, also shown in the inset in Figure 3(a). These robots relied on odometry to localize their locations. Three topologies were tested with physical robots: chain, random, and fully connected. In the random topology tests, the robots were randomly placed and the CenGen API automatically defined the neighbors, whereas the robots had a fixed set of neighbors over all trials in the chain and fully connected tests. Each of the three experiments were repeated 5 times with a time horizon of 20 rounds. Figure 3(b) shows the results of running BE-Rebid and SE-Mean on the robots. SEMean and BE-Rebid were chosen because they were empirically found out to be the best algorithms for settings with few agents. The y-axis shows the actual gain achieved by the algorithm over the 20 rounds over no optimization. The values are signal strengths in decibels (dB). BE-Rebid performs better than SE-Mean in the chain and random graphs, but worse than SE-Mean in the fully connected graph. While too few trials 4

The simulator and algorithm implementations may be found at http://teamcore.usc.edu/dcop/.

Two Decades of Multiagent Teamwork Research: Past, Present, and Future

145

Physical Robot Results Absolute Gain

1000 800 600 400 200 0 Chain

SE-Mean

Random

Fully Connected

BE-Rebid

(b) Physical Robot Results

(a) iRobot Create

(c) Simulation Results Fig. 3. Experimental results for DCEE algorithms on robots and in simulation

were conducted for statistical significance, it is important to note that in all cases there is an improvement over the initial configuration of the robots. Additionally, because decibels are a log-scale metric, the gains are even more significant than one may think on first glance. Figure 3(c) compares the performance of the k=1 variants with the k=2 variants. The y-axis is the scaled gain, where 0 corresponds to no optimization and 1 corresponds to the gain of BE-Rebid-1. The x-axis shows the four different topologies that were used for the experiments. The different topologies varied the graph density from chain to fully connected with random 13 and 23 representing graphs where roughly 13 and 23 of number of links in a fully connected graph are randomly added to the network respectively. The k=2 algorithms outperform the k=1 algorithms in the majority of situations, except for SE-Optimistic-1 and BE-Rebid-1 on sparse graphs (chain and random 13 ). For instance, SE-Optimistic-1 and BE-Rebid-1 outperform their k=2 counterparts on chain graphs (paired t-tests, p < 5.3 × 10−7 ), and BE-Rebid-1 outperforms BE-Rebid2 on Random graphs with 13 of their links (although not statistically significant). This reduction in performance in k=2 algorithms is known as the team uncertainty penalty.

146

M.E. Taylor et al.

Understanding Team Uncertainty. That k=2 does not dominate k=1 is a particularly surprising result precisely because previous DCOP work showed that k=2 algorithms reached higher final rewards [16,23]. This phenomenon is solely an observation of the total reward accrued and does not consider any penalty from increased communication or computational complexity. Supplemental experiments that vary the number of agents on different topologies and vary the experiment lengths all show that the factor most critical to relative performance of k=1 versus k=2 is the graph topology. Additionally, other experiments on robots (not shown) also show the team uncertainty penalty — this surprising behavior is not limited to simulation. Two key insights used to mitigate team uncertainty penalty are: (1) k=2 variants change more constraints, because pairs of agents coordinate joint moves. Given k=2 changes more constraints, its changes could be less “valuable.” (2) k=2 variants of BERebid and SE-Optimistic algorithms can be overly aggressive, and prohibiting them from changing constraints that have relatively low bids may increase their achieved gain (just like the conservative algorithms, BE-Stay-2 and SE-Mean-2, outperform their k=1 counterparts, as shown in Figure 3(c)). Indeed, algorithms have been proposed that discourage joint actions with low bids, and/or discount the gains for exploration in the presence of uncertainty and have been shown to successfully lessen the team uncertainty penalty [29].

4 DEC-POMDPs This section provides a brief introduction to DEC-POMDPs and then highlights a method that combines planning- and execution-time reasoning. 4.1 Background The Partially Observable Markov Decision Problem (POMDP) [8] is an extension of the Markov Decision Problem (MDP), which provides a mathematical framework for modeling sequential decision-making under uncertainty. POMDPs model real world decision making process in that they allow uncertainty in the agents’ observations in addition to the agents’ actions. Agents must therefore maintain a probability distribution over the set of possible states, based on a set of observations and observation probabilities. POMDPs are used to model many real world applications including robot navigation [4,12] and machine maintenance [24]. Decentralized POMDPs (DEC-POMDPs) model sequential decision making processes in multiagent systems. In DEC-POMDPs, multiple agents interact with the environment and the state transition depends on the behavior of all the agents. 4.2 Scaling-Up DEC-POMDPs In general, the multiple agents in DEC-POMDPs have only limited communication abilities, complicating the coordination of teamwork between agents. Unfortunately, as shown by Bernstein et al. [3], finding the optimal joint policy for general DEC-POMDPs is NEXP-complete. There have been proposed solutions to this problem which typically

Two Decades of Multiagent Teamwork Research: Past, Present, and Future

147

fall into two categories. The first group consists of approaches for finding approximated solution using efficient algorithms [2,20,30]; the second group of approaches has focused on finding the global optimal solution by identifying useful subclasses of DEC-POMDPs [1,21]. The limitation of first category of work is the lack of guarantee on the quality of the solution, while the second category of approaches sacrifices expressiveness. 4.3 Execution-Time Reasoning in DEC-POMDPs Although DEC-POMDPs have emerged as an expressive planning framework, in many domains agents will have an erroneous world model due to model uncertainty. Under such uncertainty, inspired by BDI teamwork, we question the wisdom of paying a high computational cost for a promised high-quality DEC-POMDP policy — which may not be realized in practice because of inaccuracies in the problem model. This work focuses on finding an approximate but efficient solution built upon the first category as discussed earlier to achieve effective teamwork via execution-centric framework [26,31,32], which simplifies planning by shifting coordination (i.e., communication) reasoning from planning time to execution time. Execution-centric frameworks have been considered as a promising technique as they significantly reduce the worst-case planning complexity by collapsing the multiagent problem to a single-agent POMDP at plan-time [25,26]. They avoid paying unwanted planning costs for a “highquality” DEC-POMDP policy by postponing coordination reasoning to execution-time. The presence of model uncertainty exposes three key weaknesses in past executioncentric approaches. They: (i) rely on complete but erroneous model for precise online planning; (ii) can be computationally inefficient at execution-time because they plan for joint actions and communication at every time step; and (iii) do not explicitly consider the effect caused by given uncertainty while reasoning about communication, leading to a significant degradation of the overall performance. MODERN (MOdel uncertainty in Dec-pomdp Execution-time ReasoNing) is a new execution-time algorithm that addresses model uncertainties via execution-time communication. MODERN provides three major contributions to execution-time reasoning in DEC-POMDPs that overcome limitations in previous work. First, MODERN maintains an approximate model rather than a complete model of other agents’ beliefs, leading to space costs exponentially smaller than previous approaches. Second, MODERN selectively reasons about communication at critical time steps, which are heuristically chosen by trigger points motivated by BDI theories. Third, MODERN simplifies its decision-theoretic reasoning to overcome model uncertainty by boosting communication rather than relying on a precise local computation over erroneous models. We now introduce the key concepts of Individual estimate of joint Beliefs (IB) and Trigger Points. IBt is the set of nodes of the possible belief trees of depth t, which is used in MODERN to decide whether or not communication would be beneficial and to choose a joint action when not communicating. IB can be conceptualized as a subset of team beliefs that depends on an agent’s local history, leading to an exponential reduction in belief space compared to past work [26,31]. The definition of trigger points is formally defined as follows:

148

M.E. Taylor et al.

Definition 3. Time step t is a trigger point for agent i if either of the following conditions are satisfied. Asking. In order to form a joint commitment, an agent requests others to commit to its goal, P . Time step t is an Asking trigger point for agent i if its action changes based on response from the other agent. Telling. Once jointly committed to P , if an agent privately comes to believe that P is achieved, unachievable, or irrelevant, it communicates this to its teammates. Time step t is a Telling trigger point for agent i if the other agent’s action changes due to the communication. Empirical Validation: The MODERN algorithm first takes a joint policy for the team of agents from an offline planner as input. As an agent interacts with the environment, each node in IB is expanded using possible observations and joint actions from the given policy, and then MODERN detects trigger points based on the belief tree. Once an agent detects a trigger point, it reasons about whether or not communication would be beneficial using cost-utility analysis. MODERN’s reasoning about communication is governed by the following formula: UC (i) − UNC (i) > σ. UC (i) is the expected utility of agent i if agents were to communicate and synchronize their beliefs. UNC (i) is the expected utility of agent i when it does not communicate, and σ is a given communication cost. UNC (i) is computed based on the individual evaluation of heuristically estimated actions of other agents. If agents do not detect trigger points, this implies there is little chance of miscoordination, and they take individual actions as per the given policy. We first compare the performance of MODERN for four different levels of model uncertainty (α) in the 1-by-5 and 2-by-3 grid domains with two previous techniques: ACE-PJB-COMM (APC) [26] and MAOP-COMM (MAOP) [31] as shown in Table 1. In both domains, there are two agents trying to perform a joint task. The 1-by-5 grid domain is defined to have 50 joint states, 9 joint actions, and 4 joint observations. In the 2-by-3 grid, there are 72 joint states, 25 joint actions, and 4 joint observations. In both tasks, each movement action incurs a small penalty. The joint task requires that both agents perform the task together at a pre-specified location. If the joint task is successfully performed, a high reward is obtained. If the agents do not both attempt to perform the joint task at the same time in the correct location, a large penalty is assessed to the team5 . The communication cost is 50% of the expected value of the policies. The time horizon (i.e., the deadline to finish the given task) is set to 3 in this set of experiments. In Table 1, α in column 1 represents the level of model error. Error increases (i.e., the agents’ model of the world becomes less correct, relative to the ground truth) as α decreases. Columns 2–4 display the average reward achieved by each algorithm in the 1-by-5 grid domain. Columns 5–7 show the results in the 2-by-3 grid domain. We performed experiments with a belief bound of 10 per time-step for our algorithm. Table 1 shows that MODERN (columns 2 and 5) significantly outperformed APC (columns 3 and 6) and MAOP (columns 4 and 7). MODERN received statistically significant improvements (via t-tests), relative to other algorithms. MAOP showed the worst results regardless of α. 5

More detailed domain descriptions and comparisons are available elsewhere [13].

Two Decades of Multiagent Teamwork Research: Past, Present, and Future

149

Table 1. Comparison MODERN with APC and MAOP: Average Performance 1x5 Grid 2x3 Grid α MODERN APC MAOP MODERN APC MAOP 10 3.38 -1.20 -1.90 3.30 -1.20 -3.69 50 3.26 -1.20 -2.15 3.30 -1.20 -3.80 100 3.18 -1.20 -2.12 3.04 -1.20 -3.79 10000 2.48 -1.20 -2.61 2.64 -1.20 -4.01

Another trend in Table 1 is that the solution quality generally increases as α decreases. When model uncertainty is high, the true transition and observation probabilities in the world have larger differences from the values in the given model. If the true probabilities are lower than the given model values, communication helps agents avoid miscoordination so that they can avoid a huge penalty. If the true values are higher, agents have more opportunity to successfully perform joint actions leading to a higher solution quality. When model uncertainty is low, the true probability values in the world are similar or the same as the given model values. Thus, agents mostly get an average value (i.e., the same as the expected reward). Thus, as model error increases, the average reward could increase. We then measured runtime of each algorithm in the same domain settings. Note that the planning time for all algorithms is identical and thus we only measure the average execution-reasoning time per agent. In both grid domains, MODERN and APC showed similar runtime (i.e., the runtime difference between two algorithms was not statistically significant). MAOP took more time than MODERN and APC by about 80% in the 1-by-5 grid domain and about 30% in the 2-by-3 grid domain. Then, we further make the 2-by-3 grid domain complex to test the scalability of each algorithm. Two individual tasks are added to the grid, which require only one agent to perform. In this new domain, the number of joint states is 288, the number of joint actions is 49, and the number of joint observations is 9. If any agent performs the individual task action at the correct location, the team receives a small amount of reward. If an agent attempts to perform the individual task in a location where the action is inappropriate, a small penalty will be assessed. If an agent chooses the action wait, there will be no penalty or reward. In this domain, while APC or MAOP could not solve the problem within the time limit (i.e., 1800 seconds), MODERN only took about 120 seconds to get the solution. These results experimentally show that MODERN is substantially faster than previous approaches while achieving significantly higher reward. One of our key design decisions in MODERN is to use trigger points to reason about communication. In these experiments, we show how significant the benefits of selective reasoning are. We used the same scaled-up 2-by-3 grid domain that was used for runtime comparisons. Figure 4 shows runtime in seconds on the y-axis and the time horizon on the x-axis. Time horizon was varied from 3 to 8. The communication cost was set to 5% of the expected utility of the given policy. As shown in the figure, MODERN can speedup runtime by over 300% using trigger points. In particular, the average number of trigger points when T=8 was about 2.6. This means MODERN only reasons about communication for about 1/3 of the total time steps, which leads to roughly three-fold improvement in runtime.

M.E. Taylor et al.

MODERN thus represents a significant step forward because it allows agents to efficiently reason about communication at execution-time, as well as to be more robust to errors in the model than other DEC-POMDP methods.

5 Conclusion

Trigger Points (TPs) in MODERN 250 200

Runtime (sec)

150

MODERN w/ TPs MODERN w/o TPs

150 100 50

The multi-agent community was started 0 3 4 5 6 7 8 with a BDI mindset, emphasizing Time horizon execution-item reasoning. In recent years, however, much of the work has Fig. 4. Selective reasoning in MODERN shifted to planning-time reasoning. The primary argument in this chapter is that we believe execution-time reasoning to be a critical component to multi-agent systems and that it must be robustly combined with planning-time computation. We have presented recent techniques in the cooperative multi-agent domains of DCOPs and DEC-POMDPs, emphasizing asynchronous reasoning, run-time exploration, and execution-time communication reasoning. Our hope is that as methods combining planning- and execution-time reasoning become more common, the capability of large teams of complex agents will continue to improve and deployments of such teams in real-world problems will become increasingly common.

References 1. Becker, R., Zilberstein, S., Lesser, V., Goldman, C.V.: Solving Transition Independent Decentralized Markov Decision Processes. Journal of Artifical Intelligence Research (2004) 2. Bernstein, D.S., Hansen, E.A., Zilberstein, S.: Bounded policy iteration for decentralized POMDPs. In: IJCAI (2005) 3. Bernstein, D.S., Givan, R., Immerman, N., Zilberstein, S.: The complexity of decentralized control of MDPs. In: UAI (2000) 4. Cassandra, A., Kaelbling, L., Kurien, J.: Acting under uncertainty: Discrete bayesian models for mobile-robot navigation. In: IROS (1996) 5. Farinelli, A., Rogers, A., Petcu, A., Jennings, N.R.: Decentralised coordination of low-power embedded devices using the max-sum algorithm. In: AAMAS (2008) 6. Grosz, B.J., Sidner, C.L.: Plans for discourse. In: Cogent, P.R., Morgan, J., Pollack, M. (eds.) Intentions in Communication. MIT Press, Cambridge (1990) 7. Jain, M., Taylor, M.E., Yokoo, M., Tambe, M.: DCOPs meet the real world: Exploring unknown reward matrices with applications to mobile sensor networks. In: IJCAI (2009) 8. Kaelbling, L., Littman, M., Cassandra, A.: Planning and acting in partially observable stochastic domains. Artificial Intelligence 101 (1998) 9. Kaminka, G.A., Tambe, M.: Robust multi-agent teams via socially attentive monitoring. Journal of Artificial Intelligence Research 12, 105–147 (2000) 10. Katagishi, H., Pearce, J.P.: KOPT: Distributed DCOP algorithm for arbitrary k-optima with monotonically increasing utility. In: Ninth DCR Workshop (2007) 11. Kiekintveld, C., Yin, Z., Kumar, A., Tambe, M.: Asynchronous algorithms for approximate distributed constraint optimization with quality bounds. In: AAMAS (2010)

Two Decades of Multiagent Teamwork Research: Past, Present, and Future

151

12. Koenig, S., Simmons, R.: Unsupervised learning of probabilistic models for robot navigation. In: ICRA (1996) 13. Kwak, J., Yang, R., Yin, Z., Taylor, M.E., Tambe, M.: Teamwork and coordination under model uncertainty in DEC-POMDPs. In: AAAI Workshop on Interactive Decision Theory and Game Theory (2010) 14. Lesser, V., Ortiz, C., Tambe, M.: Distributed sensor nets: A multiagent perspective. Kluwer Academic Publishers, Dordrecht (2003) 15. Levesque, H.J., Cohen, P.R., Nunes, J.H.T.: On acting together. In: AAAI (1990) 16. Maheswaran, R.T., Pearce, J.P., Tambe, M.: Distributed algorithms for DCOP: A graphicalgame-based approach. In: PDCS (2004) 17. Maheswaran, R.T., Tambe, M., Bowring, E., Pearce, J.P., Varakantham, P.: Taking DCOP to the real world: efficient complete solutions for distributed multi-event scheduling. In: AAMAS (2004) 18. Mailler, R., Lesser, V.: Solving distributed constraint optimization problems using cooperative mediation. In: AAMAS (2004) 19. Modi, P.J., Shen, W., Tambe, M., Yokoo, M.: ADOPT: Asynchronous distributed constraint optimization with quality guarantees. AIJ 161, 149–180 (2005) 20. Nair, R., Pynadath, D., Yokoo, M., Tambe, M., Marsella, S.: Taming decentralized POMDPs: Towards efficient policy computation for multiagent settings. In: IJCAI (2003) 21. Nair, R., Varakantham, P., Tambe, M., Yokoo., M.: Networked distributed POMDPs: A synthesis of distributed constraint optimization and POMDPs. In: AAAI (2005) 22. Pearce, J., Tambe, M.: Quality guarantees on k-optimal solutions for distributed constraint optimization. In: IJCAI (2007) 23. Pearce, J.P., Tambe, M., Maheswaran, R.T.: Solving multiagent networks using distributed constraint optimization. AI Magazine 29(3) (2008) 24. Pierskalla, W., Voelker, J.: A survey of maintenance models: The control and surveillance of deteriorating systems. Naval Research Logistics Quarterly 23, 353–388 (1976) 25. Pynadath, D.V., Tambe, M.: The communicative multiagent team decision problem: Analyzing teamwork theories and models. JAIR (2002) 26. Roth, M., Simmons, R., Veloso, M.: Reasoning about joint beliefs for execution-time communication decisions. In: AAMAS (2005) 27. Schurr, N., Okamoto, S., Maheswaran, R.T., Scerri, P., Tambe, M.: Evolution of a teamwork model. In: Cognition and Multi-agent Interaction: From Cognitive Modeling to Social Simulation, pp. 307–327. Cambridge University Press, Cambridge (2005) 28. Tambe, M.: Towards flexible teamwork. JAIR 7, 83–124 (1997) 29. Taylor, M.E., Jain, M., Jin, Y., Yooko, M., Tambe, M.: When should there be a me in team? distributed multi-agent optimization under uncertainty. In: AAMAS (2010) 30. Varakantham, P., Kwak, J., Taylor, M.E., Marecki, J., Scerri, P., Tambe, M.: Exploiting coordination locales in distributed POMDPs via social model shaping. In: ICAPS (2009) 31. Wu, F., Zilberstein, S., Chen, X.: Multi-agent online planning with communication. In: ICAPS (2009) 32. Xuan, P., Lesser, V.: Multi-agent policies: from centralized ones to decentralized ones. In: AAMAS (2002)

Designing Intelligent Healthcare Operations Nilmini Wickramasinghe1, Christian Guttmann2, and Jonathan Schaffer3 1

School of Business IT and Logistics, RMIT University, GPO Box 2476 Melbourne VIC 3001 Australia 2 Monash University Faculty of Medicine, Nursing and Health Sciences School of Primary Health Care, Bldg 1 270 Ferntree Gully Road Notting Hill, 3168 Melbourne, Australia 3 Advanced Operative Technology Group Orthopaedic and Rheumatologic Research Center Cleveland Clinic 9500 Euclid Avenue, Desk A41 Cleveland, OH 44195 USA

Abstract. Effective decision making is vital in all healthcare activities. While this decision making is typically complex and unstructured, it requires the decision maker to gather multi-spectral data and information in order to make an effective choice when faced with numerous options. Unstructured decision making in dynamic and complex environments is challenging and in almost every situation the decision maker is undoubtedly faced with information inferiority. The need for germane knowledge, pertinent information and relevant data are critical and hence the value of harnessing knowledge and embracing the tools, techniques, technologies and tactics of knowledge management are essential to ensuring efficiency and efficacy in the decision making process. The systematic approach and application of knowledge management (KM) principles and tools can provide the necessary foundation for improving the decision making processes in healthcare. The example of the orthopaedic operating room processes will illustrate the application of the integrated model to support effective decision making in the clinical environment. Keywords: intelligence continuum, healthcare operations, multi agent systems, knowledge management.

1 Introduction As the population ages, an increasing number of people will experience debilitating degenerative arthritis of the knee and hip joint. In degenerative arthritis, the articular, gliding surface of the joint becomes worn and exposes the underlying bone of the joint. This is a painful condition for which patients seek medical care to decrease their C. Guttmann, F. Dignum, and M. Georgeff (Eds.): CARE 2009/2010, LNAI 6066, pp. 152–162, 2011. © Springer-Verlag Berlin Heidelberg 2011

Designing Intelligent Healthcare Operations

153

pain and increase their functional status. In fact, the number of persons age 65 or older is expected to double between the year 2000 and the year 2040 [1]. One of the most successful procedures in the treatment of knee and hip arthritis is to replace the worn surfaces with the metal and plastic components of a hip or knee joint replacement. This procedure is completed in a hospital’s operating room; the patient stays in hospital for a few days to start their recovery and is then discharged to home or a rehabilitation facility for further rehabilitation. Within the next twenty years, domestic demand for joint replacements is expected to increase by 174% for hips and 673% for knees [2]. The demands on the healthcare system for effective decision making in this patient population will be staggering. 1.1 Challenges Once the patient’s arthritis is end-stage and the articular cartilage is worn away, patients with painful degenerative arthritis will seek the expertise of an orthopaedic surgeon. Replacement of the degenerative surfaces of the hip and knee joint has become one of healthcare’s most successful procedures in terms of providing the patient with pain relief and improved function. These operations are performed by a surgeon in a hospital in which a surgeon has been credentialed and has privileges to admit their patients and perform operations in which the surgeon has expertise. As the population requiring medical care increases, hospitals worldwide are being challenged to provide sufficient resources, including operating rooms, for these patients. There is also more pressure on the hospitals to decrease their cost structure in the face of increasing volumes while the introduction of newer medical technology, including new and presumably more advanced implants complicates the situation. Patient preparation for a hip or knee replacement is dependent on their surgeon’s evaluation and treatment plan as well as the preoperative evaluation by anesthesia providers. In many cases, a medical evaluation is also needed to ensure that the operative procedure is done in the safest manner possible. Ensuring that patients are optimally prepared for the day of surgery is critical to keeping both the surgeon’s schedule and an operating room schedule accurate and optimal and not affected by the late cancellations that lead to lost opportunity costs. Additionally, the healthcare system must provide sufficient hospital resources so that patients can efficiently move from the operating room to the recovery room to their nursing floor bed and then to either a rehabilitation hospital bed or home with the provision of home care services such as physical therapy. The entire process can initially be represented by three distinct phases: preoperative, intraoperative and postoperative (figure 1). Each of these phases is dependent on a previous state or event and the capture of the data from that previous state is

Preoperative Phase

Intraoperative Phase

Postoperative Phase

Fig. 1. The phases of care and processes for patients undergoing joint replacement

154

N. Wickramasinghe, C. Guttmann, and J. Schaffer

important to the optimization of the next phase. Many surgeons focus their practice on joint replacement surgery and will attest that their methods and procedures don’t change significantly from operation to operation. While this statement may be disputed to some extent by the operating room personnel, every joint replacement follows a very similar pattern of events. The surgeon’s performance of a joint replacement is fairly similar across hundreds of procedures but the most significant difference between each of the operations is the substrate changes, i.e. each patient is different. Successful execution of the processes in the operating room are dependent on the preoperative, intraoperative and postoperative processes that comprise the spectrum of orthopaedic care. 1.2 Stakeholders and Objectives The process stream begins with a patient experiencing pain and decreased function sufficient to present with their complaints to an orthopedic surgeon. Once the decision is made to proceed with the procedure, the patient is scheduled for the necessary preoperative evaluations and the surgery is scheduled at the hospital. The surgeon will also indicate their preference for a specific implant system and the hospital will ensure that those implants and the instruments used for their insertion are present for the surgery. In further preparation for the day of surgery, the sterilization and supply teams at the hospital are charged with assembling all of the necessary materials, the operating room nursing team sets up the sterile instruments and equipment and the operating room and the anesthesia team is tasked with providing the patient with a pain-free operative experience. The surgeon and their assistants then can complete the operative procedure as scheduled. The postoperative recovery room nursing team provides the next step in the process by helping the patient recover from the operative episode. Then the patient will go to the most appropriate nursing floor to start their recuperation with the assistance of the nursing team, the physical and occupational therapists, the surgical team and when necessary, various medical consultants. Once specific surgeon and institutional milestones have been reached, the patient is discharged to a rehabilitation nursing facility or to home where further physical therapy is provided. In all over 250 people and over 435 individual processes are involved with a single patient’s operative procedure [3]. Each entity, hospital and surgeon’s office has specific fixed and variable expenses that are greatly influenced by every process in the patient’s care. 1.3 Technology Hip and knee implants are undergoing a constant state of innovation and improved technology. While the benefits of these purported improvements are not always proven in a stringent or conclusive examination, the implant companies are under immense pressure to improve their market share and their profitability. As technologic advances in implants evolve in the marketplace, the implant companies are challenged to maintain pricing levels that provide the desired financial margins. As with all products, as the time from initial introduction increases, the products are seen as a commodity and the downward pricing pressures increase. In many cases, the hospital bears the increased costs of the new technology that the surgeons want to use while

Designing Intelligent Healthcare Operations

155

the margins of the implant companies increase. In the last 10 years, additional developments in implant insertion methodologies have included computerized navigation systems, newer instrument sets, new bearing surfaces and newer imaging based custom insertion instrument development. While direct to consumer marketing efforts have attempted to influence and pressure the surgeon’s behavior through the demands of the consumer, the market has not been significantly influenced [4]. Additionally, the value of these newer technologies has not yet been conclusively demonstrated. Many hospitals are also involved in the implementation of electronic medical records systems to document and make available for dissemination the details of the care processes through nursing and physician notes and provide clinical decision support and computerized order entry processes. Hospital supply chain management and human resource teams have also been implementing electronic systems to improve the scheduling of personnel and the stocking, ordering and billing reconciliation of supplies and implants. The incremental costs of implementing these electronic systems have been borne by hospitals and doctor’s offices while the payers’ “reimbursement” for services rendered have been consistently decreasing. 1.4 Creating Value from Knowledge As in the context of the orthopaedic operating room, in most healthcare activities, a critical function in the care process is decision making. While providers strive to bring order and structure to the care process, most decision making processes are more typically complex and unstructured. Unstructured decision making requires the gathering of multi-spectral data and information if the decision maker is to make a prudent choice [5]. Unstructured decision making in dynamic and complex environments is challenging and the decision maker is always at a point of information inferiority [6] as the decision maker is almost always missing information. It is in such situations that the need for germane knowledge, pertinent information and relevant data are critical (ibid) and hence the value of knowledge and the tools, techniques, technologies and tactics of KM are most beneficial. Hierarchically, the gathering of information precedes the transformation of information into useable knowledge [7,8]. Hence, the rate of information collection and the quality of the collected information will have a major impact on the quality (usefulness) of the generated knowledge [9]. In the dynamic and, to a large degree, unpredictable world of global healthcare, “action space awareness” (or synonymous “competitive space awareness”) and information superiority [10,11] have become the key factors to all successful operations. Such awareness however, can only be enabled through the extraction of multi-spectral data. 1.5 Tools and Techniques of KM KM tools and techniques are defined by their social and community role in the organization in 1) the facilitation of knowledge sharing and socialization of knowledge (production of organizational knowledge); 2) the conversion of information into knowledge through easy access, opportunities of internalization and learning (supported by the right work environment and culture); 3) the conversion of tacit knowledge into "explicit knowledge" or information, for purposes of efficient and systematic

156

N. Wickramasinghe, C. Guttmann, and J. Schaffer

storage, retrieval, wider sharing and application [5]. The most useful KM tools and techniques can be grouped as those that capture and codify knowledge and those that share and distribute knowledge [12,13] (refer to Table 1. Table 1. KM Tools and Techniques KM Tools and Techniques Capture & codify databases

Description

Case-Based Reasoning Applications

• • •

Expert Systems

•

I-net Agents: Creating Individual Views from Unstructured Content

•

Systems to share and distribute Knowledge Internet, intranet and extranet

•

Group collaboration systems

•

Store structured information Assist in storing and sharing knowledge applications that combine narratives and knowledge codification to assist in problem solving represent the knowledge of experts and typically query and guide users during a decision making process are the building blocks that make individual customization of information possible in the unstructured environment of I-nets the internet is an open-access platform, the intranet and extranet, however, restrict members of a community/organization through multi-layered security controls various software to enable and support group work and collaboration

1.6 The Role of Multi Agent Systems Decision making by health care professionals is performed in complex domains, at different locations and at different times. The Multi-Agent Systems (MAS) paradigm offers an intuitive platform to investigate distributed decision making problems in real world applications and addresses the issues discussed in the Section 1.4. MAS make realistic assumptions of the autonomous and distributed nature of the components in system networks [14, 15, 16, 17]. MAS are particularly useful for the study of distributed decision making problems where a group of agents make decisions together to improve social outcomes while keeping costs low [18,19]. As discussed in Section 1.3, the system must be improved but at low costs. For the preparation of hip surgeries, agents can assist in keeping plan execution on track. Previous research has shown that MAS do not only extend existing methodologies, such as CBR [20] and expert systems [21], but they are useful to keep the execution of careplans on track in complex health care enterprise architectures [22]. Hence, MAS play a critical role in building these systems at all levels of coordination, including diagnosis, preparation of patients and the hospital setting, during surgery, and particularly at the rehabilitation phase (Section 1.1).

Designing Intelligent Healthcare Operations

157

Generic Healthcare Info System P eople

Health C are E vent ven t / Proc ess

(

Data

Com C omputer puter T echn ech nology olog y

n

Inform ation S ys tem

)

Data M art / W arehous e

es: ng ll e n c e a C h F in a re , C a h i c s lo g y h p lt a no a r H e mo g T ech De And

Fig. 2. Generic Healthcare Information System with Healthcare Challenges

People

Health Care Event / Process

(

Data

Computer Technology

n

Information System

)

Data Mart / Warehouse

Diagnostic

?

Prescriptive

? Knowledge Data Management Mining

Business Intelligence/Analytics

Data Mining DataMining

Fig. 3. The impact of the intelligence continuum on the generic healthcare system

1.7 The Intelligence Continuum To understand the role of the intelligence continuum, an examination of a generic healthcare information system is necessary (figure 2). The important aspects in this generic system include the socio-technical perspective; i.e. the people, processes and

158

N. Wickramasinghe, C. Guttmann, and J. Schaffer

technology inputs required in conjunction with data as a key input. The combination of these elements comprises an information system and within any one organization, multiple such systems could exist. To this generic system, we add the influences of healthcare challenges; i.e. the challenges of demographics, technology and finance. As baby boomers age, the incidence of people over the age of 65 is projected to increase for the next forty years [1]. Moreover, as people age, improved healthcare is providing those people over the age of 65 a longer lifespan and the ability to tell about it while also ultimately enduring many complicated medical problems and diseases. Certainly technology is helping to keep everyone alive and younger and in better health but the cost to do so is escalating exponentially [23]. Addressing these challenges is best approached through a closer examination of the data generated by the information systems and stored in the larger data warehouses and/or smaller data marts. In particular, it is important to make decisions and invoke the intelligence continuum; apply the tools, techniques and processes of data mining, business intelligence/analytics and knowledge management respectively. On applying these tools and techniques to the data generated from healthcare information systems, it is first possible to diagnose the “as is” or current state processes in order to make further decisions regarding how existing processes should be modified and thereby provide appropriate prescriptions to enable the achievement of a better future state; i.e. improve the respective inputs of the people, process, technology and data so that the system as a whole is significantly improved. The Intelligence Continuum [24] is a representation of the collection of key tools, techniques and processes of today’s knowledge economy; i.e. including but not limited to data mining, business intelligence/analytics and knowledge management. Taken together they represent a very powerful system for refining the data raw material stored in data marts and/or data warehouses and thereby maximizing the value and utility of these data assets for any organization. The first component is a generic information system which generates data that is then captured in a data repository. In order to maximize the value of the data and use it to improve processes, the techniques and tools of data mining, business intelligence and analytics and knowledge management must be applied to the data warehouse. Once applied, the results become part of the data set that are reintroduced into the system and combined with the other inputs of people, processes, and technology to develop an improvement continuum. Thus, the Intelligence Continuum includes the generation of data, the analysis of these data to provide a “diagnosis” and the reintroduction into the cycle as a “prescriptive” solution (figure 3). In today's context of escalating costs in healthcare, managed care in the US, regulations and a technology and health information savvy patient, the healthcare industry can no longer be complacent regarding embracing key processes and techniques to enable better, more effective and efficient practice management. The proliferation of databases in every quadrant of healthcare practice and research is evident in the large number of isolated claims databases, registries, electronic medical record data warehouses, disease surveillance systems, and other ad hoc research database systems [25]. Not only does the number of databases grow daily, but even more importantly, so does the amount of data within them. Pattern-identification tasks such as detecting associations between certain risk factors and outcomes, ascertaining trends in healthcare utilization, or discovering new models of disease in populations of individuals rapidly becomes

Designing Intelligent Healthcare Operations

159

CREATE/GENERATE

REPRESENT/STORE

TRANSFER/DISEMINATE

Knowledge

ACESS/USE/RE-USE Availability to docs, lab, RNs, admin staff

Fig. 4. The Key Steps of Knowledge Management

daunting even to the most experienced healthcare researcher or manager [26]. Yet these tasks may hold the answers to many clinical issues such as treatment protocols or the identification across geographic areas of newly emerging pathogens and thus are important. Add to all of this the daily volumes of data generated and then accumulated from a healthcare organization administrative system, clearly then, the gap between data collection and data comprehension and analysis becomes even more problematic. Information technology (IT) tools coupled with new business approaches such as data mining, business intelligence/analytics and knowledge management should be embraced in an attempt to address such healthcare woes [27, 28]. Figure 4 highlights important aspects of knowledge in essential healthcare operations. 1.8 Returning to the Orthopaedic Operating Room The orthopaedic operating room represents an ideal environment for the application of a continuous improvement cycle that is dependent on the Intelligence Continuum. For those patients with advanced degeneration of their hips and knees, arthroplasty of the knee and hip represent an opportunity to regain their function. Before the operation ever begins in the operating room, there are a large number of interdependent individual processes that must be completed. Each process requires data input and produces a data output such as patient history, diagnostic tests and consultations. Keeping the process moving for each patient and maintaining a full schedule for the surgeon and the hospital are challenges that require accurate and timely information for successful

160

N. Wickramasinghe, C. Guttmann, and J. Schaffer

process completion and achieving the goals for each patient, surgeon and the hospital simultaneously. The interaction between these data elements is not always maximized in terms of operating room scheduling and completion of the procedure. Moreover, as the population ages and patients’ functional expectations continue to increase with their advanced knowledge of medical issues, reconstructive orthopaedic surgeons are being presented with an increasing patient population requiring hip and knee arthroplasty. Simultaneously, the implants are becoming more sophisticated and thus more expensive. In turn, the surgeons are experiencing little change in system capacity, but are being told to improve efficiency and output, improve procedure time and eliminate redundancy. However, the system legacy is for insufficient room designs that have not been updated with the introduction of new equipment, poor integration of the equipment, inefficient scheduling and time consuming procedure preparation. Although there are many barriers to re-engineering the Operating Room and the processes involved in the complex choreography of the perioperative processes, a dearth of data and the difficulty of aligning incentives, it is indeed possible to effect significant improvements through the application of the intelligence continuum. The entire process of getting a patient to the operating room for a surgical procedure can be represented by three distinct phases: preoperative, intraopertive and postoperative (refer to figure 2). In turn, each of these phases can be further subdivided into the individual yet interdependent processes that represent each step on the surgical trajectory. As each of the individual processes are often dependant on a previous event, the capture of event and process data in a data warehouse is necessary. The diagnostic evaluation of this data set and the re-engineering of each of the deficient processes will then lead to increased efficiency. For example, many patients are allergic to the penicillin family of antibiotics that are often administered preoperatively in order to minimize the risk of infection. For those patients who are allergic, a substitute drug requires a 60 minute monitored administration time as opposed to the much shorter administration time of the default agent. Since the antibiotic is only effective when administered prior to starting the procedure, this often means that a delay is experienced. When identified in the preoperative phase, these patients should be prepared earlier on the day of surgery and the medication administered in sufficient time such that the schedule is not delayed. This prescriptive reengineering has directly resulted from mining of the data in the information system in conjunction with an examination of the business processes and their flows. By scrutinizing the delivery of care and each individual process, increased efficiency and improved quality should be realized while maximizing value. For knee and hip arthroplasty, there are over 432 discrete processes that can be evaluated and reengineered as necessary through the application of the Intelligence Continuum [29]. 1.9 Summary To improve the efficiency and efficacy of patient care especially for those patients requiring hip or knee replacement, every healthcare process on the pathway from evaluation to operation to recovery should be optimized – the inputs, transformation and outputs should be measured against specification for process time, scheduling, expenses, personnel, etc. Each individual in the long chain of processes has tacit knowledge that increases with each day of experience while the explicit knowledge in

Designing Intelligent Healthcare Operations

161

the institutional or surgeon’s policies and procedures manual are infrequently updated. The opportunity to improve the knowledge spiral and use the Intelligence Continuum to capitalize on realizing the full value of the system is unparalleled. The inherent limitations of organizational structure must be overcome to make these improvements. The first steps in a process improvement project include the identification of each knowledge point, i.e. the process mapping for joint replacement procedures with the goal of improving performance and predictability while minimizing variances, decreasing “waste” and increasing value while minimizing costs. The generation, representation, storage, transfer and transformation of knowledge are key steps in making the desired improvements in clinical and management practices and incorporating continuous innovation. The current state is that the daily volume data that is generated and accumulated is often lost, further increasing the gaps between data collection, comprehension and analysis. More patients with degenerative knee and hip arthritis will need joint replacement. Surgeons and hospitals with successful clinical outcomes will use process engineering tools to identify critical path processes and the stakeholders to optimize process efficiency, efficacy, productivity, safety and satisfaction.

References 1. U.S. Census Bureau, Decennial Census Data and Population Projections (August 14, 2008), http://www.census.gov/population/www/projections/files/ nation/summary/np2008-t1.xls 2. Kurtz, S., Ong, K., Lau, E., Mowat, F., Halpern, M.: Projections of Primary and Revision Hip and Knee Arthroplasty in the United States from 2005 to 2030. J. Bone Joint Surg. Am. 89, 780–785 (2007) 3. Choi, J.H.J., Blueter, J., Fahey, B., Leonard, J., Omilanowski, T., Riley, V., Schauer, M., Sulli-van, M., Krebs, V., Schaffer, J.: Optimization of Perioperative Processes Through Innovation and Technology for the Orthopaedic Operating Room of the Future. Cleveland Clinic J. of Med. 74(esuppl. 1), S26 (2007) 4. Schaffer, J.L., Bozic, K.J., Dorr, L.D., Miller, D.A., Nepola, J.V.: Direct-to-Consumer Marketing in Orthopaedic Surgery: Boon or Boondoggle. Journal of Bone and Joint Surgery 90, 2534–2543 (2008) 5. Wickramasinghe, N., von Lubitz, D.: Knowledge-based Enterprise Theories and Fundamentals. IGI, Hersey (2007) 6. von Lubitz, D., Wickramasinghe, N.: Creating Germane Knowledge In Dynamic Environments. Intl. J. Innovation and Learning (IJIL) 3(3), 326–347 (2006) 7. Massey, A., Montoya-Weiss, M., O’Driscoll, T.: Knowledge Management In Pursuit of Performance: Insights From Nortel Networks. MIS Quarterly 26(3), 269–289 (2002) 8. Alavi, M., Leidner, D.: Knowledge Management Systems: Issues, Challenges and Benefits. Communications of the Association for Information Systems 1, Paper #5 9. Award, E., Ghaziri, H.: Knowledge Management. Prentice Hall, Upper Saddle River (2004) 10. Boyd, J.R.: Destruction and Creation. In: Coram, R. (ed.) Boyd. Little, Brown & Co., New York (1976) 11. von Lubitz, D., Wickramasinghe, N.: Healthcare and technology: the doctrine of networkcentric healthcare. Intl. J. Electronic Healthcare 4, 322–344 (2006)

162

N. Wickramasinghe, C. Guttmann, and J. Schaffer

12. Duffy, J.: The tools and technologies needed for knowledge management. Information Management Journal 35(1), 64–67 (2001) 13. Maier, R.: Knowledge Management Systems. Springer, Berlin (2001) 14. Hewitt, C.: The Challenge of Open Systems. Byte 4(10) (1985) 15. Weiß, G.: Multiagent systems: A modern approach to distributed artificial intelligence. MIT Press, Cambridge (1999) 16. Scott, W.R.: Organizations: Rational, Natural, and Open Systems. Prentice-Hall, Upper Saddle River (2002) 17. Wooldridge, M.: Introduction to Multiagent Systems, 2nd edn. John Wiley & Sons, Inc., Chichester (2009) 18. Guttmann, C.: Collective Iterative Allocation, PhD thesis, Monash University (2008) 19. Guttmann, C., Georgeff, M., Rahwan, I.: Collective Iterative Allocation: Enabling Fast and Optimal Group Decision Making. International Journal of Web Intelligence and Agent Systems (2010) 20. Plaza, E., Ontanón, S.: Ensemble case-based reasoning: Collaboration policies for multiagent cooperative CBR. Lecture Notes in Computer Science, pp. 437–451 (2001) 21. Shaalan, K., El-Badry, M., Rafea, A.: A multiagent approach for diagnostic expert systems via the internet. Expert Systems with Applications 27(1), 1–10 (2004) 22. Wickramasinghe, K., Guttmann, C., Georgeff, M., Gharib, H., Thomas, I., Thompson, S., Schmidt, H.: Agent-based intelligent collaborative care management. In: Proceedings of The 8th International Conference on Autonomous Agents and Multiagent Systems-Volume 2, pp. 1387–1388 (2009) 23. Wickramasinghe, N., Silvers, J.B.: IS/IT The Prescription To Enable Medical Group Practices To Manage Managed Care. Health Care Management Science 6, 75–86 (2003) 24. Wickramasinghe, N., Schaffer, J.: Creating Knowledge Driven Healthcare Processes With The Intelligence Continuum. Intl. J. Electronic Healthcare (IJEH) 2(2), 164–174 (2006) 25. Lawrence, P.F., Lund, O.I., Eko, F., Sarabi, E., Wu, J.: 2009) Assessing the Database Needs of Vascular Surgeons. Journal of the American College of Surgeons 208(1), 21–27 (2009) 26. Holmes, J.P., Abbott, P., Cullen, R.N., Moody, L., Phillips, K., Zupan, B.: Clinical Data Mining: Who does it, and what do they do? In: AMIA 2002 Symposium, November 9-13 (2002) 27. Berinato, S.: CIOs at the Heart of Health-Care Change; CIO Magazine (2002) 28. McGee, M.: High-Tech Healing; Information Week (September 22, 1997) 29. Schaffer, Steiner, Krebs, Hahn: Orthopaedic Operating Room of the Future (2004) (unpublished data)

Collaborative Learning in Uncertain Environments Waiho Wong and Joseph G. Davis School of Information Technologies, the University of Sydney, Sydney, Australia {whwong,jdavis}@it.usyd.edu.au Abstract. We investigate three aspects of collaborative learning in uncertain environments - aversion to exploration, structure of the collaborative network, and strategic policy. In uncertain environments, these three aspects are likely to operate simultaneously whereas previous research at best has focussed on two, if not one, of these aspects. This paper introduces our research eﬀort to study phenomena of learning under these circumstances. We propose a model of the three aspects of learning and conduct a simulation experiment using this model to analyse the eﬀect of the three variables on learning performance and knowledge development. Keywords: collaborative learning, exploration and exploitation.

1

Introduction

Collaborative learning is expected to beneﬁt from the sharing of knowledge which can lead to improved performance. However, in environments where the veracity of the knowledge being shared is unclear, those beneﬁts may not be certain. This problem is related to balancing exploration and exploitation, a fundamental concern in adaptive learning in systems such as human organisations [1]. Exploration and exploitation operate at diﬀerent levels and for this paper, we investigate their operation at the level of the individual, the collaborative group, and the organisation. We identify three aspects, each operating at their respective levels, which inﬂuence collaborative learning in uncertain environments - aversion to exploration, structure of the collaborative network, and organisational strategic policy. In uncertain environments, these three aspects are likely to operate simultaneously whereas previous research at best has focussed on two, if not one, of these aspects. This paper introduces our research eﬀort to study phenomena of learning under these circumstances: by analysing learning performance and knowledge development, which of the three aspects, or a combination, dominate? Existing literature is ﬁrst reviewed to justify the three independent variables that we have incorporated in our model. The model development is described and a simulation experiment is then performed using this model. The section that follows includes the details of the simulation experiment and the measurements used. The results are then presented with accompanying discussion. Finally, some future work based on this model is suggested. C. Guttmann, F. Dignum, and M. Georgeﬀ (Eds.): CARE 2009/2010, LNAI 6066, pp. 163–174, 2011. c Springer-Verlag Berlin Heidelberg 2011

164

2

W. Wong and J.G. Davis

Literature Review

Human adaptation to their environment is facilitated by the capacity to learn. Causes and eﬀects learned through interacting with the environment are reinforced and transformed into knowledge. Reusing this knowledge is eﬃcient so long as it remains applicable to the environment but when the environment changes it should be relinquished in favour of new knowledge. Learning new knowledge, however, can be an expensive and risky endeavour as the results are uncertain and the resources used in the process may not be recovered. In addition, it is not clearly known when to switch between the former and latter behaviours. This has been termed the ’exploitation versus exploration’ problem [2]. Exploitation involves pursuit of reuse, eﬃciency, production and implementation. In contrast, exploration is based on notions of variation, risk taking, experimentation and discovery. Several models have been proposed to explain how the human brain manages this problem [3]. Of interest is the theory on how human neuro-modulatory systems regulate the balance between exploratory and exploitative behaviour [4]. These systems have also been correlated to human loss aversion when making decisions [5]. It may be that the disposition to hesitate in undertaking risky exploration may stem from the loss aversion inherent in humans. Therefore, human learning may have a behavioural aspect. We investigate this aspect of learning in the proposed model using the factor of aversion to exploration. The exploitation versus exploration problem is not limited to individual organisms but extends to structures with collective intelligence. This can be seen in the cases of ant colonies [6] and bumble bees [7]. In these cases, the problem is choosing the best time the colony should explore for new sources of food when existing sources are being exploited to near depletion. Human organisations are similar. For example, this can be seen in the management of research and development budgets - deciding how much of present resources should be allocated to exploring for potential new revenue sources when those resources could be used in current operations to generate revenue [8]. An approach to studying this problem in human organisations has been through the use of simulation where it was shown that greater sharing of knowledge improved the speed of learning but led to lower diversity in the knowledgebase of the population [1]. In addition, it has been shown that provided the organisation is connected i.e. every member is accessible to every other, but not necessarily directly, the population knowledgebase not only converges but can possibly be caught in an absorbing state from where it cannot escape [9]. So, eventually diversity is lost with the added risk of the organisation stopping exploration because all its members have conformed to the norm which may not be optimal. It must be noted, however, that the model used a closed-system - external stimuli may move the organisation out of the absorbing state. Before this eventuality of convergence, the structure of the sharing network leads to rich patterns in the diﬀusion of knowledge in

Collaborative Learning in Uncertain Environments

165

the population [10]. We study the collaborative network structure as the second aspect of learning to investigate. We extend the question of exploration and exploitation in a human organisation by considering the strategic goals of that organisation. Chandler proposed that the structure of an organisation, including its policies and processes, follows the strategic goals it is pursuing [11]. A hierarchical structure is typically driven by plans formulated at the top to exploit the resources at the lower levels; for the beneﬁt of eﬃciency, organisational learning may be restricted by the adherence to inﬂexible policies. In contrast, a more ﬂexible structure, as is often seen in early entrepreneurial ventures, encourages a higher rate of exploration as the organisation adapts to a new environment [12]. Indeed, structural policy can be a mechanism to balance eﬃcient exploitation and explorative learning in the pursuit of strategy. However, the formulation of strategy and the structure designed to implement it may not be appropriate in an uncertain environment [13]. We investigate the third aspect of learning in uncertain environments: the role of strategic policies.

3

Research Questions

Having identiﬁed three aspects of learning - aversion to exploration, collaborative network and strategic policy, we use two outcomes that may possibly be in conﬂict - learning performance and knowledge development - to judge the inﬂuence of those aspects. Learning performance is a measure of the learning process whereas knowledge development is the measure of the results of learning. Learning performance is measured by time-to-completion - the time taken to complete learning. Knowledge development is measured through knowledge diversity and knowledge accessibility. Knowledge diversity is a measure of the specialisation of knowledge in the population. Knowledge accessibility is a measure of the extent collective knowledge is accessible to individual members when needed. We pose the following questions, also expressed as null hypotheses, on the inﬂuence of the three aspects on learning performance and knowledge diversity: Is being averse to exploration a barrier to learning? H1: There is no diﬀerence in the learning time of agents with low aversion to exploration and agents with high aversion to exploration. Does collaboration improve learning especially when the veracity of the knowledgebase is uncertain? H2: The network that enables agents to collaborate has no eﬀect on the time they take to complete learning from an uncertain environment. What strategic policies are detrimental to learning? H3: Preferential policies have no eﬀect on the time it takes agents to complete their learning.

166

W. Wong and J.G. Davis

Is knowledge more developed in populations more willing to explore? H4a: There is no diﬀerence in the diversity of knowledge between populations of low and high aversion. H4b: There is no diﬀerence in the accessibility of knowledge between populations of low and high aversion. Does collaboration reduce the knowledge diversity of the population but make what knowledge there is more accessible? H5a: The network that enables agents to collaborate has no eﬀect on the knowledge diversity of the population H5b: The network that enables agents to collaborate does not make the collective knowledge more accessible. Is knowledge development contingent on a supporting policy? H6a: Preferential policies have no eﬀect on knowledge diversity. H6b: Preferential policies have no eﬀect on knowledge accessibility. To answer these questions, we propose a model of the three aspects of learning. We formalise the measures of learning performance and knowledge development to be used in the experiment of the model.

4

Model and Simulation

We propose the three-tier model shown in Fig. 1 to investigate the above questions. The three aspects of learning - agent aversion to exploration (A), their collaborative network (N) and the control policy in eﬀect (P) - are studied through a process of extracting the maximum value from a set of scenarios. This process is illustrated in Fig. 2.

Fig. 1. A Three-tier Model of Aspects of Learning

4.1

Scenario Set

Each scenario is modelled as a multi-armed bandit problem which is typically used to study exploitation and exploration [14] [15]. In an n-armed bandit problem, n options are available. Each option delivers a payoﬀ with a unique probability, pi if it is chosen. pi can change with time depending on the frequency the

Collaborative Learning in Uncertain Environments

167

option is chosen i.e. diminishing rewards. The goal is to maximise the expected payoﬀ. In this experiment, we have set each pi = 1 and unchanging in time. These two conditions will be removed in future work. Each scenario has a payoﬀ function f (x) for which the maximum is to be determined. The function used in simulations has a domain of [0, 50). Within this are 100 value sub-domains which are evenly distributed in the range. This ensures that f (x) returns a value for every solution x[0, 50). Therefore, f (x1 ) = f (x2 )if |x1 − x2 | < 0.5. However, f (x1 ) = f (x2 )if |x1 − x2 | ≥ 0.5 may not necessarily be true because each sub-domain’s value is set at initiation to a random number in the range of [1, 2]. The uniform distribution gives each value in the range an equal chance of being assigned. In short, each problem is a 100-armed bandit problem with random payoﬀs distributed uniformly. Also associated with each scenario is a deadline. If the maximum for a scenario is not found before this deadline, the scenario is marked as having failed. 4.2

Agents(A)

Each agent has a personal knowledgebase of its results of past scenarios. An agent learns by updating this knowledgebase. Updates occur in two ways; ﬁrst, an agent, when faced with a scenario, may consult its collaborators on their past results and update its knowledgebase if a better result is obtained. Second, after generating a new solution and having it evaluated, the agent’s memory is updated if this new solution produced a better result for the scenario. Agents generate new solutions through a random Gaussian process. Suppose a problem is posed to an agent. First, let the agent’s best known solution to this problem be μ. If the agent has no best solution nor can acquire one from its peer agents, μ is set to 25, the middle of the [0, 50) interval. Using μ as a baseline, the agent explores for a new solution. However, exploration incurs risk. We model the risk-averse behaviour of the agent through a parameter k. An agent generates solutions through a Gaussian random number generator, 50 − 0 being the length of the scenario domain interval: N (μ,

50 − 0 ) k

(1)

For a higher k, the random number generator produces a greater portion of numbers closer to μ. Therefore, the agent’s new solutions tend to be close to the currently known best solution. Conversely, with lower risk aversion, there is a greater chance that numbers further from μ are generated hence the agent explores a wider portion of the solution space. 4.3

Dispatcher (P)

The Dispatcher implements the strategic policy, at the operational level, on how scenarios are assigned to agents. At each time-step, the dispatch algorithm determines which agent is to generate a solution to the scenario it has been assigned. Depending on the algorithm, a scenario may or may not be assigned to the same agent at the next time-step. We used three algorithms in this study.

168

W. Wong and J.G. Davis

In the Random Assignment algorithm a scenario is assigned to a random available agent. The Best Performer algorithm preferentially assigns a given scenario to the agent that has since produced the best payoﬀ. When no such agent exists or when the agent is busy, the scenario is randomly assigned. This random assignment is also used when there is a tie between preferred agents. This can be seen as a strategy, albeit a naive one, to speed up expert development by concentrating scenarios with agents most familiar with them. Under the First Available Agent policy, the agents are ordered in a queue. Problems are preferentially assigned to agents closer to the head of the queue. Hence, the agent nearer the head of the queue is, on average, busier than another nearer the end. Again, this is a naive strategy to reduce the overhead of the determining assignments. 4.4

Collaborative Network (N)

Collaboration between agents is modelled through networks that allow an agent to access the knowledge of other agents. We used four network conﬁgurations in this study. In the Empty network, agents are working independently without any collaboration. In the Ring network, an agent is connected to agents on its either side. In the Hub network, there is a central agent to which all other agents are connected. Finally, in the Small-world conﬁguration, agents are connected in a Ring and then a small proportion of agents, 10% in this case, are connected randomly.

5

Simulation Process and Experiment Parameters

The process used in the simulation is illustrated in Fig. 2. A simulation run starts with a working set of scenarios to be solved. At each time step of the run, the Dispatcher assigns at most one scenario to an agent. Each agent assigned a scenario ﬁrst updates its knowledgebase by consulting its collaborating peers before generating a new solution. It then updates it knowledgebase if the new solution is evaluated to be better. If the solution produces the optimal result for the scenarios, that scenario is considered complete and removed from the working set. For suboptimal results, the scenario may is also removed if its deadline has passed. Otherwise it is returned to the working set for the next time step. A simulation run ends when the scenario working set is empty. Preliminary simulations were performed to ﬁnd suitable values for the following parameters: scenario deadline, number of scenarios, number of Agents, and values of k to reﬂect high and low aversion to exploration. These simulations, which were executed using the Empty network conﬁguration and Random dispatch algorithm, showed the following values were satisfactory: deadline - 2000 time-steps; number of scenarios - 1500; number of agents - 300; and k taking on 5.0 and 10.0 for low and high aversion to exploration respectively.

Collaborative Learning in Uncertain Environments

169

Fig. 2. Simulation Process

5.1

Experiment Setup

A particular combination of the independent variables k, network conﬁguration and dispatch algorithm is termed a learning strategy. In this study, A × N × P = 2 × 4 × 3 = 24 strategies were simulated. Thirteen simulation runs were executed for each strategy. The following measurements of the dependent variables were taken for each run: expected time-to-solution, knowledge diversity and knowledge accessibility. We derived the expected time-to-solution T¯ as the average number of timesteps taken for the system under a particular strategy to solve an arbitrary scenario. For each run, the time taken for each of the successful scenarios t was recorded and those times averaged over the number of scenarios. This was then averaged over the thirteen runs. t (2) T¯ = 13 × 1500 Next we measured the distribution of knowledge acquired by the agents in the course of training. At the end of a run and for each scenario used in the run, a count nk was taken of the number of agents having a solution, not necessarily the best, for that scenario. The ratio of the count to the number of agents is the probability of ﬁnding a solution to the scenario if some agent is selected at random. Averaged over the set of scenarios, this gives the probability pk of a randomly selected agent having a solution for some arbitrary scenario. This probability was then averaged over the thirteen runs for each strategy. nk pk |scenario pk ; pk = ; p¯k = (3) pk |scenario = 500 1500 13 This was then extended to the accessibility of knowledge in the collaborative network. In contrast to the above, if an agent did not have a solution to a scenario, a solution was attempted to be sourced from one of its collaborators.

170

W. Wong and J.G. Davis

This gives pknet , the probability of an agent ﬁnding amongst its collaborators a solution to an arbitrary solution it has never encountered. pknet and p¯knet were calculated similarly to pk and p¯k .

6

Results

The results of the descriptive statistics are presented as values corresponding to Low Aversion (k = 5.0) and High Aversion (k = 10.0). Variance analyses were done using the Kruskal-Wallis rank test for variance between data groups. Data were grouped according to the factor combination under analysis. For example, to analyse factor A, data from the experiment of all 24 strategies were grouped into either ’Low Aversion’ and ’High Aversion’ before testing for variance between the two groups.

ƌĂŶĚŽŵͲůŽǁ

ĨŝƌƐƚͲůŽǁ

ďĞƐƚͲůŽǁ

ƌĂŶĚŽŵͲŚŝŐŚ

ĨŝƌƐƚͲŚŝŐŚ

ďĞƐƚͲŚŝŐŚ

ϱϬϬ ϰϱϬ ϰϬϬ ϯϱϬ ϯϬϬ ϮϱϬ ϮϬϬ ϭϱϬ ϭϬϬ ϱϬ Ϭ ĞŵƉƚǇ

ŚƵď

ƌŝŶŐ

ƐŵĂůůͲǁŽƌůĚ

Fig. 3. Expected Time-to-Completion - lower is better

Table 1. Variance Analysis of Expected Time to Completion Factor N

p-value 0.33

H-value

Factor

p-value

H-value

3.45

NA

0.00

552.08

NP

0.00

220.28

AP

0.00

758.01

NAP

0.00

772.05

A

0.00

547.71

P

0.00

209.96

The results in Fig. 3 and Table 1 clearly show the distinctive inﬂuence agents’ aversion to exploration have on their learning performance. Taking more risk and exploring the wider problem space increases the chances of ﬁnding the maximal payoﬀ. Of note is the unexpectedly little statistical eﬀect the collaborative network has on this metric i.e. p-value = 0.33. Collaboration should

Collaborative Learning in Uncertain Environments

171

eﬀectively give agents a wider view of the scenario and a better direction to the maximal. However, in random and uncertain scenarios such as those used here, it is diﬃcult to determine the true ’better’ direction. It is then exploration that leads to achieving the goal. The data allows the rejection of hypothesis H1 aversion to exploration has a signiﬁcant inﬂuence on learning performance. H2 cannot be rejected - collaboration does not have a signiﬁcant eﬀect on learning completion time. The data lends weight to rejecting H3 - control policy has statistically signiﬁcant eﬀect on the time to complete learning.

ƌĂŶĚŽŵͲůŽǁ

ĨŝƌƐƚͲůŽǁ

ďĞƐƚͲůŽǁ

ƌĂŶĚŽŵͲŚŝŐŚ

ĨŝƌƐƚͲŚŝŐŚ

ďĞƐƚͲŚŝŐŚ

ϰϬй ϯϱй ϯϬй Ϯϱй ϮϬй ϭϱй ϭϬй ϱй Ϭй ĞŵƉƚǇ

ŚƵď

ƌŝŶŐ

ƐŵĂůůͲǁŽƌůĚ

Fig. 4. Expected Knowledge Diversity - expressed as probability; higher is better

Table 2. Variance Analysis for Expected Knowledge Diversity Factor

p-value

H-value

Factor

p-value

H-value

N

0.40

2.98

NA

0.00

29.35

A

0.00

25.51

NP

0.00

864.82

P

0.00

856.64

AP

0.00

898.06

NAP

0.00

908.78

The data in Fig. 4 and Table 2 do not support the notion that collaboration reduces knowledge diversity - the collaborative network has little statistically signiﬁcant eﬀect and hence H5a cannot be rejected. This result is at odds to those presented in the literature possibly because the dispatch policy has a stronger inﬂuence than the network. There is a statistical diﬀerence in knowledge diversity between populations of agents with low or high aversion to exploration - we reject H4a. However, this inﬂuence appears dominated by the dispatch policy in operation. Hence, we reject H6a because of the three main factors the dispatch policy aﬀects knowledge diversity the most. Given that the dispatch policy

172

W. Wong and J.G. Davis

determines the distribution of opportunities to learn in the agent population, it is expected that randomly assigning scenarios leads to higher diversity. Note also the consistently higher diversity when agents have an aversion to exploration. In contrast, it was previously shown that higher aversion led to longer learning times but by keeping the scenario in the working set longer, greater diversity is achieved. This demonstrates the conﬂicting goals of increasing speed of learning versus achieving greater diversity of knowledge. From the results in Fig. 5 and Table 3, it can be seen that although the network, under these conditions, has little eﬀect on knowledge diversity, it does enable collaboration and makes knowledge more accessible. This eﬀect is clearly shown in the contrasting results of the unconnected Empty network conﬁguration and other connected conﬁgurations. Noteworthy is the superior result of using a Small-world network to enable knowledge accessibility because it gives a higher average number of connected peers per agent (4.21) compared to the Hub (1.99) and Ring (2.00) conﬁgurations. Thus, of the three aspects, the structure of the network has the greatest eﬀect on knowledge accessibility - hypothesis H5b is rejected. H4b cannot be rejected as aversion to exploration has no statistically signiﬁcant eﬀect. The preferential Best Performer dispatch policy has resulted in a population where knowledge accessibility is low because it concentrates learning opportunities to a few agents. In contrast, the Random dispatch policy distributes

ƌĂŶĚŽŵͲůŽǁ

ĨŝƌƐƚͲůŽǁ

ďĞƐƚͲůŽǁ

ƌĂŶĚŽŵͲŚŝŐŚ

ĨŝƌƐƚͲŚŝŐŚ

ďĞƐƚͲŚŝŐŚ

ϰϱй ϰϬй ϯϱй ϯϬй Ϯϱй ϮϬй ϭϱй ϭϬй ϱй Ϭй ĞŵƉƚǇ

ŚƵď

ƌŝŶŐ

ƐŵĂůůͲǁŽƌůĚ

Fig. 5. Expected Knowledge Accessibility - expressed as probability; higher is better Table 3. Variance Analysis for Expected Knowledge Accessibility Factor

p-value

H-value

Factor

p-value

H-value

655.29

NA

0.00

657.54 907.23

N

0.00

A

0.21

1.58

NP

0.00

P

0.00

169.87

AP

0.00

175.7

NAP

0.00

916.09

Collaborative Learning in Uncertain Environments

173

opportunities uniformly throughout the population and hence increasing knowledge accessibility. Hence H6b is rejected - dispatch policy can aﬀect knowledge accessibility in the population. The data show how the three aspects of learning dominate in certain cases and have no eﬀect in others. However, the strategic dispatch policy eﬀects learning time, knowledge diversity and knowledge accessibility. Of interest is the trade-oﬀ between leaning strategies that favour learning time - low aversion to exploration - and those that favour knowledge diversity - high aversion to exploration. Also noteworthy is the lack of interaction between the three aspects.

7

Conclusion

In this paper, we have shown how three aspects of learning - aversion to exploration, structure of the collaborative network, and strategic policy - inﬂuence learning performance and knowledge development. We presented a model of these aspects operating in an uncertain environment. Through an experimental simulation of this model, we have presented results that illustrate the complexity of learning in this environment and the challenge of setting the right conditions for successful collaborative learning. The aspect with the most consistent inﬂuence is the strategic dispatch policy: opportunities to learn and develop are better provided in a fair manner across the population without too early preferential treatment of certain members i.e. premature exploitation. Although collaboration can be expected to produce positive beneﬁts, it should operate in a suitable environment. The above results have shown that when the environment is uncertain, collaboration has little eﬀect on how rapidly the right knowledge of that environment is obtained. Also, in such an environment, the eﬀect of collaboration in reducing diversity may be minimal. Our results have shown the role of the individuals in collaborative learning. A population, given fair opportunities and the resources to collaborate, still needs members willing to take the risks inherent in exploratory learning. The model proposed here is undergoing development. Here, the population has built up knowledge by learning from a static scenario set. The next phase would be to have the scenario payoﬀ probabilities change with time and determine the strategies that best shift the population between exploratory and exploitative behaviours. Also, the collaborative network could be made dynamic with links formed and broken based on the utility from the knowledge shared e.g. reputation. We expect to produce richer behaviour of collaborative learning.

References 1. March, J.G.: Exploration and exploitation in organizational learning. Organization Science 2(1), 71–87 (1991) 2. Kaelbing, L., Littman, M., Moore, A.: Reinforcement learning: a survey. Journal of Artiﬁcial Intelligence Research 4, 237–285 (1996)

174

W. Wong and J.G. Davis

3. Cohen, J.D., McClure, S.M., Yu, A.J.: Should i stay or should i go? how the human brain manages the trade-oﬀ between exploitation and exploration. Philosophical Transactions of the Royal Society B: Biological Sciences 362(1481), 931–942 (2007) 4. Aston-Jones, G., Cohen, J.D.: An integrative theory of locus coeruleusnorepinephrine function: Adaptive gain and optimal performance. Annual Review of Neuroscience 28(1), 403–450 (2005) 5. Tom, S.M., Fox, C.R., Trepel, C., Poldrack, R.A.: The neural basis of loss aversion in Decision-Making under risk. Science 315(5811), 515–518 (2007) 6. Pratt, S.C., Sumpter, D.J.T.: A tunable algorithm for collective Decision-Making. Proceedings of the National Academy of Sciences of the United States of America 103(43), 15906–15910 (2006) 7. Harder, L.D., Real, L.A.: Why are bumble bees risk averse? Ecology 68(4), 1104–1108 (1987) 8. Bange, M.M., De Bondt, W.F.M.: R&D budgets and corporate earnings targets. Journal of Corporate Finance 4(2), 153–184 (1998) 9. Gale, D., Kariv, S.: Bayesian learning in social networks. Games and Economic Behavior 45(2), 329–346 (2003) 10. Bala, V., Goyal, S.: Learning from neighbours. The Review of Economic Studies 65(3), 595–621 (1998) 11. Chandler, A.D.: Strategy and Structure: Chapters in the History of the American Industrial Enterprise. MIT Press, Cambridge (1969) 12. Cyert, R., March, J.: Behavioral Theory of the Firm, 2nd edn. Wiley-Blackwell (August 1992) 13. Mintzberg, H.: Patterns in strategy formation. Management Science 24(9), 934–948 (1978) 14. Macready, W., Wolpert, D.: Bandit problems and the exploration/exploitation tradeoﬀ. IEEE Transactions on Evolutionary Computation 2(1), 2–22 (1998) 15. Daw, N.D., O’Doherty, J.P., Dayan, P., Seymour, B., Dolan, R.J.: Cortical substrates for exploratory decisions in humans. Nature 441(7095), 876–879 (2006)

Improving Task-Based Plan Coordination Chetan Yadati Narasimha, Cees Witteveen, and Yingqian Zhang Faculty EEMCS, Delft University of Technology, Mekelweg 4, 2628 CD, Delft, The Netherlands {C.Yadati,C.Witteveen,Yingqian.Zhang}@tudelft.nl

Abstract. A multi-agent planning problem consists of a set of activities that need to be planned by several autonomous agents. Here, plan coordination methods play an important role, since independently generated plans by different agents can easily lead to an infeasible joint plan. We study a coordination-by-design approach which allows each agent to make its own plan completely independently of the others, while still guaranteeing the feasibility of the joint plan. The essence of this coordination approach is to determine a minimum number of additional constraints (a minimum coordination set) such that autonomously developed plans satisfying these constraints are always mergeable into an overall feasible plan. It has been shown that such coordination problems are very hard to solve. Therefore, approximation algorithms have been developed to compute a sufficient, but not necessarily minimum coordination set. In this paper, we concentrate on a special class of multi-agent planning problems. These problems arise in several practical applications such as supply chain management and hospital patient treatment. The plan coordination instances in these applications turn out to have a special structure. Using so-called agent dependency graphs, we show that for this special class of problems a better approximation algorithm to compute a sufficient coordination set can be obtained.

1 Introduction Multi-agent planning requires a collection of agents to solve a joint problem together. Typically, each of the agents is capable of solving only a (disjoint) part of the problem and would like to construct and apply its solution (or plan) independently of the others. Due to possible dependencies between the solutions provided, it is not guaranteed that partial solutions always can be merged into a feasible overall solution. Therefore, we need to provide a coordination mechanism. Coordination in multi-agent planning can be viewed as an attempt to achieve a conflict-free joint plan given a set of self-interested multi-agent planners. In the multiagent system community, quite some effort has been done to attack this problem [1]. The plan merging approach [2,3,4], for example, focuses on techniques that resolve conflicts after independently generated plans have been developed by different agents. Here, agents are required to exchange, negotiate, and revise their (partial) individual plans to arrive at a joint solution. Another approach treats coordination and planning as intertwined processes [5,6,7], where agents continuously exchange information during their planning in order to achieve a conflict-free solution. In these two coordination approaches, agents have to be cooperative in the sense that they should be willing to exchange their private information with others, and to revise their plans if necessary. C. Guttmann, F. Dignum, and M. Georgeff (Eds.): CARE 2009/2010, LNAI 6066, pp. 175–186, 2011. c Springer-Verlag Berlin Heidelberg 2011

176

C. Yadati Narasimha, C. Witteveen, and Y. Zhang

One particular approach has addressed the problem from the coordination-by-design perspective, that is, how to provide just sufficient coordination constraints on agents before the agents start to solve their own sub-problem, such that (i) a joint solution is always coordinated, yet (ii) each agent can construct its plan independently from others. One advantage of such a pre-planning coordination method is that it can be applied without relying on the assumption of cooperative agents. General results obtained in this approach can be summarized as follows [8,9,10]: First of all, given a coordination instance, checking whether or not agents can plan independently without coordinating them is a coNP-complete problem. Secondly, finding a minimum number of constraints such that coordination is achieved is a computationally even more demanding problem (Σ2p -complete) [9]. Later, in [10], the authors presented a polynomial approximation algorithm, called the Depth-Partitioning Algorithm, which can be used to obtain a sufficient (but not necessarily minimum) set of coordination constraints. However, it only has been shown that this algorithm provides rather good results for logistic planning problems (as used in the AIPS-planning competition). Mors [8] identified two special cases where the coordination problem becomes computationally easier: the case where an agent has at most one task, and the case where all tasks with incoming or outgoing inter-agent arcs are totally ordered amongst themselves. However, these two cases are very restricted and hardly represent any real-world problems. Built upon previous work on coordination-by-design approach, our main contributions in this paper are as follows: Firstly, we identify two interesting multi-agent planning problems supply chain management and hospital patient treatment and we demonstrate that instances of these planning problems can be identified as coordination instances with a particular structure, called intra-free coordination instances. Secondly, we show (in Section 3) that besides its practical interest, for a intra-free coordination instance it can be easily decided whether it is coordinated or not, using a so-called agent dependency graph. Consequently, the coordination checking problem for this class of instances becomes feasible. We also show that finding a minimum coordination set is NP-hard. Thirdly, we propose a new polynomial approximation algorithm (in Section 4) for obtaining sufficient, but not necessarily minimum coordination sets for the intra-free instances. This approximation algorithm is developed based on a so-called agent dependency graph. We also demonstrate that the proposed algorithm is superior to the existing depth-partitioning algorithm that has been used to provide an approximation solution to the general coordination problem.

2 Two Multi-agent Planning Problems To motivate the subject of our paper, in this section we discuss supply chain management problem and patient planning in hospitals as two particular multi-agent planning problems.

Improving Task-Based Plan Coordination

177

Plan coordination in supply chain management In general, supply chain management is the management of material and information flows both in and between enterprises, such as vendors, manufacturers, and distribution centres in a network [11]. Figure 1(a). depicts a simple supply chain network, where four different enterprises are involved: a product manufacturer, a cross dock, a raw material supplier and a retailer. The cross dock is an enterprise which does not produce anything but is simply involved in distribution of products and raw materials. Each enterprise has to perform some specific tasks (sending, shipping, or selling goods). The flow of goods between the enterprises induces precedence constraints on the tasks to be performed. For example, goods cannot be sold before they are shipped to the retailer. These dependencies are indicated by a directed arrow between tasks. Each enterprise wants to make a plan for its set of tasks. Such a plan for the manufacturer, for example, might be to wait first for the receipt of R1 and R2 before sending P1 and P2 . In general such a plan is a partially ordered set of precedence constraints, respecting the internal (intra-agent) dependency constraints. Note that here the supply chain coordination instance has a special property: the set of intra-agent constraints is empty. Enterprises in this system might be unwilling to communicate with their partners about their own plans for many reasons like privacy, etc. Therefore, coordination in a supply chain network focusses on coordinating the activities and plans of the individual enterprises involved in the supply chain1 . Now if the cross dock plans to send raw materials R1 , R2 after sending products P1 , P2 and the manufacturer decides to send P1 , P2 after receiving R1 , R2 , then we have an infeasible joint plan as shown in Figure 1(b): it contains a dependency cycle. The coordination problem thus requires us to find a minimal set of additional constraints such that whatever specific plan (sequence of tasks) is chosen by an individual agent, their joint plan is always feasible.

(a) Example logistics scenario.

(b) Plan deadlock in supply chains.

Fig. 1. Plan coordination in supply chain management

Plan coordination in Hospital patient treatment Treatment procedures for patients in a hospital involve different sets/sequences of resource usage for each patient. This leads to significant amounts of time spent in setting up equipment/resources. Whenever set-up times are considerable, departments also 1

Note here that we focus only on the topological task structure and ignore all other factors such as costs, time points and inventory levels.

178

C. Yadati Narasimha, C. Witteveen, and Y. Zhang

need to carefully plan the order in which they treat their patients. Owing to their treatment sequences patients might need to get serviced by different departments in some prescribed order. For example, the anaesthetics department is required to anaesthetize the patient before any major surgery is performed. In such cases, there exists a precedence relationship between the activities of the departments involved. Since each of the specialists involved might belong to a different department, the effectiveness of patient management depends heavily on the ‘health’ of the coordination between departments. Typically, because each department has to customize its equipment/resources for a patient, each department develops a plan to customize its equipment in a particular sequence (based on prior information about the patient). Because each department does its planning independently of others, there is a possibility that department plans conflict. This is particularly serious if patients have to follow a strict sequence of treatment procedures from different departments. Departments handle several patients from several treatment procedures and hence it gets extremely complicated for them to communicate with other departments while they are planning. As an example of this problem, consider two patients p1 and p2 who need to visit departments A1 and A2 as illustrated in Figure 2(a). The treatment sequence for patient p1 might be stipulated as “visit department A1 (task t1 ) and then visit department A2 (task t3 )”. Similarly the treatment sequence for patient p2 could be to “first visit department A2 (task t4 ) and then visit department A1 (task t2 )”. Now if department A1 chooses to treat p2 first and A2 chooses to treat patient p1 first, we clearly have a deadlock. If the treatment procedure in both departments requires a significant amount of set up time, then each department could lose a lot of time in undoing an earlier set up and setting up a new environment for a different patient. A1

t1

t2

Patient p1

Patient p2

A2

A1

t3

t1

t4

t2

(a) Treatment sequences for two patients p1 and p2 .

A2 Patient p1

Patient p2

t3

t4

(b) A1 plans to treat patient p2 first and A2 plans to treat patient p1 first.

Fig. 2. A patient planning example

We observe that in the applications presented above, several commonalities exist: – both applications are characterised by the presence of autonomous agents; – agents in both applications have tasks that are dependent upon tasks allocated to other agents; – there exist no (intra-agent) dependency constraints between tasks allocated to a single agent. As we will see, this feature greatly reduces the computational complexity

Improving Task-Based Plan Coordination

179

of some aspects of coordination and enables us to come up with better approximation algorithms. Before presenting our main results, in the next section, we will first introduce a formal model of the multi-agent coordination problem.

3 Multi-agent System Model In modelling the hospital scenario as well as the supply chain management scenario as a multi-agent system, we assume that there is a set A = {A1 , . . . , Am } agents (enterprises, departments) who need to decide on (partially) ordering a set T = {t1 , . . .tn } of n tasks given to them. Each task ti (a treatment, service or diagnosis) has to be accomplished by a unique agent A j . There might exist dependency relations between tasks: if task ti has to be performed before task t j , this is indicated by a precedence relation ti ≺ t j between them. We assume that there exist no dependencies between tasks belonging to the same agent. The precedence relation induces a partial order on the set T . We represent this partial order by an (acyclic) task graph GT = (T, ≺) where T is represented as a set of nodes and the precedence relationships between them as a set of directed arcs. Since we assume that tasks (services, treatments) are uniquely coupled to agents, we can partition the set of concrete tasks T into subsets Ti , each of which represents the set of tasks that can be performed by a single agent Ai . Hence, the agents induce a partitioning {Ti }ni=1 of the set of tasks T . The (distributed) task graph can also be represented as the tuple GT = {Ti }ni=1 , ≺. Each agent Ai , is given a set of tasks Ti together with a set ≺i of partial order constraints induced by ≺ and has to come up with a local plan to complete its set of tasks Ti , while not violating the set of constraints ≺i . We need not know what kind of planning software is used by the agents, neither how the details of their resulting plan do look like. The only information that is needed is the partial ordering that is induced on the set Ti as a result of this plan. Hence, (the consequences of) each such local plan of an agent Ai can be represented as a partial order Ti , ≺∗i , where ≺∗i is a refinement of ≺i , i.e., the relation ≺∗i is a partial order relation that extends ≺i . Although each of these agents might have a perfect local plan Ti , ≺∗i , their combination might not be a valid plan since some inter-agent constraints might be violated. Example 1. Referring to the situation in Figure 2(b), both agents A1 and A2 have perfectly valid local plans, where each plan satisfies the (empty set of) local constraints. But as demonstrated in the example, in combination these plans might suffer a deadlock. Given the local plans of the agents, a joint plan for GT is a structure {Ti }ni=1 , ≺ ∪ ≺∗1 ∪ . . . ∪ ≺∗n . Such a plan is called infeasible if the relation ≺∗ =≺ ∪ ≺∗1 ∪ . . . ∪ ≺∗n is not acyclic. A cycle in a joint plan would indicate that the plan cannot be executed: completion of task t would need the completion of another task t which in turn would require the completion of t. To prevent such infeasible joint plans, in the coordination by design approach, a set of additional precedence constraints C, called the coordination set, is added to the

180

C. Yadati Narasimha, C. Witteveen, and Y. Zhang

planning instance. Such a set guarantees that, by respecting all the local constraints and the coordination constraints while preparing their local plans, agents can be sure that joint plans always will be feasible. These constraints in C are always intra-agent constraints and restrict agents from making plans that could lead to cycles in the joint plan. Although quite some work already has been done on the analysis and construction of coordination sets, in this paper we will deal with the problem of providing suitable coordination sets in special cases like patient planning and supply-chain management. These cases are special in the sense that before coordination sets are provided, planning agents are completely free in planning the activities they have to complete. We call such planning instances intra-free: Definition 1. A coordination instance GT = {Ti }ni=1 , ≺ is intra-free if, for every Ti , the subgraph GTi = Ti , ≺i of GT is the empty graph Ti , 0 / on Ti , that is, there are no precedence relations between tasks assigned to the same agent. If an instance is intra-free, then every task t is either a completely isolated node in the task graph or is connected to a task t assigned to another agent. We call a task t a source task if its in-degree is 0, i.e., in(t) = 0. A task t is a sink task if its out-degree is 0, i.e., out(t) = 0. If for all i and each task t ∈ Ti , it holds that t is either a sink or a source task, then such an instance is called strictly intra-free.2 Given an arbitrary intra-free instance, as we will show, there is a simple procedure to transform it into an equivalent strictly intra-free structure. Hence, we will assume that plan coordination instances are always strictly intra-free. The significance of intrafree coordination instances is that they alleviate the computational complexity of the plan coordination problem and allow for the investigation of new algorithms for finding coordination sets. First, we will show that coordination-checking, which in general is known to be coNP-complete [9], is solvable in polynomial time for intra-free coordination instances. Here, the coordination checking problem is to decide whether a given coordination instance is already coordinated, i.e., there is no need to provide a non-empty coordination set to ensure that a joint plan is always feasible whenever the individual agent plans are feasible. 3.1 Coordination Checking for Intra-free Coordination Instances The technique used to prove that the coordination checking problem can be solved efficiently, is to use a so-called agent dependency graph. Essentially, this agent dependency graph is the contraction of the distributed task graph ({Ti }ni=1 , ≺) to a graph consisting only of the agents Ai and their dependencies: Definition 2. Let GT = {Ti }ni=1 , ≺ be a distributed task graph and A = {Ai }ni=1 the associated set of agents. The agent dependency graph derived from GT is a graph GA = (VA , EA ), where VA = {vAi : Ai ∈ A} is the set of nodes corresponding to agents, and EA = {(vAi , vA j ) : ∃t ∈ Ti , ∃t ∈ T j , [t ≺ t ]} the dependency relation between them. 2

There might be tasks t such that in(t) = out(t) = 0. Without loss of generality, however, such tasks can be completely neglected for coordination purposes.

Improving Task-Based Plan Coordination

181

For general coordination instances, we can use the acyclicity of the agent dependency graph to infer that the instance is already coordinated, Proposition 1. Let GT = {Ti }ni=1 , ≺ be a coordination instance and GA its associated agent dependency graph. Then the acyclicity of GA implies that GT is coordinated, i.e., GT is a yes-instance of the coordination checking problem. Proof. If GA is acyclic, the only cycles that could occur in any extension of GT are cycles within a task set Ti of an agent Ai . This is excluded, since each individual extension (Ti , ≺∗i ) of (Ti , ≺i ) has to be a plan, i.e., an acyclic refinement of ≺i . In general, the converse of this proposition is not true: even if GA is cyclic, we might have a yes-instance of GT . For a simple example, take Figure 2(a) and suppose that there is an intra-agent constraint in A1 , i.e., ≺T1 = {t1 ≺ t2 }. Then the instance is coordinated, but the agent dependency graph, having two nodes A1 and A2 has a cycle. If, however, GT = {Ti }ni=1 , ≺ is a strictly intra-free coordination instance, we can actually show that the converse holds, too: Theorem 1. Let GT = {Ti }ni=1 , ≺ be a strictly intra-free coordination instance and GA its agent dependency graph. Then GA is acyclic iff GT is a yes-instance of the coordination checking problem. Proof. Given Proposition 1, it suffices to show that GT is not coordinated if GA contains a cycle. So let GA contain a simple cycle (Ai1 , Ai2 , . . . , Aik , Ai1 ). Since the coordination instance is strictly intra-agent free, there must exist a sequence of tasks (ti1 ,2 ,ti2 ,1 ,ti2 ,2 , . . . ,tik ,1 ,tik ,2 ,ti1 ,1 ) with two tasks per agent, such that for j = 1, . . . , k, we have ti j ,1 ,ti j ,2 ∈ Ti j , ti j ,2 ≺ ti j+1 ,1 and tik ,2 ≺ ti1 ,1 . But then it immediately follows that each empty relation in GTj = VTj , 0 / has a simple acyclic extension {ti j ,1 ≺∗i j ti j ,2 } such that the graph G∗T = VT , ≺ ∪ ≺∗ , where ≺∗ is the union of these extensions ≺∗i j , contains a cycle. Hence, GT is not coordinated. Note that these results hold for strictly intra-free coordination instances. It is, however, very easy to generalize them to intra-free coordination instances by polynomially reducing intra-free instances to strictly intra-free instances: we apply the following tasksplitting procedure to tasks that violate the strict intra-free property. Given an arbitrary intra-free coordination instance GT = {Ti }ni=1 , ≺, for every i and every t ∈ Ti such that in(t), out(t) > 0, do the following: 1. split t into two tasks t1 and t2 2. for all t ≺ t, add a precedence constraint t ≺ t1 , and for all t ≺ t , add a constraint t2 ≺ t ; 3. remove t and all precedence constraints it is mentioned in. Clearly, the result is an equivalent, strictly intra-free coordination instance. Example 2. As a simple example, Figure 3 illustrates how to convert an intra-free instance to a strictly intra-free instance. The task t of agent A2 is split into two tasks t1 and t2 . We can simply verify that when the (converted) strictly intra-free instance is coordinated, the (original) intra-free instance must be also coordinated.

182

C. Yadati Narasimha, C. Witteveen, and Y. Zhang A1

A1

A2 ta

A2 ta

t

t1

t2

A3

A3

tb

tb

(a) An intra-free instance.

(b) Resulting strictly intra-free instance.

Fig. 3. An example on converting an intra-free instance to a strictly intra free instance

As a result, we have the following Proposition: Proposition 2. An intra-free coordination instance GT is coordinated whenever its strictly intra-free variant GT is coordinated. Therefore, the coordination checking problem for intra-free coordination instances can be decided in polynomial time.

4 An Approximation Algorithm for Finding Coordination Sets Given the fact that coordination checking for intra-free instances is easy, one might hope that finding a minimum coordination set is also tractable in the case of intra-free coordination instances. This turns out to be wrong. Finding such a minimum set of coordination constraints has been shown to be NP-hard: Proposition 3. The NP-complete D IRECTED F EEDBACK V ERTEX S ET problem3 is polynomially reducible to the decision version of the minimum coordination problem for intra-free coordination instances. Proof. (Sketch) Let (G = (V, E), K) be an instance of the directed feedback vertex set (FVS) problem [12]. We obtain an instance of the intra-free coordination problem by first duplicating tasks: T = V ∪ {v : v ∈ V }, for every v ∈ V creating an agent Av having the tasks v and v to complete, and adding constraints vi ≺ v j , whenever {vi , v j } ∈ E. It is not difficult to see that the resulting instance T, ≺ is an intra-free coordination instance, since for tasks v we have in(v) = 0 and for tasks v we have out(v) = 0. We easily observe that G has a feedback vertex set of size K whenever K coordination arcs are needed to ensure that T, ≺ is coordinated: For, let W ⊆ V be a feedback vertex set of G of size K. Construct the coordination set Δ = {v ≺ v) : v ∈ W }. Adding these constraints makes it impossible to use both v and v in any cycle of an extension of GT . Hence, it has exactly the same effect as removing the node Aw from the agent dependency graph. Since W is a feedback vertex set, this is precisely the set of agent nodes that render the agent dependency graph acyclic. 3

The Directed Feedback Vertex Set problem is, given a directed graph G = (V, E) and a K ∈ Z + , to decide whether there exists a subset of at most K nodes from V whose removal will render the remaining graph G acyclic.

Improving Task-Based Plan Coordination

183

Although Propostion 3 clearly indicates that finding the least number of constraints required to coordinate a given coordination instance is NP-hard, we might apply approximation algorithms to find an approximate solution to this problem. The Depth Partitioning (DP) algorithm by Steenhuisen et al. [10] is such an algorithm that finds a sufficient, but not necessarily minimum, number of coordination constraints required to coordinate a given instance. This algorithm works as follows: 1. Take the partially ordered set (T, ≺) of all tasks and determine the depth depth(t) of each task t ∈ T with respect to the precedence relation ≺. This depth is defined in the standard way as follows: 0 ∃t ∈ T [t ≺ t] depth(t) = 1 + max{depth(t ) : t ≺ t} else 2. For each task set Ti and each pair of tasks t,t ∈ Ti , we add a coordination constraint t ≺ t whenever depth(t) < depth(t ). It is not difficult to show that the coordination set thus obtained is always sufficient to guarantee that the planning instance obtained is coordinated [10]. 4.1 Finding a Better Algorithm for Intra-free Instances The drawback, however, of the DP algorithm is that the algorithm adds a constraint whenever there are two tasks of different depths, without paying attention to whether such an addition is strictly necessary or not. We propose a more frugal way of applying depth partitioning while ensuring coordination: Instead of using the depth partitioning principle for every agent we will provide a filter to select between those agents where the depth partitioning needs to be applied and those where we do not need to apply it. The basic idea is to use the agent dependency graph as such a filter. Whenever the agent dependency graph is acyclic, we do not need to add any coordination constraints. Whenever the agent dependency graph contains a cycle, we know, since the coordination instance is intra-free, that at least some coordination arcs have to be added. In the latter case, we can use an approximation of the minimum feedback vertex set to select an agent Ai occurring in the feedback set. We then apply the depth-partitioning algorithm to the task set Ti associated with this agent Ai . In general this will remove some of the cycles in the agent dependency graph. In order to make the instance intra-free again, we split the task set Ti into the k depth levels of the tasks induced by applying the depth partitioning algorithm. These task sets Ti,1 , Ti,2 , . . . , Ti,k then together with the other task sets constitute an intra-free coordination instance again and can be represented by k agents Ai,1 , . . . Ai,k in the resulting agent dependency graph. In order to ensure that these agents will not be chosen again in a feedback vertex set, we include them in a blackout list of vertices and test the acyclicity of the dependency graph again. Technically then, we need to use an approximation algorithm for the (Blackout) Feedback Vertex Set to indicate which agents create the cycle.

184

C. Yadati Narasimha, C. Witteveen, and Y. Zhang

A Blackout Feedback Vertex Set problem is an extension of the general Feedback Vertex Set (FVS) problem. In the FVS problem, one is given a directed graph and is asked to find a subset of vertices that intersect every directed cycle in the graph such that cardinality of this subset is minimum. The blackout version also has an additional special subset of vertices B which cannot be part of the feedback vertex set. An approximation algorithm for this problem has been proposed by Even et al. [13]. Note that task sets Ti consisting of tasks all having the same depths will never be affected by the DP algorithm. In order to avoid that an agent Ai associated with such a task set will be included in a feedback vertex set, we include such agents in the blackout set in the beginning. This implies that the algorithm can only choose agents whose task sets consist of tasks that can be affected by the DP algorithm. The resulting algorithm can be stated as in Algorithm 1.

Algorithm 1. Advanced Depth Partitioning Algorithm (DP*) Require: An intra free coordination instance GT = {Ti }ni=1 , ≺; A = {A1 , A2 , . . . , An } Ensure: A set of coordination constraints C added to GT to make it coordinated 1: compute depth(t) for every t ∈ T 2: let GA = VA , EA be the dependency graph associated with GT 3: B = {Ai : all tasks t ∈ Ti have the same depth } 4: C = 0/ 5: while GA contains a cycle do 6: F = BlackoutFV S(GA , B) 7: select Ai ∈ F 8: for every pair of tasks t,t ∈ Ti do 9: if depth(t) < depth(t ) then 10: add t ≺ t to C 11: end if 12: Split Ti in k subsets Ti, j having the same depth 13: T = (T − Ti ) ∪ {Ti,1 , . . . Ti,k } 14: let GA be the new dependency graph associated with GT 15: add the nodes Ai, j corresponding to the sets Ti, j to B 16: end for 17: end while 18: return C

Example 3. Consider the situation in Figure 4. It has three agents A1 , A2 , A3 needing to perform the tasks t1 , . . . ,t8 . In the first step the blackout set B is empty and the feedback vertex set algorithm is free to choose any agent into its feedback vertex set. Suppose it picks agents A1 , A3 , i.e., F = {A1 , A3 }. In agent A3 we first add coordination constraints t6 ≺ t3 , t6 ≺ t8 , t8 ≺ t3 , and then split the agent into three agents A3,1 , A3,2 , A3,3 . We then add these agents into B, i.e., B = {A3,1 , A3,2 , A3,3 } and check if the agent dependency graph has a cycle. It turns out that the agent dependency graph still has a cycle and the blackout FVS algorithm returns F = {A1 }. So we now constrain A1 by adding constraint t1 ≺ t4 . The agent dependency graph now becomes acyclic. Thus, the procedure returns four constraints t6 ≺ t3 , t6 ≺ t8 , t8 ≺ t3 , and t1 ≺ t4 . Notice here that DP algorithm would

Improving Task-Based Plan Coordination A1

A2

A3

A1

A2

t1

t2

t3

t1

t2

185

A3,3 t3 A3,2

t7

t8

t7

t8 A3,1

t4

t5

t6

(a) Given intra-free instance;

t4

t5

t6

(b) Constrain A3 ;

A1,1

A2

A3,3

A1

A2

A3

t1

t2

t3

t1

t2

t3

t7

t8

t5

t6

A3,2 A1,3 t4

t5

t8 A3,1

t5

t6

(c) Compute blackout set and constrain A1 .

t4

(d) Coordination set derived from the DP algorithm.

Fig. 4. An example of how the advanced depth partitioning algorithm works

add constraints between all tasks that have different depth (among tasks of each agent), resulting in two more constraints t7 ≺ t2 and t7 ≺ t5 as shown in Figure 4 (d). We now show the Advanced Depth Partitioning algorithm DP* is correct. Proposition 4. Let GT = ({Ti }ni=1 , ≺) be a plan coordination instance and C the set of additional precedence constraints returned by the Advanced Depth Partitioning algorithm given in Algorithm 1. Then the instance GT = (T, ≺ ∪ C) is plan coordinated. Proof. (Sketch) The correctness of this algorithm is almost immediate from the correctness of the DP-algorithm. In the DP*-algorithm, every time a different subset of agents is affected, the number of agents who can be constrained reduces because once an agent Ti has been constrained each of the Ti, j agents that result by splitting it enter the Blackout set (all tasks within Ti, j have the same depth). The DP*-algorithm will continue with adding constraints to agents until the associated agent dependency graph is acyclic. In the worst case all agents that could be affected by the depth-partitioning algorithm are affected. Note that the DP*-algorithm does not choose agents that will never be affected by the DP-algorithm to add constrains. Hence by the correctness of the DP-algorithm, the resulting instance must be coordinated.

5 Discussion and Conclusions This paper identified a new subclass of coordination problems called intra-free coordination instances which have a relevance to patient planning problems and supply chain planning problems. We have shown that the decision problem of whether or not such

186

C. Yadati Narasimha, C. Witteveen, and Y. Zhang

intra-free instances are coordinated can be solved efficiently, although the problem to find a minimum coordination set still remains NP-hard. Based on the efficient coordination checking method, we proposed a specialized approximation technique to design a ‘good’ coordination set efficiently. As part of our future research, first of all we are investigating better heuristics for producing subset minimal coordination sets and develop the theoretical performance comparison analysis. We also seek to identify other and broader classes of plan coordination problems that can be efficiently solved. Hospitals typically need to handle a significant number of emergencies and patients without prior appointments. Online mechanisms are popular models for managing such scenarios. It would be interesting to extend the DP*-algorithm to handle online scenarios and compare its performance with respect to other online coordinating mechanisms.

References 1. Durfee, E.H.: Distributed problem solving and planning. In: Weiß, G. (ed.) Multiagent Systems: A Modern Approach to Distributed Artificial Intelligence, pp. 121–164. MIT Press, Cambridge (1999) 2. Foulser, D.E., Li, M., Yang, Q.: Theory and algorithms for plan merging. Artificial Intelligence 57(2-3), 143–181 (1992) 3. von Martial, F.: Coordinating Plans of Autonomous Agents. LNCS, vol. 610. Springer, Heidelberg (1992) 4. Tonino, H., Bos, A., de Weerdt, M., Witteveen, C.: Plan coordination by revision in collective agent based systems. Artificial Intelligence 142, 121–145 (2002) 5. Durfee, E.H., Lesser, V.R.: Partial global planning: A coordination framework for distributed hypothesis formation. IEEE Transactions on Systems, Man, and Cybernetics 21(5), 1167–1183 (1991) 6. Ephrati, E., Rosenschein, J.S.: Multi-agent planning as the process of merging distributed sub-plans. In: Proceedings of DAI 1993, pp. 115–129 (May 1993) 7. Lesser, V., Decker, K., Wagner, T., Carver, N., Garvey, A., Horling, B., Neiman, D., Podorozhny, R., NagendraPrasad, M., Raja, A., Vincent, R., Xuan, P., Zhang, X.: Evolution of the GPGP/TAEMS Domain-Independent Coordination Framework. Journal of AAMAS 9(1), 87–143 (2004) 8. Mors, A.T.: Coordinating autonomous planning agents. Master’s thesis, TU Delft, Delft, The Netherlands (April 2004) 9. Valk, J.: Coordination among Autonomous Planners. PhD thesis, Department of Software technology, EWI, TUDelft (2005) 10. Steenhuisen, J.R., Witteveen, C., Zhang, Y.: Plan-coordination mechanisms and the price of autonomy. In: Sadri, F., Satoh, K. (eds.) CLIMA VIII 2007. LNCS (LNAI), vol. 5056, pp. 1–21. Springer, Heidelberg (2008) 11. Kazemi, A., Zarandi, M.F., Husseini, S.M.: A multi-agent system to solve the productiondistribution planning problem for a supply chain: a genetic algorithm approach. International Journal of Advanced Manufacturing Technilogy 44, 180–193 (2009) 12. Festa, P., Pardalos, P., Resende, M.: Feedback set problems. In: Du, D.Z., Pardalos, P.M. (eds.) Handbook of Combinatorial Optimization, pp. 209–259. Kluwer Academic Publishers, Dordrecht (1999) 13. Even, G., Naor, J.S., Schieber, B., Sudan, M.: Approximating minimum feedback sets and multi-cuts in directed graphs. In: Proceedings of the 4th IIPCO Conference, London, UK, pp. 14–28. Springer, Heidelberg (1998)

Author Index

Aldewereld, Huib Barakat, Lina Cheng, Ke

14, 31

Oren, Nir

99

Payandeh, Shahram 85 Pini, Maria Silvia 44 Prettejohn, Brenton J. 114

1

Dasgupta, Prithviraj 1 Davis, Joseph G. 163 Dignum, Frank 14, 31 Dignum, Virginia 31 Dong, Yingsai 125 Gelain, Mirco 44 Georgeﬀ, Michael 68 Gerstner, Christian 56 Gharib, Hamid 68 Guttmann, Christian 68, 152 Hiel, Marcel Jain, Manish

14 137

Keppens, Jeroen 99 Keshmiri, Soheil 85 Kiekintveld, Christopher Kratzke, Nane 56 Kwak, Jun-young 137 Li, Guanyi 125 Luck, Michael 99 Ma, Ying 125 Mahmoud, Samhar 99 McDonnell, Mark D. 114 Miles, Simon 99

99

Qin, Zengchang

125

Rossi, Francesca

44

Schaﬀer, Jonathan 152 Schmidt, Heinz W. 68 Shaw, Matthew 99 Siegfried, Robert 56 Tambe, Milind 137 Taweel, Adel 99 Taylor, Matthew E. 137 Thomas, Ian 68 Thompson, Simon 68 Tranier, John 31 Venable, K. Brent

137

44

Walsh, Toby 44 Wickramasinghe, Kumari Wickramasinghe, Nilmini Witteveen, Cees 175 Wong, Waiho 163 Yadati Narasimha, Chetan Yang, Rong 137 Yin, Zhengyu 137 Zhang, Yingqian

175

68 152

175