The Iterated Prisoners’ Dilemma 20 Years On
ADVANCES IN NATURAL COMPUTATION Series Editor:
Xin Yao (University of Birmingham, UK)
Assoc. Editors: Hans-Paul Schwefel (University of Dortmund, Germany) Byoung-Tak Zhang (Seoul National University, South Korea) Martyn Amos (University of Liverpool, UK)
Published Vol. 1:
Applications of Multi-Objective Evolutionary Algorithms Eds: Carlos A. Coello Coello (CINVESTAV-IPN, Mexico) and Gary B. Lamont (Air Force Institute of Technology, USA)
Vol. 2:
Recent Advances in Simulated Evolution and Learning Eds: Kay Chen Tan (National University of Singapore, Singapore), Meng Hiot Lim (Nanyang Technological University, Singapore), Xin Yao (University of Birmingham, UK) and Lipo Wang (Nanyang Technological University, Singapore)
Vol. 3:
Recent Advances in Artificial Life Eds: H. A. Abbass (University of New South Wales, Australia), T. Bossomaier (Charles Sturt University, Australia) and J. Wiles (The University of Queensland, Australia)
Vol. 4:
The Iterated Prisoners’ Dilemma Eds: Graham Kendall (The University of Nottingham, UK) Xin Yao (The University of Birmingham, UK)
A d v a nces in Natural Computation — Vol. 4
The Iterated Prisoners’ Dilemma 20 Years On
Graham Kendall The University of Nottingham, UK
Xin Yao Siang Yew Chong The University of Birmingham, UK
World Scientific NEW JERSEY
•
LONDON
•
SINGAPORE
•
BEIJING
•
SHANGHAI
•
HONG KONG
•
TA I P E I
•
CHENNAI
Published by World Scientific Publishing Co. Pte. Ltd. 5 Toh Tuck Link, Singapore 596224 USA office: 27 Warren Street, Suite 401-402, Hackensack, NJ 07601 UK office: 57 Shelton Street, Covent Garden, London WC2H 9HE
British Library Cataloguing-in-Publication Data A catalogue record for this book is available from the British Library.
Advances in Natural Computation — Vol. 4 THE ITERATED PRISONERS’ DILEMMA 20 Years On Copyright © 2007 by World Scientific Publishing Co. Pte. Ltd. All rights reserved. This book, or parts thereof, may not be reproduced in any form or by any means, electronic or mechanical, including photocopying, recording or any information storage and retrieval system now known or to be invented, without written permission from the Publisher.
For photocopying of material in this volume, please pay a copying fee through the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, USA. In this case permission to photocopy is not required from the publisher.
ISBN-13 978-981-270-697-3 ISBN-10 981-270-697-6
Printed in Singapore.
Contents
List of Contributors
vii
Chapter 1
The Iterated Prisoner’s Dilemma: 20 Years On Siang Yew Chong, Jan Humble, Graham Kendall, Jiawei Li and Xin Yao
Chapter 2
Iterated Prisoner’s Dilemma and Evolutinary Game Theory Siang Yew Chong, Jan Humble, Graham Kendall, Jiawei Li and Xin Yao
23
Chapter 3
Learning IPD Strategies Through Co-evolution Siang Yew Chong, Jan Humble, Graham Kendall, Jiawei Li and Xin Yao
63
Chapter 4
How to Design a Strategy to Win an IPD Tournament Jiawei Li
89
Chapter 5
An Immune Adaptive Agent for the Iterated Prisoner’s Dilemma Oscar Alonso and Fernando Ni˜ no
Chapter 6
Exponential Smoothed Tit-for-Tat Michael Filzmoser
Chapter 7
Opponent Modelling, Evolution, and The Iterated Prisoner’s Dilemma Philip Hingston, Dan Dyer, Luigi Barone, Tim French and Graham Kendall
Chapter 8
On Some Winning Strategies for the Iterated Prisoner’s Dilemma Wolfgang Slany and Wolfgang Kienreich v
1
105
127
139
171
vi
Chapter 9
Chapter 10
Contents
Error-Correcting Codes for Team Coordination within a Noisy Iterated Prisoner’s Dilemma Tournament Alex Rogers, Rajdeep K. Dash, Sarvapali D. Ramchurn, Perukrishnen Vytelingum and Nicholas R. Jennings Is it Accidental or Intentional? A Symbolic Approach to the Noisy Iterated Prisoner’s Dilemma Tsz-Chiu Au and Dana Nau
205
231
List of Contributors
Oscar Alonso, Computer Systems and Industrial Engineering Department, National University of Colombia, Bogota Colombia Email:
[email protected] Tsz-Chiu Au, Department of Computer Science and Institute for Systems Research, University of Maryland, College Park, MD 20742 USA Email:
[email protected] Luigi Barone, Department of Computer Science and Software Engineering, The Univesity of Western Australia, 35 Stirling Highway, Crawley, WA, 6009 Australia Email:
[email protected] Siang Yew Chong, School of Computer Science, University of Birmingham, Birmingham, B15 2TT UK Email:
[email protected]
vii
viii
List of Contributors
Rajdeep K. Dash, Electronics and Computer Science, University of Southampton, Southampton, SO17 1BJ UK Email:
[email protected] Dan Dyer, Department of Computer Science and Software Engineering, The University of Western Australia, 35 Stirling Highway, Crawley, WA, 6009 Australia Email:
[email protected] Michael Filzmoser, School of Business Administration, Economics, and Statistics, University of Vienna, Vienna, A-1210 Austria Email:
[email protected] Tim French, Department of Computer Science and Software Engineering, The University of Western Australia, 35 Stirling Highway, Crawley, WA, 6009 Australia Email:
[email protected] Philip Hingston, School of Computer and Information Science, Edith Cowan University - Mt Lawley Campus, 2 Bradford Street, Mt Lawley, WA 6050 Australia Email:
[email protected]
List of Contributors
Jan Humble, School of Computer Science and Information Technology, University of Nottingham, Nottingham, NG8 1BB UK Email:
[email protected] Nicholas R. Jennings, Electronics and Computer Science, University of Southampton, Southampton, SO17 1BJ, UK Email:
[email protected] Graham Kendall, School of Computer Science and Information Technology, University of Nottingham, Nottingham, NG8 1BB UK Email:
[email protected] Wolfgang Kienreich, Know-Center, Inffeldgasse 21a/II, 8010 Graz Austria Email:
[email protected] Jiawei Li, Robot Institute, Harbin Institute of Technology, Heilongjiang, 150001, P. R. China Email: lijiawei
[email protected] Dana Nau, Department of Computer Science and Institute for Systems Research, University of Maryland, College Park, MD 20742 USA Email:
[email protected]
ix
x
List of Contributors
Fernando Ni˜ no, Computer Systems and Industrial Engineering Department, National University of Colombia, Bogota Colombia Email:
[email protected] Sarvapali D. Ramchurn, Electronics and Computer Science, University of Southampton, Southampton, SO17 1BJ UK Email:
[email protected] Alex Rogers, Electronics and Computer Science, University of Southampton, Southampton, SO17 1BJ UK Email:
[email protected] Wolfgang Slany, Institut f¨ ur Softwaretechnologie, Inffeldgasse 16b/II, TU Graz, A-8010 Graz Austria Email:
[email protected] Perukrishnen Vytelingum, Electronics and Computer Science, University of Southampton, Southampton, SO17 1BJ UK Email:
[email protected] Xin Yao, School of Computer Science, University of Birmingham, Birmingham, B15 2TT UK Email:
[email protected]
Chapter 1 The Iterated Prisoner’s Dilemma: 20 Years On
Siang Yew Chong1 , Jan Humble2 , Graham Kendall2 , Jiawei Li2,3 , Xin Yao1 University of Birmingham1 , University of Nottingham2 , Harbin Institute of Technology3
1.1. Introduction In 1984, Robert Axelrod reported the results of two iterated prisoner’s dilemma (IPD) competitions [Axelrod (1984)]. The booked was to be a catalyst for much of the research in this area since that time. It is unlikely that you would write a scientific paper about IPD, without citing Axelrod’s 1984 book. The book is even more remarkable in that it is just as accessible to a general audience, as well as being an important source of inspiration for the scientific community. In 2001, whilst attending the Congress on Evolutionary Computation (CEC) conference, we were discussing some of the presentations we had seen which reported recent some of the latest work on the iterated prisoner’s dilemma. We were paying tribute to the fact that Axelrod’s book had stood the test of time when somebody made a casual comment suggesting that we should re-run the competition in 2004, to celebrate the 20th anniversary. And, so, this book was born. Of course, since the conversation in Hawaii and the publication of this book, there have been a lot of people doing a lot of work. Not least of all Robert Axelrod who was good enough to give up his time to present a plenary talk at the CEC conference in 2004. At that talk he presented his latest work which is investigating evolution on a grid based world. We owe a debt of thanks to the UK’s EPSRC (Engineering and Physical Sciences Research Council). This is the largest of the UK research councils which funds research in the UK. When we returned from Hawaii, we 1
2
S. Y. Chong et al.
submitted a proposal,a which requested a small amount of funds (£23,718) in order to re-run, and extend, the competitions that Axelrod had run 20 years earlier. The funds we received from EPSRC allowed us to run two competitions, one in 2004 and one in 2005. The entrants to the competitions were invited to submit a chapter for consideration in this book. These chapters underwent a peer review process (see later in this chapter for an acknowledgement of the reviewers) and those chapters that were successful form the latter part of this book. As editors, we feel fortunate to have several winners, second and third place entries reported in this book. This affords the reader the opportunity to learn, first hand from the authors, what made these strategies so successful and, perhaps, use some of the ideas and innovations in their own strategies for future competitions.
1.2. Iterated Prisoner’s Dilemma Almost every chapter in this book has its own description of the iterated prisoner’s dilemma. As each chapter can be read in isolation and, for completeness, we present our own interpretation of the IPD here, along with a short review of some of the important work in the area. The prisoner’s dilemma (PD) and iterated prisoners dilemma (IPD) have been a rich source of research material since the 1950’s. However, the publication of Axelrod’s book [Axelrod (1984)] in the 1980’s was largely responsible for bringing this research to the attention to other areas, outside of game theory, including evolutionary computing, evolutionary biology, networked computer systems and promoting cooperation between opposing countries [Goldstein (1991); Fogel (1993); Axelrod and D’Ambrosio (1995)]. Despite the large literature base that now exists (see, for example, [Poundstone (1992); Boyd and Lorberbaum (1987); Maynard Smith (1982); Davis (1997), Axelrod (1997)], this is an on-going area of research, with Darwen and Yao [Darwen and Yao (1995, 2001); Yao and Darwen (1999)] carrying out some recent work. Their 2001 work [Darwen and Yao (2001)] extends the prisoner’s dilemma by offering more choices, other than simply “cooperate” or “defect,” and by providing indirect interactions (reputation). When you play the prisoner’s dilemma you have to decide whether to cooperate with an opponent, or defect. Both you and your opponent make a a The
EPSRC grant reference numbers are GR/S63465/01 and GR/S63472/01.
The Iterated Prisoner’s Dilemma: 20 Years On
3
choice and then your decisions are revealed. You receive a payoff according to the following matrix (where the top line is the payoff to the column).
Cooperate Defect
Cooperate R=3 R=3 S=0 T =5
Defect T =5 S=0 P =1 P =1
• R is a Reward for mutual cooperation. Therefore, if both players cooperate then both receive a reward of 3 points. • If one player defects and the other cooperates then one player receives the T emptation to defect payoff (5 in this case) and the other player (the cooperator) receives the Sucker payoff (zero in this case). • If both players defect then they both receive the P unishment for mutual defection payoff (1 in this case). The question arises: what should you do in such a game? • Suppose you think the other player will cooperate. If you cooperate then you will receive a payoff of 3 for mutual cooperation. If you defect then you will receive a payoff of 5 for the Temptation to Defect payoff. Therefore, if you think the other player will cooperate then you should defect, to give you a payoff of 5. • But what if you think the other player will defect? If you cooperate, then you get the Sucker payoff of zero. If you defect then you would both receive the Punishment for Mutual Defection of 1 point. Therefore, if you think the other player will defect, you should defect as well. So, you should defect, no matter what option your opponent chooses. Of course, the same logic holds for your opponent. And, if you both defect you receive a payoff of 1 each, whereas, the better outcome would have been mutual cooperation with a payoff of 3. The payoff for an individual is less than that could have been achieved by two cooperating players, thus the dilemma and the research challenge of finding strategies that promote mutual cooperation. In defining a prisoner’s dilemma, certain conditions have to hold. The values we used above, to demonstrate the game, are not the only values that could have been used, but they do have to adhere to the conditions listed below.
4
S. Y. Chong et al.
Firstly, the order of the payoffs is important. The best a player can do is T (temptation to defect). The worst a player can do is to get the sucker payoff, S. If the two players cooperate then the reward for that mutual cooperation, R, should be better than the punishment for mutual defection, P . Therefore, the following must hold. T >R>P >S.
(1.1)
Secondly, players should not be allowed to get out of the dilemma by taking it in turns to exploit each other. Or, to be a little more precise, the players should not play the game so that they end up with half the time being exploited and the other half of the time exploiting their opponent. In other words, an even chance of being exploited or doing the exploiting is not as good an outcome as both players mutually cooperating. Therefore, the reward for mutual cooperation should be greater than the average of the payoff for the temptation and the sucker. That is, the following must hold. R > (S + T )/2 .
(1.2)
Playing a “one-shot” prisoners dilemma, it is not difficult to decide which strategy to adopt, but the question arises: can cooperation evolve from playing the game over and over again, against the same opponent? If you know how many times you are to play, then there is an argument that the game is exactly the same as playing the “one-shot” prisoners dilemma. This is based on the observation that you will defect on the last iteration as that is the sensible thing to do as, you are in effect playing a single iteration. Knowing this, it is sensible to defect on the second to last one as well; and this logic can be applied all the way to the first iteration. However, this reasoning cannot be used when the number of iterations is infinite as you know there is always another iteration. In practise, this translates to not knowing when the game will end. Experiments, using human players [Scodel (1962, 1963); Minas et al. (1960); Scodel and Philburn (1959), Scodel et al. (1959); Scodel et al. (1960)] showed that they, generally, did not cooperate even when it should have been obvious that the other person was going to cooperate, just as long as you do. It has been a long term aim to find strategies which causes players to cooperate. If players would only cooperate then their payoff, over an indefinite number of games could be maximised, rather than tending towards defection and hoping the other player would cooperate. In 1979 Axelrod organised a prisoner’s dilemma competition and invited game theorists to
The Iterated Prisoner’s Dilemma: 20 Years On
5
submit their strategies [Axelrod (1980a)]. Fourteen entries were received with an extra one being added (defect or cooperate with equal probability). The strategies were competed against each other, including itself. The winner was Anatol Rapoport who submitted the simple strategy (Tit-forTat) which cooperates on the first move, then does whatever your opponent did on the previous move. In a second tournament [Axelrod (1980b)], 62 entries were received but, again, the winner was Tit-for-Tat. These two competitions formed the basis of his important book [Axelrod (1984)]. The prisoners dilemma has a modern day version in the form of the TV show “Shafted” - a game show recently screened on terrestrial TV in the UK (note that this show is not a true prisoners dilemma as defined by Rapoport [Rapoport (1996)], but does demonstrate that the ideas have wider applicability). At the end of the show two contestants have accumulated a sum of money and they have to decide if to share the money or to try and get all the money for themselves. Their decision is made without the knowledge of what the other person has decided to do. If both contestants cooperate then they share the money. If they both defect then they both receive nothing. If one cooperates and the other defects, the one that defected gets all the money and the contestant that cooperated gets nothing. Although the prisoners dilemma, in the context of game theory, has been an active research area for at least 50 [Scodel (1962); Scodel (1963); Minas et al. (1960); Scodel and Philburn (1959); Scodel et al. (1959); Scodel et al. (1960)] years (it can be traced back to von Neumann and Morgenstern [von Neumann and Morgenstern (1944)] and, of course, John Nash [Nash (1950, 1953)]), it is still an active research area with, among other research aims, researchers trying to evolve strategies [O’Riordan (2000)] that promote cooperation. Recent research has also considered the prisoner’s dilemma where there are more than two choices and more than two players. Darwen and Yao have shown that offering more choices leads to less cooperation [Darwen and Yao (2001)], although reputation may help [Darwen and Yao (2002); Yao and Darwen (1999)]. Birk [Birk (1999)] used a multi-payer IPD. His model had continuous degrees of cooperation (as opposed to the binary; cooperate or defect). He used a robotic environment and showed that a justifiedsnobism strategy, that tries to cooperate slightly more than the average, is a successful strategy and is evolutionarily stable (that is, it cannot be invaded by another strategy). O’Riordan and Bradish (2000) also simulated a multi-player game where the players are involved in many types of games.
6
S. Y. Chong et al.
They show that cooperation can emerge in a high percentage of 2-player games. As well as the academic papers on the subject, there are many books devoted to game theory and/or the prisoners dilemma. The 1997 book by Axelrod (1997) re-produces a range of his papers (with commentary) ranging from 1986 through to 1997. The papers consider areas such as promoting cooperation using a genetic algorithm, coping with noise and promoting norms.
1.3. Contents of the Book This book does not have to be read from cover to cover. Each chapter can be read independently, with most of the chapters describing the IPD. This was a conscious decision by the editors as we realised that the book would be dipped into and we did not want to make any chapter dependent on any other. Also, each chapter has its own set of references, rather than having one complete list of references at the end of the book. The book is structured as follows Chapter 1 This chapter provides a general introduction to the book. In keeping with the rest of the book, we also briefly describe the IPD. As well as briefly describing each chapter. This chapter also presents the results of the two competitions that we ran in 2004 and 2005. Chapter 2 Chapter 2 (“Iterated Prisoner’s Dilemma and Evolutionary Game Theory”) reviews some of the important work in IPD, with particular emphasis (in the latter part of the chapter) on evolutionary game theory. The chapter contains over 250 references, which we hope will be a good starting point for other researchers who are looking to start work in this area. We have concentrated on the evolutionary aspects of IPD for two reasons. Firstly, this seemed to be an area that was exploited in the entries we received. Secondly, the literature on IPD is truly vast (perhaps only exceeded by literature on the traveling salesman problem), and we had to draw some boundaries and, given the close links that this competition had
The Iterated Prisoner’s Dilemma: 20 Years On
7
with the Congress on Evolutionary Computation, it seemed appropriate to report on the evolutionary aspects of IPD. We apologise to any authors who feel their work should have been included in this chapter. We hope you understand that we simply could not list every paper. However, if you would like to drop us an EMAIL, we would be happy to consider the inclusion of the reference in any later editions. Chapter 3 Chapter 3 (“Learning IPD Strategies Through Co-evolution”) reviews another area of IPD that has received scientific interest in recent years; that of co-evolution. This chapter also discusses an extension to the classic IPD formulation. That is when there are more than two players and when they have more than two choices. Similar to chapter two, there is an extensive list of references for the interested reader. Chapter 4 This chapter reports the winning strategy from competition 4, from the event held in 2005. This competition mimics the original ones held by Axelrod. Only one entry was allowed per person, to stop the cooperating strategies that had dominated the first competition. Although we believe that having cooperating strategies is a valid tactic, some competitors felt that this did not truly mimic the original competitions. For this reason we introduced an additional competition for the 2005 event. The result was a win for Jiawei Li, who details his winning strategy in chapter 4, which is entitled How to Design a Strategy to Win an IPD Tournament. Chapter 5 The strategy in this chapter attempts to model its opponent using an artificial immune system. It is interesting to see how relatively new methodologies are being used for problems such as IPD, demonstrating that there is a continuous flow of new ideas which might just be shown to be superior to all other methods so far. Whilst not appearing in the top ten of any of the competitions that it entered, it does present an exciting new research direction for IPD tournaments.
8
S. Y. Chong et al.
Chapter 6 Michael Filzmoser, reports on a variation of tit-for-tat, which he calls Exponential Smoothed Tit-for-Tat. Whereas tit-for-tat only considers the last move of the opponent, exponential smoothed tit-for-tat considers the complete history of the opponent. This discussion is extended to IPD with noise, as well as the more common IPD, where the actions by the player are reliably reported. Chapter 7 In chapter 7 (“Opponent modelling, Evolution, and the Iterated Prisoner’s Dilemma”), the authors explore the idea of modeling an opponent. It does this by playing tit-for-tat for the first 50 moves, whilst trying to model the moves played by the opponent. After 50 moves, subsequent moves are then based on the model that has been built. It is interesting to compare this strategy (which came 3rd in competition 4 in 2005), with the strategy described in chapter 4, which also uses a type of modeling but over a shorter time period. Perhaps this explains why it was able to achieve better payoffs, as it was able to exploit opponents much earlier in the game? Chapter 8 The strategies reported in this chapter were entered in both the 2004 and 2005 events, and performed well in many of the competitions, winning competition 1 in the 2005 event. This chapter, more than any other, touches on the debate about cooperating strategies, which is why we introduced competition 4 in the 2005 event. If you followed the discussion at the time, many entrants (with some justification) questioned if allowing multiple strategies from one person was in the spirit of the original Axelrod competitions. Whilst we agreed with this, so introduced a single entry rule in 2005, we also argue that these competitions were about the research that was being carried out and some of the chapters in this book report on those results. Of course, as the authors of chapter 8 admit, there are still ways of flouting the rules by submitting cooperating entries under different names. We hope that the other entrants will accept this in the spirit of research under which this was done. As the authors point out, the organisers failed to recognise that
The Iterated Prisoner’s Dilemma: 20 Years On
9
cooperating strategies had been submitted, but, as they also say, this is a theoretically difficult problem. We would also like to take this opportunity to the authors of chapter 8 for missing their OTFT strategy from some of the competitions. It is still unclear to us why this happened. Chapter 9 A team from Southampton, who took the first three places in competition 1, in the 2004 competition present chapter 9. Their chapter is an excellent example of how strategies can cooperate. As strategies have no mechanism to interact directly, the only way to recognise one of your collaborators is to somehow communicate through the defect/cooperate choices that you make. Chapter 10 One of the competitions that we run included noise, with some low probability. By noise, we mean that a defect or cooperate signal might be misinterpreted. This final chapter by Tsz-Chiu Au and Dana Nau explores this issue using a strategy they call Derived Belief Strategy. It attempts to model their opponent and then judge if their choice has been affected by noise. They performed very well in the competition, even when up against strategies which were cooperating.
1.4. Celebrating the 20th Anniversary: The Competitions We ran two events. The first was held during the Congress of Evolutionary Computation Conference in 2004 (June 19-23, Portland, Oregon, USA) and the next at the Computational Intelligence and Games Conference in 2005 (April 4-6, 2005, Essex UK). At the 2004 event we ran three competitions, with an additional competition being held in 2005. (1) The first competition aimed to emulate the original Axelrod competition. We received some enquiries about whether multiple entries were allowed. As we had not stated this as a restriction, we allowed it (but did state we had the right to limit the number, else running the competition may become intractable). At the time, we did not realise the
S. Y. Chong et al.
10
controversy that this decision would cause, which is why we modified the competitions in the 2005 event. (2) The second competition had noise in it. Each decision had a 0.1 probability of being mis-interpreted. (3) The third competition allowed competitors to submit a strategy to an IPD that has more than one player and more than one payoff, that is, multi player and multi-choice. (4) The fourth competition (which was only run in 2005) emulated the original Axelrod competition. The definition was exactly the same as competition 1, but we only allowed one entry per person. The payoff table we used for competitions 1, 2 and 4 is shown in table 1.1. The payoff table for competition 3 is shown in table 1.2. Table 1.1. Payoff table for all IPD competitions except for the IPD with multiple players and multiple choices.
Cooperate Defect
Cooperate R=3 R=3 S=0 T =5
Defect T =5 S=0 P =1 P =1
Table 1.2. Payoff table for IPD competition with multiple players and multiple payoffs Player BLevels of Cooperation. Player B Levels of Cooperation
1
3 4
1 2
1 4
0
1
4
3
2
1
0
3 4 1 2 1 4
4 14
3 14
2 14
1 14
4 12
3 12
2 12
1 12
4 34
3 34
2 34
0
5
4
3
1 34 2
1 4 1 2 3 4
Player A
1
To support the competitions, we developed a software framework. This is discussed in the Appendix, and a URL is supplied so that the software can be downloaded. 1.5. Competition Results In the following tables we present the top ten entries from each of the competitions. The full listings of the results can be seen at
The Iterated Prisoner’s Dilemma: 20 Years On
11
http://www.prisoners-dilemma.com. Also available on the web site is a log containing all the interactions that took place.
Table 1.3. Results from 2004 event, competition 1. There were 223 entries (19 web based entries, 195 java based entries and 9 standard entries (RAND, NEG, ALLC, ALLD, TFT, STFT, TFTT, GRIM, Pavlov)). Rank 1 2 3
4
5 6 7 8
Strategy
Won
Drawn
Lost
Total Points
StarSN (StarSN)
105
21
98
117,057
StarS (StarS)
113
48
63
110,611
StarSL (StarSL)
115
46
63
110,511
GRIM (GRIM Trigger)
120
76
28
100,611
90
70
64
100,604
Player Gopal Ramchurn Gopal Ramchurn Gopal Ramchurn GRIM (GRIM Trigger) 1 Wolfgang Kienreich Wolfgang Kienreich Wolfgang Kienreich Emp 1 Bingzhong Wang
9
Hannes Payer
10
Nanlin Jin
OTFT (Omega tit for tat) ADEPT (ADEPT Strategy) EMP (Emperor)
95
72
57
96,291
90
73
61
95,927
()
31
94
99
94,161
95
75
54
94,123
27
95
102
93,953
PRobbary (PRobbary Historylength 2) HCO (HCO)
S. Y. Chong et al.
12
Table 1.4. Results from 2004 event, competition 2. There were 223 entries (19 web based entries, 195 java based entries and 9 standard entries (RAND, NEG, ALLC, ALLD, TFT, STFT, TFTT, GRIM, Pavlov)). Rank 1 2 3 4 5 6 7 8 9 10
Won
Drawn
Lost
Total Points
StarSN (StarSN)
42
2
180
93,962
Mem1 (Mem1)
5
1
218
83,049
CoordinateCDCSIAN (CoordinateCDCSIAN)
158
6
60
83,015
PoorD (PoorD)
190
7
27
82,890
OTFT (Omega tit for tat)
158
8
58
82,838
ltft (ltft)
66
8
150
82,765
GRIM (GRIM Trigger)
184
7
33
82,591
MooD (MooD)
193
3
28
82,578
AITFT (AITFT)
60
9
155
82,504
GSTFT (GSTFT)
64
9
151
82,502
Player Gopal Ramchurn Colm O’Riordan Gopal Ramchurn Gopal Ramchurn Wolfgang Kienreich Wayne Davis GRIM (GRIM Trigger) 1 Gopal Ramchurn Gopal Ramchurn Gopal Ramchurn
Strategy
The Iterated Prisoner’s Dilemma: 20 Years On
13
Table 1.5. Results from 2004 event, competition 3. There were 15 entries. Note that there is only one round in this competition. Rank 1 2 3 4 5 6 7 8 9 10
Player Gopal Ramchurn Gopal Ramchurn Deirdre Murrihy Deirdre Murrihy Deirdre Murrihy Enda Howley Enda Howley Enda Howley Wolfgang Kienreich Wolfgang Kienreich
Strategy
Total Points
AgentSoton (SOTON AGENT)
3,756
HarshTFT (HarshTFT)
3,756
PCurvepower1Memory2 (Penalty Curve of 1 using opponent’s previous 2 moves) PCurvepower2Memory2 (Penalty Curve of 2 using opponent’s previous 2 moves) PCurvepower0.5Memory2 (Penalty Curve of 0.5 using opponent’s previous 2 moves) PCurvepower2 (Penalty Curve of 2 using opponent’s previous move) PCurvepower1 (Penalty Curve of 1 using opponent’s previous move) PCurvepower0.5 (Penalty Curve of 0.5 using opponent’s previous move)
3,738 3,738 3,738 3,738 3,738 3,738
CNHM (CosaNostra Hitman)
3,738
CNHM (CosaNostra Hitman)
3,738
S. Y. Chong et al.
14
Table 1.6. Results from 2005 event, competition 1. There were 192 entries (41 web based entries, 142 java based entries and 9 standard entries (RAND, NEG, ALLC, ALLD, TFT, STFT, TFTT, GRIM, Pavlov)). Rank
Player
1
Wolfgang Kienreich
2
Jia-wei Li
3
Carlos G. Tardon
4
Perukrishnen Vytelingum
5 6
Constantin Ionescu GRIM (GRIM Trigger) 1
7
Tsz-Chiu Au
8
Tsz-Chiu Au
9 10
Richard Brunauer Carlos G. Tardon
Strategy CNGF (CosaNostra Godfather) IMM01 (Intelligent Machine Master 01) CLAS- (CLAS-) SWIN (Soton Agent RA Competition 1) LORD (the lord strategy) GRIM (GRIM Trigger) LSF (Learning of opponent strategy with forgiveness) DBStft (DBS with TFT) PRobberyL2 (PRobberyL2) CLAS2 (CLAS2)
Won
Drawn
Lost
Total Points
48
96
49
100,905
46
112
35
98,922
23
95
75
92,174
61
44
88
90,918
20
102
71
87,617
73
114
6
84,805
28
94
71
84,698
23
97
73
83,867
14
98
81
83,837
72
96
25
83,746
The Iterated Prisoner’s Dilemma: 20 Years On
15
Table 1.7. Results from 2005 event, competition 2. There were 165 entries (26 web based entries, 130 java based entries and 9 standard entries (RAND, NEG, ALLC, ALLD, TFT, STFT, TFTT, GRIM, Pavlov)). Rank
Player
1
Perukrishnen Vytelingum
2
Jia-wei Li
3
Tsz-Chiu Au
4
Tsz-Chiu Au
5
Tsz-Chiu Au
6
Tsz-Chiu Au
7
Tsz-Chiu Au
8
Tsz-Chiu Au
9
Tsz-Chiu Au
10
Tsz-Chiu Au
Strategy BWIN (S2Agent1 ZEUS Competition 2) IMM01 (Intelligent Machine Master 01) DBSy (DBS (version y)) DBSz (DBS (version z)) DBSpl (DBS with learning prevention) DBSd (Derivative Belief Strategy (version d)) DBSx (DBS (version x)) TFTIc (TFT improved (ver. c)) DBSf (Derivative Belief Strategy (version f)) TFTIm (TFT improved (ver. m))
Won
Drawn
Lost
Total Points
85
1
80
73,330
108
7
51
70,506
35
3
128
68,370
27
3
136
68,339
37
2
127
67,979
42
6
118
67,392
19
9
138
66,719
41
4
121
66,409
48
2
116
66,269
38
3
125
66,239
S. Y. Chong et al.
16
Table 1.8. Results from 2005 event, competition 3. There were 34 entries. Note that there is only one round in this competition. Rank 1 2 3
Player Perukrishnen Vytelingum Deirdre Murrihy Deirdre Murrihy
4
Deirdre Murrihy
5
Enda Howley
6
Enda Howley
7
Enda Howley
8 9 10
Wolfgang Kienreich Wolfgang Kienreich Wolfgang Kienreich
Strategy $AgentSoton ($SOTON AGENT) PCurvepower1Memory2 (Penalty Curve of 1 using opponent’s previous 2 moves) PCurvepower2Memory2 (Penalty Curve of 2 using opponent’s previous 2 moves) PCurvepower0.5Memory2 (Penalty Curve of 0.5 using opponent’s previous 2 moves) PCurvepower2 (Penalty Curve of 2 using opponent’s previous move) PCurvepower1 (Penalty Curve of 1 using opponent’s previous move) PCurvepower0.5 (Penalty Curve of 0.5 using opponent’s previous move)
Total Points 7,558 7,521 7,521 7,521 7,521 7,521 7,521
CNHM (CosaNostra Hitman)
7,521
CNHM (CosaNostra Hitman)
7,521
CNHM (CosaNostra Hitman)
7,521
The Iterated Prisoner’s Dilemma: 20 Years On
17
Table 1.9. Results from 2005 event, competition 4. There were 50 entries (26 web based entries, 15 java based entries and 9 standard entries (RAND, NEG, ALLC, ALLD, TFT, STFT, TFTT, GRIM, Pavlov)). Rank 1 2 3 4 5 6 7 8 9 10
Player Jia-wei Li Wolfgang Kienreich Philip HingstonMod Bruno Beaufils Tim Romberg Richard Brunauer Hannes Payer Bennett McElwee Gerhard Mitterlechner Wayne Davis
Won
Drawn
Lost
Total Points
11
34
6
30,096
9
36
6
29,554
(Modeller)
7
36
8
29,003
GRAD (Gradual)
8
32
11
28,707
13
32
6
28,692
12
32
7
28,523
11
33
7
28,292
22
11
18
28,110
11
32
8
27,893
1
44
6
27,834
Strategy APavlov (Adaptive Pavlov) OTFT (Omega tit for tat)
tro1 (tro1) DETerminatorL6C4 (DETerminatorL6C4) DETerminatorL4C4 (DETerminatorL4C4) LOOKDB (LookaheadDB) PRobberyM5C4 (PRobberyM5C4) ltft (ltft)
S. Y. Chong et al.
18
1.6. Acknowledgements We would like to thank the following people who acted as reviewers for the chapters in this book. • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • •
Muhammad A. Ahmad Oscar Alonso Dan Ashlock Tsz-Chiu Au Carlos Eduardo Rodriguez Calderon Michel Charpentier Wayne Davis J¨ org Denzinger Eugene Eberbach Michael Filzmoser Nelis Franken Nicholas Gessler Michal Glomba Philip Hingston Enda Howley Nick Jennings Nanlin Jin Jacint Jordana Wolfgang Kienreich Eun-Youn Kim Jia-wei Li Helmut A. Mayer Bennett McElwee Gerhard Mitterlechner Colm O’Riordan Sarvapali Ramchurn, Alex Rogers Tim Romberg Darryl A. Seale Wolfgang Slany Elpida Tzafestas Perukrishnen Vytelingum Georgios N. Yannakakis Lukas Zebedin
The Iterated Prisoner’s Dilemma: 20 Years On
19
Appendix: Software Framework A software library and corresponding application was developed to easily implement prisoner’s dilemma strategies and tournament competitions between populations of these. Although a vast array of software is available for the same purpose they did not contain all our feature requirements. For several of our experiments we required a game engine that would, among other things, handle a continuous [normalised] range of moves, arbitrarily sized payoff matrices, different types of signal noise, multiple (> 2) strategies per game, and logging of partial and completed game results. The software suite was developed in Java, allowing ease in development and web deployment. New strategies are easily implemented by implementing a subclass of the Strategy class. The principal requirements are the implementations of the getMove() and reset() methods which returns the current strategy move and clears the strategy state between games respectively. Currently we define two types of games: standard and multi-player. A standard game involves two competing strategies playing for a number of rounds, and should mimic the basic game mechanics in the competitions run by Axelrod. A multi-player game involves several competing strategies obtaining payoffs for every other opponent it plays against on each round. A tournament involves every participating strategy and differs for standard and multi-player type games. A standard tournament pits every strategy against every other (including self) in a standard game [a la Round Robin]. A multi-player tournament plays a single multi-player game. An option is available to introduce a Gaussian distributed random number of rounds to be played, so as to discourage strategies from using the knowledge of a predefined or static parameter for an unfair advantage. There is also an option to introduce noise into the output moves, in principle to test the robustness of the algorithms. Besides the programming API, a graphical user interface is available to set up and run PD tournament competitions (see Figure 1.1). The software monitors and allows users to log the output of a tournament with different degrees of detail. However, detailed logs will degrade performance. Besides the standard 2 × 2 payoff matrix for classic games, there is the ability to define an arbitrarily sized payoff matrix allowing for a wider range of allowable moves. Moves are normalised and payoffs are calculated from the closest allowable move in the payoff matrix.
S. Y. Chong et al.
20
Fig. 1.1.
IPD tournament application.
A number of standard classic strategies are included in the library. The software can be downloaded for http://prisoners-dilemma.com. References Axelrod, R. (1980a). Effective Choices in the Prisoner’s Dilemma, J. Conflict Resolution, 24, pp. 3-25. Axelrod, R. (1980b). More Effective Choices in the Prisoner’s Dilemma, J. Conflict Resolution, 24, pp. 379-403. Axelrod R. M. (1984). The Evolution of Cooperation (BASIC Books, New York). Axelrod R. and D’Ambrosio L. (1995). Announcement for Bibliography on the Evolution of Cooperation, Journal of Conflict Resolution 39, pp. 190. Axelrod R. (1997). The Compleity of Cooperation (Princeton University Press). Birk A. (1999). Evolution of Continuous Degrees of Cooperations in an N-Player Iterated Prisoner’s Dilemma, Technical Report under review, Vrije Universiteit Brussel, AI-Laboratory. Boyd R. and Lorberbaum J. P. (1987). No Pure Strategy is Evolutionary Stable in the Repeated Prisoner’s Dilemma, Nature, 327, pp. 58-59. Darwen P and Yao X. (2002). Co-Evolution in Iterated Prisoners Dilemma with Intermediate Levels of Cooperation: Application to Missile Defense, International Journal of Computational Intelligence and Applications, 2, 1, pp. 83-107. Darwen P. and Yao X. (1995). On Evolving Robust Strategies for Iterated Prisoners Dilemma, In Progress in Evolutionary Computation, LNAI, 956, pp. 276-292.
The Iterated Prisoner’s Dilemma: 20 Years On
21
Darwen P. and Yao X. (2001). Why More Choices Cause Less Cooperation in Iterated Prisoner’s Dilemma, Proc. Congress of Evolutionary Computation, pp. 987-994. Davis M. Game Theory. (1997). A Nontechnical Introduction (Dover Publications). Fogel D. (1993). Evolving Behaviours in the Iterated Prisoners Dilemma. Evolutionary Computation, 1, 1, pp. 77-97. Goldstein J. (1991). Reciprocity in Superpower Relations: An Empirical Analysis, International Studies Quarterly, 35, pp. 195-209. Maynard Smith J. (1982). Evolution and the Theory of Games (Cambridge University Press). Minas J. S., Scodel A., Marlowe D. and Rawson H. (1960). Some Descriptive Aspects of Two-Person, Non-Zero-Sum Games, II, Journal of Conflict Resolution, 4, pp. 193-197. Nash J. (1950). The Bargaining Problem, Econometrica, 18, pp. 150-155. Nash J. (1953). Two-Person Cooperative Games, Econometrica, 21, pp. 128-140. O’Riordan and Bradish S. (2000). Experiments in the Iterated Prisoner’s Dilemma and the Voter’s Paradox. 11th Irish Conference on Artificial Intelligence and Cognitive Science. O’Riordan C. (2000). A Forgiving Strategy for the Iterated Prisoner’s Dilemma, Journal of Artificial Societies and Social Simulation, 3, 1. Poundstone W. (1992). Prisoner’s Dilemma, Doubleday Rapoport A. (1996). Optimal policies for the prisoners dilemma, Tech report No. 50, Psychometric Laboratory, Univ. North Carolina, NIH Grant, MH10006. Scodel A. and Philburn R. (1959). Some Personality Correlates of Decision Making under Conditions of Risk, Behavioral Science, 4, pp. 19-28. Scodel A., Minas J. S., Ratoosh P.and Lipetz M. (1959). Some Descriptive Aspects of Two-Person, Non-Zero-Sum Games, Journal of Conflict Resolution, 3, pp. 114-119. Scodel A. and Minas J. S. (1960). The Behavior of Prisoners in a “Prisoner’s Dilemma” Game, Journal of Psychology, 50, pp. 133-138. Scodel A. (1962). Induced Collaboration in Some Non-Zero-Sum Games, Journal of Conflict Resolution, 6, pp. 335-340. Scodel A. (1963). Probability Preferences and Expected Values. Journal of Psychology, 56, pp. 429-434. von Neumann J. and Morgenstern O. (1944). Theory of Games and Economic Behavior (Princeton University Press). Yao, X and Darwen P. (1999). How Important is Your Reputation in a MultiAgent Environment. Proc. Of the 1999 IEEE Conference on Systems, Man and Cybernetics, IEEE Press, Piscataway, NJ, USA, pp. II-575 – II-580, Oct.
This page intentionally left blank
Chapter 2 Iterated Prisoner’s Dilemma and Evolutionary Game Theory Siang Yew Chong1 , Jan Humble2 , Graham Kendall2 , Jiawei Li2,3 , Xin Yao1 University of Birmingham1 , University of Nottingham2 , Harbin Institute of Technology3
2.1. Introduction The prisoner’s dilemma is a type of non-zero-sum game in which two players try to maximize their payoff by cooperating with, or betraying the other player. The term non-zero-sum indicates that whatever benefits accrue to one player do not necessarily imply similar penalties imposed on the other player. The Prisoner’s dilemma was originally framed by Merrill Flood and Melvin Dresher working at RAND Corporation in 1950. Albert W. Tucker formalized the game with prison sentence payoffs and gave it the “Prisoner’s Dilemma” name. The classical prisoner’s dilemma (PD) is as follows: Two suspects, A and B, are arrested by the police. The police have insufficient evidence for a conviction, and, having separated both prisoners, visit each of them to offer the same deal: if one testifies for the prosecution against the other and the other remains silent, the betrayer goes free and the silent accomplice receives the full 10-year sentence. If both stay silent, the police can sentence both prisoners to only six months in jail for a minor charge. If each betrays the other, each will receive a two-year sentence. Each prisoner must make the choice of whether to betray the other or to remain silent. However, neither prisoner knows for sure what choice the other prisoner will make. So the question this dilemma poses is: What will happen? How will the prisoners act?
The general form of the PD is represented as the following matrix [Scodel et al. (1959)]: 23
S. Y. Chong et al.
24
Prisoner 1
Cooperate Defect
Prisoner 2 Cooperate Defect (R, R) (S, T ) (T, S) (P, P )
where R, S, T , and P denote Reward for mutual cooperation, Sucker’s payoff, Temptation to defect, and Punishment for mutual defection respectively, and T > R > P > S and R > 1/2(S + T ). The two constraints motivate each player to play noncooperatively and prevent any incentive to alternate between cooperation and defection [Rapoport (1966, 1999)]. Neither prisoner knows the choice of his accomplice. Even if they were able to talk to each other, neither could be sure that he could trust the other. The “dilemma” faced by the prisoners here is that, whatever the other does, each is better off confessing than remaining silent. However, the payoff when both confess is worse for each player than the outcome they would have received if they had both remained silent. Traditional game theory predicts the outcome of PD be mutual defection based on the concept of Nash equilibrium. To defect is dominant because if both players choose to defect, no player has anything to gain by changing their own strategy [Hardin (1968); Nash (1950, 1951, 1996)]. In the Iterated Prisoner’s Dilemma (IPD) game, two players have to choose their mutual strategy repeatedly, and have memory of their previous behaviors. Because players who defect in one round can be “punished” by defections in subsequent rounds and those who cooperate can be rewarded by cooperation, the appropriate strategy for self-interested players is no longer obvious in IPD games. If the precise length of an IPD is known to the players, then the optimal strategy is to defect on each round (often called All Defect of AllD) [Luce and Raiffa (1957)]. This single rational play strategy which is deduced from propagating the single stage Nash equilibrium of mutual defection backwards through every stage of the game prevents players from cooperating to achieve higher payoffs [Selten (1965, 1983, 1988); Noldeke and Samuelson (1993)]. If the game has infinite length or at least the players are not aware of the length of the game, backward induction is no longer effective and there exists the possibility that cooperation can take place. In fact, there is still controversy about whether or not backward induction can be applied to infinite (or finite) IPDs [Sobel (1975, 1976); Kavka (1986); Becker and Cudd (1990); Binmore (1997); Binmore et al. (2002); Bovens (1997)]. However, in IPD experiments, it was not uncommon to see people cooperate to gain a greater payoff not only in repeated
Iterated Prisoner’s Dilemma and Evolutionary Game Theory
25
games but even in one-shot games [Cooper et al. (1996); Croson (2000); Davis and Holt (1999); Milinski and Wedekind (1998)]. Traditional game theory interprets the cooperation phenomena in IPDs by means of reputation [Fudenberg and Maskin (1986); Kreps and Wilson (1982); Milgrom and Roberts (1982)], incomplete information [Harsanyi (1967); Kreps et al. (1982; Sarin (1999)], or bounded rationality [Anthonisen (1999); Harborne (1997); Radner (1980, 1986); Simon (1955, 1990); Vegaredondo (1994)]. Evolutionary game theory differs from classical game theory in respect of focusing on the dynamics of strategy change in a population more than the properties of strategy equilibrium. In evolutionary game theory, IPD is an ideal experimental platform for the problem as to how cooperation occurs and persists, which is considered to be impossible in the static or deterministic environment. IPD attracted wide interest after Robert Axelrod’s famous book “The Evolution of Cooperation”. In 1979, Robert Axelrod organized a prisoner’s dilemma tournaments and solicited strategies from game theorists [Axelrod (1980a, 1980b)]. Each of the 14 entries competed against all others (including itself) over a sequence of 200 moves. The specific payoff function used is as follows.
Prisoner 1
Cooperate Defect
Prisoner 2 Cooperate Defect (3, 3) (0, 5) (5, 0) (1, 1)
The winner of the tournament was “tit-for-tat” (TFT) submitted by Anatol Rapoport. TFT always cooperate on the first move and then mimics whatever the other player did on the previous move. In a second tournament with 62 entries, again the winner was TFT. Axelrod discovered that “greedy” strategies tended to do very poorly in the long run while “altruistic” strategies did better when PD were repeated over a long period of time with many players. Then genetic algorithms were introduced to show how these altruistic strategies evolve in the populations that are initially dominated by selfishness. The prisoner’s dilemma is therefore of interest to the social sciences such as economics, politics and sociology, and to the biological sciences such as ethology and evolutionary biology, as well to the applied mathematics such as evolutionary computing. Many social and natural processes, for example arm race between states and price setting for duopolistic firms, have been abstracted into models in which independent groups or individuals are engaged in PD games [Brelis (1992); Bunn and Payne (1988); Hauser (1992); Hemelrijk (1991); Surowiecki (2004)].
26
S. Y. Chong et al.
The optimal strategy for the one-shot PD game is simply defection. However, in the IPD game the optimal strategy depends upon the strategies of the possible opponents. For example, the strategy of Always Cooperate (AllC) is dominated by the strategy of Always Defect (AllD), and AllD is optimal in a population consisting of AllD and AllC. However, in a population consisting of AllD, AllC, and TFT, AllD is not necessarily the optimal strategy. It appears that all the strategies in the population determine which strategy is optimal. Although TFT was proved to be efficient in lots of IPD tournaments and was long considered to be the best basic strategy, it could be defeated in some specific circumstances [Beaufils, Delahaye and Mathieu (1996); Wu and Axelrod (1994)]. Therefore, there is lasting interest for game theorists to find optimal strategies or at least novel strategies which outperform TFT in IPD tournaments. Since Axelrod, two types of approaches are developed to test the efficiency or robustness of a strategy and further to derive optimal strategies: (1) Round-robin tournaments. (2) Evolutionary dynamics. Round-robin tournament shows the efficiency of a strategy in competing with others, while ecological simulation illustrates the evolutionary robustness of a strategy in terms of the number of descendants or survivability in a certain environment. Lots of novel strategies have been developed and analyzed by means of these approaches. By using round-robin tournaments, the interactions between different strategies can be observed and analyzed. If the statistical distribution of opposing strategies can be determined an optimal counter-strategy can be derived mathematically. For example, if the population consists of 50% TFT and 50% AllC, the optimal strategy should cooperate with TFT and defect with AllC in order to maximize the payoff. It is easy to design such a strategy that defects in the first two moves, and then plays always C if the opponent defected on the second move, otherwise plays always D. A similar concept in analyzing optimal strategy is Bayesian Nash equilibrium which is widely used in experimental economics [Bedford and Meilijson (1997); Gilboa and Schmeidler (2001); Kagel and Roth (1995); Kalai and Lehrer (1993); Rubinstein (1998);]. In evolutionary dynamics, the processes like natural selection are simulated where individuals with low scores die off, and those with high scores flourish. The evolutionary rule that describes what future states follow from the current state is fixed and deterministic: for a given time interval only one future state follows from the current state
Iterated Prisoner’s Dilemma and Evolutionary Game Theory
27
[Katok and Hasselblatt (1996)]. The common methodology of the evolution rule is replicator equations that assume infinite populations, continuous time, complete mixing and that strategies breed true. Given a population of strategies and the dynamic equations, the evolutionary process can be simulated, and how strategies evolve in the population over a short or long time period can be shown. Optimal strategies can be developed in this way [Axelrod (1987); Darwen and Yao (1995, 1996, 2001); Lindgren (1992); Miller (1996)]. 2.2. Strategies in IPD Tournaments Axelrod is the first who attempts to search for efficient strategies by means of IPD tournament [Axelrod (1980a, 1980b)]. TFT had long been studied as a strategy of IPD game [Komorita, Sheposh and Braver (1968); Rapoport and Chammah (1965)]. However, it is after Axelrods tournaments that TFT become well-known. According to Axelrod, several conditions are necessary for a strategy to be successful. These conditions include: Nice The most important condition is that the strategy must be “nice”. That is, it will not defect before its opponent does. Almost all of the top-scoring strategies are nice. Therefore a selfish strategy will never defect first. Retaliating Axelrod contended that a successful strategy must not be a blind optimist. It must always retaliate. An example of a non-retaliating strategy is AllC. This is a very bad choice, as “nasty” strategies will ruthlessly exploit such strategies. Forgiving Another quality of successful strategies is that they must be forgiving. Though they will retaliate, they will fall back to cooperating if the opponent does not continue to defect. This stops long runs of revenge and counter-revenge, thus maximising payoffs.
S. Y. Chong et al.
28
Clear The last quality is being clear, that is making it easier for other strategies to predict its behavior so as to facilitate mutually cooperation. Stochastic strategies, however, are not clear because of the uncertainty in their choice. In a further study, Axelrod noted that just a few of the 62 entries in the second tournament have reasonably influence on the performance of a given strategy. He utilized eight strategies as opponents for a simulated evolving population based on a genetic algorithm approach [Axelrod (1987)]. The population consisted of deterministic strategies that use outcomes of the three previous moves to determine a current move. The simulation was conducted using a population of 20 strategies from a total of 270 strategies executed repeatedly against the eight representatives. Mutation and crossover were used to generate new strategies. The typical results indicated that populations initially generated mutual defection, but subsequently evolved toward mutual cooperation. Moreover, most of the strategies that evolved in the simulation actually resemble TFT, having the properties of “Nice”, “Forgiving”, and “Retaliating”. Although TFT has been considered to be the most successful strategy in IPD for several decades, there still is some controversy about it. There seems to be a lack of theoretical explanation for the strategies like TFT in traditional game theory. TFT is not subgame perfect, and there are always subgame perfect equilibria that dominate TFT according to the Folk Theorem [Binmore (1992); Hargreaves and Varoufakis (1995); Myerson (1991); Rubinstein (1979); Selten (1965, 1975)]. On the other hand, whether or not TFT is the most efficient singleton strategy in IPD game is still unclear; therefore, many researchers are attempting to develop novel strategies that can outperform TFT. 2.2.1. Heterogeneous TFTs Since TFT had such success in IPD tournaments and experiments, it is natural to draw the conclusion that TFT may be improved by slightly modifying its rule. Many heterogeneous TFTs have been developed in order to overcome TFT’s shortcoming or to adapt to a certain environment, for example IPD with noise. Among these strategies, Tit-for-Two-Tats (TFTT), Generous TFT (GTFT), and Contrite TFT (CTFT) are examples.
Iterated Prisoner’s Dilemma and Evolutionary Game Theory
29
A situation that TFT does not handle well is a long series of mutual retaliations evoked by an occasional defection. The deadlock can be broken if the co-player behaves more generously than TFT and forgives at least one defection. TFTT retaliates with defection only after two successive defections and thus attempts to avoid becoming involved in mutual retaliations. Usually, TFTT performs well in a population with more cooperative strategies but does poorly in a population with more permanently defective strategies. Similar to TFTT, Benevolent TFT (BTFT) always cooperates after cooperation and normally defects after defection, but occasionally BTFT responds to defection by cooperation in order to break up a series of mutual obstruction [Komorita, Sheposh and Braver (1968)]. In experiments of Manarini (1998) and Micko (1997), fixed interval BTFT strategies were shown to be superior to, or at least equivalent to, TFT in terms of cooperation as well as in terms of cumulative pay-off. However, BTFT tends to produce irregularly alternating exploitations and sometimes resort mutual retaliations. Allowing some percentage of the other player’s defections to go unpunished has been widely accepted as a good way to cope with noise [Molander (1985); May (1987); Axelrod and Dion (1988); Bendor et al. (1991); Godfray (1992); Wu and Axelrod (1994)]. A reciprocating strategy such as TFT can be modified to forgive the other player’s defection with a certain ratio in order to decrease the influence of noise. GTFT behaves like TFT but cooperates with the probability of q = min[1−(T −R)/(R−S), (R−P )/(T −P )] when it would otherwise defect. This prevents a single error from echoing indefinitely. For example, in the case of T = 5, R = 3, P = 1, and S = 0, q = 1/3. GTFT is said to take over the dominant position of the population of homogeneous TFT strategies in an evolutionary environment with noise [Nowak and Sigmund (1992)]. In a noisy environment, retaliating unintended defection often leads to permanent bilateral retaliation. Therefore, forgiving defection evoked by unintended defection allows a quick way to recover from error. It is based upon the idea that one shouldn’t be provoked by the other player’s response to one’s own unintended defection [Sugden (1986); Boyd (1989)]. The strategy of CTFT has three states: “contrite”, “content” and “provoked”. It begins in a content state, with cooperation and stays there unless there is a unilateral defection. If it was the victim while content, it becomes provoked and defects until a cooperation from other player causes it to become content. If it was the defector while content, it becomes contrite and cooperates. When contrite, it becomes content only after it has successfully
30
S. Y. Chong et al.
cooperated. CTFT can correct its unintended defection in a noisy environment. If one of two CTFT players defects, the defecting player will contritely cooperate on the next move and the other player will defect, and then both will be content to cooperate on the following move. However, CTFT is not effective at correcting the other player’s error. For example, if CTFT is playing TFT and the TFT player defected by accident, the retaliation will continue until another error occurs. In an ecological simulation with noise, GTFT and CTFT competed with the 63 rules of the Second Round of the Computer Tournament for the Prisoner’s Dilemma [Axelrod (1984)]. CTFT is the dominant strategy, becoming 97% of the population at generation 2000 [Wu and Axelrod (1994)]. 2.2.2. Pavlov (Win-Stay Lose-Shift) A possible drawback of TFT is that it performs poorly in a noisy environment. Assume that a population of TFT strategies plays IPD with one another in a noisy environment, where every choice may be occasionally implemented in error. Although a TFT strategy cooperates with its twin at the beginning, it would get out of cooperation as soon as the other player’s action is misinterpreted, and then this induces the other player’s defection in the next round. Therefore, after an error, the result of the game turns out to be a CD, DC, CD . . . cycle. If a second error happens, the outcome is as likely to fall into defection as it is to resume cooperation. Cooperation between TFT strategies is easy to break even in the case of low noise frequency [Donninger (1986); Kraines and Kraines (1995)]. The Pavlov strategy, also known as Win-Stay Lose-Shift or Simpleton [Rapoport and Chammah (1965)], has been shown to outperform TFT in the environment with noise [Fudenberg and Maskin (1990); Kraines and Kraines (1995, 2000)]. Pavlov cooperates when both sides have cooperated or defected on the previous move, and defects otherwise. Pavlov, as well as TFT, are a type of memory-one strategies where players only remember and make use of their own move and their opponent’s move on the last round. The major difference between Pavlov and TFT is that Pavlov will choose COOPERATE after a defection as against TFT’s DEFECT, and this helps Pavlov resume cooperation with those cooperative strategies, such as TFT, in a noisy environment. When restricted to an environment of memory-one agents interacting in iterated Prisoners Dilemma games with a 1% noise level, Pavlov is the only cooperative strategy and one of the very few that cannot be invaded by a similar strategy [Nowak and Sigmund (1993, 1995)].
Iterated Prisoner’s Dilemma and Evolutionary Game Theory
31
Simulation of evolutionary dynamics of win-stay lose-shift strategies shows that these strategies are able to adapt to the uncertain environment even when the noise level is high [Posch (1997)]. In simulated stochastic memory-one strategies for the IPD games, Nowak and Sigmund (1993, 1995) report that cooperative agents using a Pavlov type strategy eventually dominate a random population. Memory-one strategies can be expressed in the form of S(p1 , p2 , p3 , p4 ), where p1 denotes the probability of playing C (Cooperate) after a CC outcome, p2 denotes the probability of playing C after a CD outcome, p3 denotes the probability of playing C after a DC outcome, and p4 denotes the probability of playing C after a DD outcome. Most of the well-known strategies can be expressed in this form. For example, AllC = S(1, 1, 1, 1), AllD = S(0, 0, 0, 0), TFT = S(1, 0, 1, 0), Pavlov = S(1, 0, 0, 1). Noise is conveniently introduced by restricting the conditional probabilities pi to range between 0 and 1. For example, S(0.999, 0.001, 0.999, 0.001) is a TFT strategy with 0.001 probability of being misinterpreted. In a computer simulation with a population using the totally random strategy S(0.5, 0.5, 0.5, 0.5), win-stay lose-shift strategy shows its evolutionary robustness in noisy environment. After each 100 generations from a total of 107 generations, 105 mutant strategies that are generated at random are introduced. Simulation results show that the populations are dominated by win-stay lose-shift strategy in 33 of a total of 40 simulations. TFT strategies perform poorly in large part because they do not exploit overly cooperate strategies. Simulations reveal that Pavlov loses against AllD but can invade TFT, and that Pavlov cannot be invaded by AllD [Milinski (1993)]. 2.2.3. Gradual The Gradual strategies are like TFT but respond to the opponent with a gradual pattern. This strategy acts as TFT, except when it is time to forgive and remember the past. It uses cooperation on the first move and then continues to do so as long as the other player cooperates. Then after the first defection of the other player, it defects one time and cooperates two times; after the second defection of the opponent, it defects two times and cooperates two times, . . . after the nth defection it reacts with n consecutive defections and then calms down its opponent with two cooperations [Beaufils, Delahaye and Mathieu (1996)]. Both round-robin competitions and ecological evolution experiments are conducted in order to compare the performance of Gradual with TFT.
32
S. Y. Chong et al.
Gradual wins in experiments where round-robin competitions are conducted with several well-known strategies, such as TFT and GRIM. In ecological evolutionary experiments, gradual and TFT have the same type of evolution, with the difference of quantity in favor of gradual, which is far away in front of all other survivors when the population is stabilised. However, it is efficient to demonstrate that TFT is not always the best, but not efficient to prove that Gradual always outperforms TFT. Gradual receives fewer points than TFT while interacting with AllD because Gradual forgives too many defections. Therefore, if there are lots of defecting strategies like AllD in the competition, it would be possible that TFT outperforms Gradual in this case. Beaufils, Delahaye and Mathieu (1996) try to improve the performance of Gradual by using a genetic algorithm. 19 different genes are used and a fitness function evaluates the quality of the strategies. Several new strategies are found after 150 generations of evolution. One of them beats Gradual and TFT in round-robin tournament, as well as in an ecological simulation. In the two cases it has finished first just in front of Gradual, TFT being two or three places behind, with a wide gap in the score, or in the size of the stabilised population. The evolution dynamics of populations including Gradual has also been studied in Delahaye and Mathieu (1996), Doebeli and Knowlton (1998), Glomba, Filak, and Kwasnicka (2005), Beaufils, Delahaye, and Mathieu (1996). 2.2.4. Adaptive strategies From the viewpoint of automation, the strategies in IPD games can be regarded as automatic agents with or without feedback mechanisms. Most well-known IPD strategies are not adaptive because their responses to any certain opponent are fixed. It is impossible to improve their performance since the parameters of their responding mechanism cannot be adjusted. However, there are still some strategies in IPDs which are adaptive. Although there is still no experimental evidence of adaptive strategies outperforming non-adaptive ones in IPD games, adaptive strategies are worth studying since creatures with higher intelligence are all adaptive. There have been two approaches to developing adaptive strategies. Firstly, adaptive mechanisms can be implemented by making the parameters of a non-adaptive strategy adjustable. Secondly, new adaptive strategies can be developed by using evolutionary computation, reinforcement
Iterated Prisoner’s Dilemma and Evolutionary Game Theory
33
learning, and other computational techniques [Darwen and Yao (1995, 1996)]. Tzafestas (2000a, 2000b) introduced adaptive tit-for-tat (ATFT) strategy that embedded an adaptive factor into the conventional TFT strategy. ATFT keeps the advantages of tit-for-tat in the sense of retaliating and forgiving, and implements some behavioural gradualness that would show as fewer oscillations between Cooperate and Defect. It uses an estimate of the opponent’s behavior, whether cooperative or defecting, and reacts to it in a tit-for-tat manner. To represent degrees of cooperation and defection, a continuous variable named “world ” which ranges from 0 (total defection) to 1 (total cooperation) is applied. The ATFT strategy can then be formulated as a simple model: If (opponent played C in the last cycle) then world = world + r*(1-world) else world = world + r*(0-world) If (world >= 0.5) play C, else play D r is the adaptation rate here. The TFT strategy corresponds to the case of r = 1 (immediate convergence to the opponent’s current move). Clearly, ATFT is an extension of the conventional TFT strategy. By simulating the spatial IPD games between ATFT, AllD, AllC, and TFT on 2D grid, it shows that ATFT is fairly stable and resistant to perturbations. Since the use of a fairly small adaptation rate r will allow more gradual behavior, ATFT tends to be more robust than TFT in a noisy environment. Since evolutionary computation has been widely used in simulating the dynamics of IPD games, it is natural to consider obtaining IPD strategies directly by using evolutionary approaches [Lindgren (1991); Fogel (1993); Darwen and Yao (1995, 1996)]. Axelrod (1987) studied how to find effective strategies by using genetic algorithms as simulation method. He established an initial population of strategies that is deterministic and uses the outcome of the three previous moves to make a choice in the current move. By means of playing IPD games between one another, successful strategies are selected to have more offspring. Then the new population will display patterns of behavior that are more like those of the successful strategies of the previous population, and less like those of the unsuccessful ones. As the evolution process continues, the strategies with relatively high scores will flourish while the unsuccessful strategies die out. Simulation results show
34
S. Y. Chong et al.
that most of the strategies that were evolved in the simulation actually resemble TFT and does substantially better than TFT. However, it would not be accurate to say that these strategies are better than TFT because they are probably not very robust in other environments [Axelrod (1987)]. Many researchers have found that evolved strategies may lack robustness, i.e., the strategies did well against the local population, but when something new and innovative appeared they fail [Lindgren (1991); Fogel (1993)]. Darwen and Yao (1996) applied a technique to prevent the genetic algorithm from converging to a single optimum and attempted to develop new IPD strategies without human intervention. It concludes that adding static opponents to the round robin tournament improves the results of final population. Optimal strategies can be determined only if the strategy of the opponent is known. By means of reinforcement learning, model-based strategies with the ability of on-line identification of an opponent can be built [Sandholm and Crites (1996); Freund et al. (1995); Schmidhuber (1996)]. How can a player acquire a model of its opponent’s strategy? One possible source of information available for the player is the history of the game. Another possible source of information is observed games between the opponent and other agents. In the case of IPD games, a player can infer an opponent’s model based on the outcome of the past moves and then adapts its strategy during the game. Reinforcement learning (RL) is based on the idea that the tendency to produce an action should be strengthened if it produces favorable results, and weakened if it produces unfavorable results [Watkins (1989); Watkins and Dayan (1992); Kaelbling and Moore (1996)]. A modelbased RL approach generates expectation about the opponent’s behavior by making use of a model of its strategy [Carmel and Markovitch (1997, 1998)]. It is well suited for use in IPD tournament against an unknown opponent because of its small computational complexity. The major problem in designing a model-based strategy (MBS) is the risk involved in the exploration, and thus the issue of exploitation versus exploration. An exploring action taken by the MBS tests unfamiliar aspects of the opponent which can yield a more accurate model of the opponent. However, this action also carries the risk of putting the MBS into a much worse position. For example, in order to distinguish the strategy ALLC from GRIM and TFT in IPD tournament, a MBS has to defect at least once and therefore loses the chance to cooperate with GRIM. The exploratory action affects not only the current payoff but also the future rewards [Berry and Fristedt (1985)]. There have been several approaches developed to solve this
Iterated Prisoner’s Dilemma and Evolutionary Game Theory
35
problem [Berry and Fristedt (1985); Gittins (1989); Sutton (1990); Narendra and Thathachar (1989); Kaelbling (1993); Moore and Atkeson (1993); Carmel and Markovitch (1998)]. Since possible strategies for a repeated game is usually infinite, computational complexity is another problem that needs to be addressed [Ben-porath (1990); Carmel and Markovitch (1998)]. There is seldom a record of an effective MBS in round-robin IPD tournaments. However, the strategy that won competition 4 in 2005 IPD tournament, Adaptive Pavlov, is such a strategy [Prisoner’s dilemma tournament result (2005)]. Furthermore, it seems that each of the strategies that ranked above TFT incorporated a mechanism to explore the opponent. 2.2.5. Group strategies In the 2004 IPD competition [20th-anniversary Iterated Prisoner’s Dilemma competition], a team from Southampton University led by Professor N. Jennings introduced a group of strategies, which proved to be more successful than Tit-for-Tat (see chapter 9). The group of strategies were designed to recognise each other through a known series of five to ten moves at the start. Once two Southampton players recognized each other, they would act as their “master” or “slave” roles – a master will always defect while a slave will always cooperate in order for the master to win the maximum points. If the program recognized that another player was not a Southampton entry, it would immediately defect to minimise the score of the oppositions. The Southampton group strategies succeeded in defeating any non-grouped strategies and won the top three positions in the competition [Prisoner’s dilemma tournament result (2004)]. According to Grossman (2004), it was difficult to tell whether a group strategy would really beat TFT because most of the “slave” group members received far lower scores than the average level and were ranked at the bottom of the table. The average score of the group strategies is not necessarily higher than that of TFT. The significance of group strategies maybe lies in their evolutionarily characters. None of known strategies in IPD games is an evolutionarily stable strategy. [Boyd and Loberbaum (1987)] The strategies that are most likely to be evolutionarily stable, such as AllD or GRIM, can resist the invasion of some types of strategies but cannot resist the invasion of others. For example, a small group of TFT strategies can not invade a large population of AllD; however, STFT can do. There exists the possibility
36
S. Y. Chong et al.
that TFT can successfully invade a population of AllD indirectly. Suppose that a large population of AllD is continuously attacked by small groups of STFT. Because every invasion makes a small positive proportion of STFT remain in the population of AllD, the number of STFT increases gradually. When the number of STFT is large enough, a small group of TFT can successfully invade and AllD will die out. However, group strategies may be evolutionarily stable. By means of cooperating with group members and defecting against non-group members, a population of group strategies can prevent any foreigner from successfully invading. This is, perhaps, the real value of group strategies. 2.3. Evolutionary Dynamics in Games Traditional game theorists have developed several effective approaches to study static games based on the assumption of rationality. By using Neumann-Morgenstern utility, refinement of Nash equilibrium, and reasoning, both cooperative and non-cooperative games are analyzed within a theoretical framework. However, in the area of repeated games, especially in games where dynamics are concerned, few approaches from traditional game theory are available. Evolutionary game theory provides novel approaches to solve dynamic games. If the precise length of an IPD is known to the players, then the optimal strategy is to defect on each round. If the game has infinite length or at least the players are not aware of the length of the game, there exists the possibility that cooperation happens [Dugatkin (1989); Darwen and Yao (2002); Akiyama and Kaneko (1995); Doebeli, Blarer, and Ackermann (1997); Axelrod (1999); Glance and Huberman (1993, 1994); Ikegami and Kaneko (1990); Schweitzer (2002)]. Nowak and May (1992, 1993) showed that cooperators and defectors coexist in certain circumstances by introducing spatial evolutionary games, in which two types of players – cooperators who always cooperate and defectors who always defect are placed in a two-dimensional spatial array. In each round, every individual plays the PD game with its immediate neighbors. The selection scheme is that each lattice is occupied either by its original owner or by one of the neighbors, depending on who scores the highest total in that round, and so on to the next round of the game. Simulation results show that cooperators remain a considerable percentage of the population in some cases, and defector can invade any a lattice but can not occupy the whole area.
Iterated Prisoner’s Dilemma and Evolutionary Game Theory
37
When the parameters of the payoff matrix are set to be T = 2.8, R = 1.1, P = 0.1, and S = 0 and the initial state is set to be a random mixture of the two types of strategies, the evolutionary dynamics of the local interaction model lead to a state where each player chooses the strategy Defect, the only ESS in the prisoner’s dilemma. Figure 2.1 shows that the population converges to a state where everyone defects and no Cooperate strategy survives after 5 generations.
Generation 1
Generation 2
Generation 3
Generation 6
Fig. 2.1. Spatial Prisoner’s Dilemma with the values T = 2.8, R = 1.1, P = 0.1, and S = 0 [Nowak and May (1993)].
However, when the parameters of the payoff matrix are set to T = 1.2, R = 1.1, P = 0.1, and S = 0, the evolutionary dynamics do not converge to the stable state of defection. Instead, a stable oscillating state where cooperators and defectors coexist and some regions are occupied in turn by different strategies.
Generation 1
Generation 2
Generation 19
Generation 20
Fig. 2.2. Spatial Prisoner’s Dilemma with the values T = 1.2, R = 1.1, P = 0.1, and S = 0 [Nowak and May (1993)].
Moreover, when the parameters of payoff matrix are set to be T = 1.61, R = 1.01, P = 0.01, and S = 0, the evolutionary dynamics lead to a chaotic state: regions occupied predominantly by Cooperators may be successfully
S. Y. Chong et al.
38
invaded by Defectors, and regions occupied predominantly by Defectors may be successfully invaded by Cooperators.
Generation 1
Generation 3
Generation 13
Generation 15
Fig. 2.3. Spatial Prisoner’s Dilemma with the values T = 1.61, R = 1.01, P = 0.01, and S = 0 [Nowak and May (1993)].
If the starting configurations are sufficiently symmetrical, this spatial version of the PD game can generate chaotically changing spatial patterns, in which cooperators and defectors both persist indefinitely. For example, if we set R = 1, P = 0.01, S = 0.0 and T = 1.4, and initial state is that every individual in a square 69 × 69 lattice is a cooperator except a defector in the middle of the lattice. The structure of the evolving lattice varies like a kaleidoscope, and the ever-changing sequences of spatial patterns can be very beautiful, as shown in Fig. 2.4. The role of the spatial interaction in the evolution of cooperation is further studied by Durrett and Levin (1998), Schweitzer, Behera, and M¨ uhlenbein (2002), Ifti, Killingback, and Doebeli (2004).
Generation 10
Generation 40
Generation 4000
Generation 6000
Fig. 2.4. Spatial Prisoner’s Dilemma with the values T = 1.4, R = 1, P = 0.01, and S = 0, where Blue, Red, Green, and Yellow denote cooperators, defectors, new cooperators, and new defectors respectively [Nowak and May (1993)].
Iterated Prisoner’s Dilemma and Evolutionary Game Theory
39
2.3.1. Evolutionary stable strategy Just like the Nash equilibrium in traditional game theory, Evolutionarily Stable Strategy (ESS) is an important concept used in theoretical analysis of evolutionary games. According to Maynard Smith (1982), an ESS is a strategy such that, if all the members of a population adopt it, then no mutant strategy could invade the population under the influence of natural selection. ESS can be seen as an equilibrium refinement to the Nash equilibrium. Suppose that a player in a game can choose between two strategies: I and J. Let E(J, I) denote the payoff he receives if he chooses the strategy J while all other players choose I. Then, the strategy I is evolutionarily stable if either (1) E(I, I) > E(J, I), or (2) E(I, I) = E(J, I) and E(I, J) > E(J, J) is true for all I 6= J [Maynard Smith and Price (1973); Maynard Smith (1982)]. Thomas (1985) rewrites the definition of ESS in a different form. Following the terminology given in the first definition above, we have (1) E(I, I) ≥ E(J, I), and (2) E(I, J) > E(J, J) From this alternative form of definition, we find that ESS is just a subset of Nash equilibrium. The benefit of this refinement of Nash equilibrium is not just to eliminate those weak Nash equilibrium, but to provide an efficient mathematical tool for dynamic games. Following the concept of ESS, two approaches to evolutionary game theory have been developed. The first approach directly applies the concept of ESS to analyze static games. The second approach simulates the evolutionary process of dynamic games by constructing a dynamic model, which may take into consideration the factors of the population, replication dynamics, and strategy fitness. As an example of using ESS in static games, consider the problem of the Hawk-Dove game. Two types of animals employ different means to obtain resources (a favorable habitat, for example) — Hawk always fights for some resources while Dove never fights. Let V denote the value of the resources, which can be considered the Darwinian fitness of an individual obtaining the resource, described by Maynard Smith (1982). Let E(H, D) denote the payoff to a Hawk against a Dove opponent. If we assume that (1) whenever two Hawks meet, conflict eventually results and the two individuals are
S. Y. Chong et al.
40
equally likely to be injured, (2) the cost of the conflict reduces individual fitness by some constant value C, (3) when a Hawk meets a Dove, the Dove immediately retreats and the Hawk obtains the resource, and (4) when two Doves meet the resource is shared equally between them, the payoff matrix for Hawk-Dove game will look like this, Hawk Dove
Hawk Dove ((V -C/2, (V -C)/2) (V, 0) (0, V ) (v/2, V /2)
Then, it is easy to verify that the strategy Dove is not an ESS because there is E(D, D) < E(H, D), which means that a pure population of Doves can be invaded by a Hawk mutant. In the case that the value V of the resource is greater than the cost C of injury, the strategy Hawk is an ESS because there is E(H, H) > E(D, H), which means that a Dove mutant can not invade a group of Hawks. If V < C is true, the Hawk-Dove game becomes the game of Chicken originated from the 1955 movie Rebel without a cause. Neither pure Hawk nor pure Dove is ESS in this game. However, there is an ESS if mixed strategies are permitted [Bishop and Cannings (1978)]. An evolutionarily stable state is a dynamical property of a population to return to using a strategy, or mix of strategies, if it is perturbed from that strategy, or mix of strategies [Maynard Smith (1982)]. A population of ESS must be evolutionarily stable because it is impossible for any mutant to invade it. Many biologists and sociologists attempt to explain animal and human behavior and social structures in terms of ESS [Cohen and Machalek (1988); Mealey (1995)]. However, a dynamic game is not necessarily converging to a stable state in which ESS is prevalent. For example, using a spatial model in which each individual plays the Prisoner’s Dilemma with his or her neighbors, Nowak and May (1992, 1993) show that the result of the game depends on the specific form of the payoff matrix. Now imagine a population of players in a society where each one has to play Prisoner’s Dilemma with another and whether or not one can survive and breed is determined by his payoff in the game. How will the population evolve? In order to show the evolutionary process of the population, a model of dynamics that takes time t into consideration is needed. 2.3.2. Genetic algorithm A genetic algorithm maintains a population of sample points from the search space. Each point is represented by a string of characters, known
Iterated Prisoner’s Dilemma and Evolutionary Game Theory
41
as genotype [Holland (1975, 1992, 1995)]. By defining a fitness function to evaluate them, genetic algorithm proceeds to initialize a population of solutions randomly, and then improve it through repetitive application of mutation, crossover, and selection operators. The common methodology to study the evolutionary dynamics in games is through replicator equations. Replicator equations usually assume infinite populations, continuous time, complete mixing and that strategies breed true [Taylor (1979); Maynard Smith (1982); Weibull (1995); Hofbauer and Sigmund (1998)]. Originated from biology and then introduced into evolutionary game theory by Taylor and Jonker (1978), replicator equations provide a continuous dynamic model for evolutionary games. Consider a population of n types of strategies, and let xi be the frequency of type i. Let A be the n × n payoff matrix. With the assumptions that the population is infinitely large and strategies are completely mixed and xi are differentiable functions of time t, a strategy’s fitness, or expected payoff can be written as (Ax)i if strategies meet one another randomly. The average fitness of the population as a whole can be written as xT Ax. Then, the replicator equation is x˙ i = xi ((Ax)i − xT Ax)
(2.1)
Evolutionary games with a replicator dynamic as described in (2.1) will converge to a result that strategies with strong fitness bloom in the population. For the Prisoner’s Dilemma, the expected fitness of the strategies Cooperate and Defect, EC and ED respectively, are EC = x C R + x D S ,
and ED = xC T + xD P
(2.2)
where xC and xD denote the proportions of the strategies of Cooperate and ¯ denote the average fitness of Defect in the population respectively. Let E the entire population, there is ¯ = x C Ec + x D ED E
(2.3)
Then, the replicator equations for this game are dxC ¯ , dxD = xD (ED − E) ¯ = xC (Ec − E) (2.4) dt dt Since there is T > R and P > S, ED −EC = xC (T −R)+xD (P −S) > 0 C holds, and there must be ED > E¯ > EC . Therefore, there are dx dt < 0 and dxD dt > 0. This means that the number of the strategies of Cooperate will always decline while the number of the strategies of Defect increases as the
42
S. Y. Chong et al.
game goes on. Sooner or later, the proportion of the population choosing the strategy Cooperate will, in theory, become extinct. Besides replicator dynamics, there exist other types of dynamics equations that can be used in modeling evolutionary systems [Akin (1993); Thomas (1985); Bomze (1998, 2002); Balkenborg and Schlag (2000); Cressman, Garay and Hofbauer (2001); Weibull (1995); Hofbauer (1996); Gilboa and Matsui (1991); Matsui (1992); Fudenberg and Levine (1998); Skyrms (1990); Swinkels (1993); Smith and Gray (1994)]. Lindgren (1995) and Hofbauer and Sigmund (2003) have given a comprehensive review of them. In general, dynamic games are of great complexity. How an evolutionary system evolves depends not only on the population and dynamic structures but also on where the evolution starts. Because of dynamic interactions between multiple players, especially those players with intelligence, genetic algorithms may converge towards local optima rather than the global optimum. Also, operating on dynamic data sets is difficult as genomes begin to converge early on towards solutions which may no longer be valid for later data [Michalewicz (1999); Schmitt (2001)]. Analysis of the evolutionary dynamic systems is not just a problem of evolutionary game theory, but a new direction in applied mathematics [Garay and Hofbauer (2003); Gaunersdorfer (1992); Gaunersdorfer, Hofbauer, and Sigmund (1991); Hofbauer (1981, 1984, 1996); Krishna and Sj¨ ostr¨ om (1998); Plank (1997); Smith (1995); Zeeman (1993), Zeeman and Zeeman (2002, 2003)].
2.3.3. Strategies What strategies should be involved in evolutionary dynamics is a difficult question. One approach is to take into consideration lots of representative strategies, for example Axelord (1984), Dacey and Pendegraft (1988), and Akimov and Soutchanski (1994), since it is impossible to enumerate all possible strategies. However, it is difficult to say what strategy should be included and which ones not, and there is little comparability between evolutionary processes with different strategies because the selection of strategies may have great influence on the outcome of the dynamics. Another approach is to study the interactions between specific strategies, for example Nowak and Sigmund (1990, 1992) and Goldstein and Freeman (1990). In this way, it is convenient to make clear the relationship between strategies in the evolutionary process; however, generality of complex evolutionary systems loses to some extent.
Iterated Prisoner’s Dilemma and Evolutionary Game Theory
43
Strategies in PD games (or in non-PD games) can be characterized as either deterministic or stochastic. Deterministic strategies leave nothing to change and respond to the opponent with predetermined actions; stochastic strategies, however, leave some uncertainty in their choices. Oskamp (1971) presents a thorough review of the early studies on the strategies involved in PD games and non-PD games, for example AllD, TFT, and lots of stochastic strategies that play C or D with some certain probabilities [Lave (1965); Bixenstine, Potash, and Wilson (1963); Solomon (1960); Crumbaugh and Evans (1967); Wilson (1969); Oskamp and Perlman (1965); Sermat (1967); Heller (1967); Knapp and Podell (1968); Lynch (1968); Swingle and Coady (1967); Whitworth and Lucker (1969)]. After Axelord’s IPD tournament, memory-one strategies that interact with the opponent according to both sides’ behavior in the previous move become prevalent. TFT, Pavlov, Grim Trigger, and many other memoryone strategies are analyzed in varies of environment: round-robin tournaments, evolutionary dynamics with or without noise [Nowak and Sigmund (1990, 1992, 1993); Pollock (1989); Wedekind and Milinski (1996); Milinski and Wedekind (1998); Sigmund (1995); Stephens (2000); Stephens, Mclinn and Stevens (2002); Sandholm and Crites (1996); Doebeli and Knowlton (1998); Brauchli, Killingback and Doebeli (1999); Sasaki, Taylor and Fudenberg (2000)]. No strategy has been shown to be superior in a dynamic environment, and even deterministic cooperators can invade defectors in specific circumstances. It is not sensible to discuss which strategy is best unless the context is defined. Comparing TFT with GTFT, Grim (1995) suggests that, in the non-stochastic Axelrod models, it is TFT that is the general winner; within a purely stochastic model, the greater generosity of GTFT pays off; in a model with both stochastic and spatial elements, a level of generosity twice that of GTFT proves optimal. Pavlov has an obvious advantage over TFT in noisy environments [Nowak and Sigmund (1993); Kraines and Kraines (1995)]. In an evolutionary process where AllC, AllD, TFT, and GTFT strategies are involved, evolution starts off toward defection but then veers toward cooperation. TFT strategies play a key role in invading the population of defectors. However, GTFT strategies and then more generous AllCs gradually become dominant once cooperation is widely established, and this provides an opportunity to AllD to invade again [Nowak and Sigmund (1992)]. Additionally, Selten and Stoecker (1986) have studied the end game behavior in finite IPD supergames, and find that cooperative behaviors last until shortly before the end of the supergame.
44
S. Y. Chong et al.
Machine Learning approaches have been introduced into evolutionary game theory to develop adaptive strategies, especially those for IPD games [Carmel and Markovitch (1996, 1997, 1998); Littman (1994); Tekol and Acan (2003); Hingston and Kendall (2004)]. Adaptive strategies, at least in theory, have obvious advantages over fixed strategies. Among the set of adaptive strategies, there may be an evolutionarily stable strategy for IPD games and potential winner of future IPD tournaments. 2.3.4. Population Population size and structure are of great importance in evolutionary dynamics. In general, evolutionary processes in a large population are quite different from that in small populations [Maynard Smith (1982); Fogel and Fogel (1995); Fogel, Fogel and Andrew (1997, 1998); Ficici and Pollack (2000)]. Young and Foster (1991) have studied stochastic effects in a population consisting of three strategies: AllD, AllC, and TFT. They show that the outcome of the evolutionary process depends crucially on the amount of noise, which is inversely proportional to the population size. The more people there are, the more that random variations in their behavior are smoothed out in the population proportions. For large populations, the system tends to drift from TFT to AllC, which is then invaded by AllD. As a result, most of the players behave as AllD, even though initially most players may have started as TFT. They conclude that cooperation is viable in the short run, but not stable in the long run in a large population. Boyd and Richerson (1988, 1989) suggest that reciprocity is unlikely to evolve in large groups as a result of natural selection because reciprocators punish defection by withholding future cooperation which will penalize other cooperators in the group. Boyd and Richerson (1990, 1992) analyze a model in which the punishment response to defection is directed solely at defectors. In this model, cooperation reinforced by retribution can lead to the evolution of cooperation in different ways. There is the possibility that strategies which cooperate and punish defectors, strategies which cooperate only if punished, and strategies which cooperate but do not punish coexist in the long run, as well as the possibility that only one type exists. As the group size grows larger, however, the conditions for co-operators’ surviving becomes more difficult. Glance and Huberman (1994) discuss how to achieve cooperation in groups of various sizes in n-person PD games and find that there are two
Iterated Prisoner’s Dilemma and Evolutionary Game Theory
45
stable points in large groups: either there is a great deal or very little cooperation. Cooperation is more likely in smaller groups than in larger ones and there is greater cooperation when players are allowed more communication with each other. Large random fluctuations are related to group size. Groups beyond a certain size may experience increased difficulty of informational exchange and coordination; further, reneging on contracts is possible to be prevalent as each member may expect that the effect of his/her action on other members will be diluted. However, Dugatkin (1990) finds that cooperation may invade large populations more easily than smaller ones, but it is likely to represent a smaller proportion of the population in larger groups. In order to consider the potential importance of the relationship between population size and cooperative behaviour, two N-person game theoretical models are presented. The results show that cooperation is frequently not a pure evolutionarily stable strategy, and that many metapopulations should be polymorphic for both cooperators and defectors. It is well accepted that communication among members of a society leads to more cooperative behaviors [Insko et al. (1987); Orbell, Kragt, and Dawes (1988)]. Insko et al. (1987, 1988, 1990, 1993) explore the role of communication on interindividual-intergroup discontinuity in the context of the extended PD game that adds a third withdrawal choice to the usual cooperative and uncooperative choices, and interindividualintergroup discontinuity is the tendency of intergroup relations to be more competitive and less cooperative than interindividual relations. The lesser tendency of individuals to cooperate when there is no communication with the opponent partially explains the group discontinuity. Choice and refusal of partners may accelerate the emergence of cooperation. Experiments have shown that people who are given the option of playing or not are more likely to choose to play if they are themselves planning to cooperate. More cooperative players are more likely to anticipate that others will be cooperative [Orbell and Dawes (1993)]. Defecting players are possible to be alienated by cooperators [Schuessler (1989); Kitcher (1992); Batali and Kitcher (1994)]. In the N-person PD game, it may be that players can change groups if they don’t satisfy the size of their groups [Hirshleifer and Rasmusen (1989)]. The option of choice and refusal of partners in IPD means that players will attempt to select partners rationally. Analytical studies reveal that the subtle interplay between choice and refusal in N-player IPD games can result in various long-run player interaction patterns: mutual cooperation; mixed mutual cooperation and mutual defection; parasitism; and wallflower seclusion. Simulation studies
46
S. Y. Chong et al.
indicate that choice and refusal can accelerate the emergence of cooperation in evolutionary IPD games [Stanley, Ashlock, and Tesfatsion (1994); Stanley, Ashlock and Smucker (1995)]. The effects of freedom to play, reciprocity and interchange, coalitions and alliances, and various sizes of groups on evolution are also studied [Orbell and Robyn (1993); Alexander and Frans (1992); Glance and Bernardo (1994); Hemelrijk (1991)]. In a specific scenario, the prestructuration of the population may determine the evolution of the patterns of interaction that constitute the final social structure [Eckert, Koch, and Mitl¨ ohner (2005)]. 2.3.5. Selection scheme Evolutionary selection schemes can be characterized as either generational or steady-state schemes [Thierens (1997)]. Generational schemes that are widely used in evolutionary game theory mean that each generation of a population is replaced in one step by a new generation. In a system with a steady-state scheme only a small percentage of the population is replaced in each generation. Evolutionary selection schemes can be further subdivided as pure or elitist selection schemes in terms of whether or not there is an overlap between successive generations. Pure selection schemes allow no overlap between successive generations: all parents from previous generation are discarded and the next generation is filled entirely with offspring from these parents. In elitist schemes, subsequent generations may be the same: parents with higher fitness are transferred to the next generation and only poorly performing parents are replaced [Mitchell (1996)]. Pure selection schemes are commonly used in IPD research [Axelord (1987); Axelrod and Dion (1988); Huberman and Glance (1993); Akimov and Soutchanski (1994); Mill (1996)]. These schemes use fitnessproportional selection of the parents in combination with single-point crossover or use a random uniform simple set to select the fittest agent to produce offspring. A robust society of cooperators emerges only if the level of competition between the players is neither too small nor too large. In elitist selection schemes, the population is firstly shuffled randomly and partitioned into pairs of parents. Then, each pair of parents creates two offspring, and a local competition between parents and their offspring is held. Finally, the best two players of each pair of parents are transferred to the next generation [Thierens and Goldberg (1994)]. In this case, stable societies of highly cooperative players evolve. It shows that a suitable model of the selection process is of crucial importance in terms of simulating
Iterated Prisoner’s Dilemma and Evolutionary Game Theory
47
real-world economic situations [Ficici, Melnik, and Pollack (2000); Bragt, Kemenade and Poutr´e (2001)]. Selection is clearly an important genetic operator, but opinion is divided over the importance of crossover versus mutation. Some argue that crossover is the most important, while mutation is only necessary to ensure that potential solutions are not lost [Grefenstette, Ramsey and Schultz (1990); Wilson (1987)]. Others argue that crossover in a largely uniform population only serves to propagate innovations originally found by mutation, and in a non-uniform population crossover is nearly always equivalent to a very large mutation [Spears (1992)]. 2.4. Evolution of Cooperation A fundamental problem in evolutionary game theory is to explain how cooperation can emerge in a population of self-interested individuals. Axelrod (1984, 1987) attributes the reason of emergence of cooperation to the “shadow of the future”: the likelihood and importance of future interaction. This implies that rewards from cooperation should be mutually expected payoff and to cooperate is a rational choice for self-interested individuals [Martinez-Coll and Hirshleifer (1991)]. Axelrod’s work has been subjected to a number of criticisms because his conclusions obviously conflict with traditional game theory [Binmore (1994, 1998)], as Nachbar’s criticism that “Axelrod mistakenly ran an evolutionary simulation of the finitely repeated Prisoners’ Dilemma. Since the use of a Nash equilibrium in the finitely repeated Prisoners’ Dilemma necessarily results in both players always defecting, we then wouldn’t need a computer simulation to know what would survive if every strategy were present in the initial population of entries. The winning strategies would never co-operate.” [Nachbar (1992)]. There are also arguments that the conflict stems from the assumption of Von Neumann-Morgenstern utility. According to Spiro (1988), the problem with Axelrod’s argument is the oft-discussed problem of interpersonal utility comparison. Axelrod’s argument, and all game theoretic modeling, welfare economics, and utilitarian moral philosophy, in fact, would require that it be possible for one to measure and compare the utilities of different people. The problem with this assumption is that it is quite impossible to construct a scale of measurement for human preferences [Rothbard (1997)]. Although evolutionary game theory is aimed primarily towards dynamic games, while traditional game theory deals with non-dynamic games, there are still area of intersection, for instance in the field of repeated games.
48
S. Y. Chong et al.
Furthermore, although evolutionary game theory mainly depends on experiments and computer simulations, its theoretical foundations, i.e. individual utility (or preference) and payoff-maximizing, stem from traditional game theory. Controversies about Axelrod’s work reflect the bifurcation between evolutionary approaches and the basic assumptions of game theory. Based on the assumption of “rational players”, traditional game theory regards a finite repeated game as a combination of many singleton games. “Backward induction” is applied in order to dissect the link between these singleton games, and then each of them can be analyzed statically [Harsanyi and Selten (1988)]. The concept of backward induction was first employed by Von Neumann and Morgenstern (1944) and then developed by Selten (1965, 1975) based on Nash equilibrium. First, one determines the optimal strategy of the player who makes the last move of the game. Then, the optimal action of the next-to-last moving player is determined taking the last player’s action as given. The process continues in this way backwards through time until all players’ actions have been determined. Subgame perfect Nash equilibrium deduced directly from backward induction is an equilibrium such that players’ strategies constitute a Nash equilibrium in every subgame of the original game [Aumann (1995)]. Selten proved that any game which can be broken into “sub-games” containing a sub-set of all the available choices in the main game will have a subgame perfect Nash equilibrium. In the case of a finite number of iterations in IPD games, the unique subgame perfect Nash equilibrium is AllD. However, many psychological and economic experiments have shown that subjects would not necessarily apply a strategy like AllD [Kahn and Murnighan (1993); McKelvey and Palfrey (1992); Cooper et al. (1996)]. Game theorists explain these experimental results in terms of incomplete information, reputation, and bounded rationality, which are all based on theoretical analysis [Harsanyi (1967); Kreps et al. (1982); Simon (1990); Bolton (1991); Bolton and Ockenfels (2000); Binmore et al. (2002); Samuelson (2001)]. In some sense, Axelrods work is a parallel of these explanations, but it seems that his approach is absolutely different. Before a soundly theoretical explanation can be established, the problem of how cooperation emerges is left unsolved. As to the problem of how cooperation can persist during evolution, sufficient evidence has been provided to support the point that cooperation can survive and flourish in a wide range of circumstances if only some conditions are satisfied. Nowak and Sigmund (1990) have shown that cooperation can emerge among a population of randomly chosen reactive strategies, as long as a stochastic version of TFT is added to the population. If cooperators
Iterated Prisoner’s Dilemma and Evolutionary Game Theory
49
can recognize each other with the help of some label they can increase their payoff by interacting selectively with one another [Frank (1988)]. Social norms aid in cooperation in many ways [Bendor and Mookherjee (1990); Kandori (1992); Sethi and Somanathan (1996)]. As to the influence of payoff variations, Mueller (1988) finds that payoff settings with increasing values of T relative to P promote cooperative behaviour; while Fogel (1993) regards that smaller values for T promote the evolution of cooperative behaviour. Nachbar (1992) selects a payoff setting strongly favouring the relative reward of cooperating and finds that this setting elicits an increased degree of cooperation. Kirchkamp (1995) finds that the value of S becomes less important with longer memory. Also, the effects of population structure, repetition, and noise have been studied [Hirshleifer and Coll (1988); Mueller (1988); Boyd (1989); Marinoff (1992); Hoffmann (2001)]. To end, we note that Binmore (1998) stated: “. . .One simply cannot get by without learning the underlying theory. Without any knowledge of the theory, one has no way of assessing the reliability of a simulation and hence no idea of how much confidence to repose in the conclusions that it suggests”.
There is still a need for an underlying theory for IPD tournaments. Evolutionary game theory has provided us with many experimental approaches; however, better theoretical explanations are still needed. Even though IPD tournaments have been run for over 40 years, we suspect there will be more as we search for new strategies and new theories which explain the complex interactions that take place. Finally, this review has been restricted to the IPD literature. Even so, we have not been able to include every article and there are, no doubt, omissions. However we hope that this chapter has provided enough information for the interested reader to follow up on. References Akimov V. and Soutchanski M. (1994) Automata simulation of N-person social dilemma games, Journal of Conflict Resolution, 38, pp. 138-148. Akin E. (1993) The general topology of dynamical systems, American Mathematics Society, Providence. Akiyama E. and Kaneko K. (1995) Evolution of cooperation, differentiation, complexity and diversity in an iterated three-person game, Artificial Life, 2, pp. 293-304. Alexander H. and Frans B. (1992) Coalitions and Alliances in Humans and Other Animals. Oxford: Oxford University Press.
50
S. Y. Chong et al.
Anthonisen N. (1999) Strong rationalizability for two-player noncooperative games, Economic Theory, 13, pp. 143-169. Aumann R. (1995) Backward Induction and Common Knowledge of Rationality, Games and Economic Behavior, 18, pp. 6-19. Axelrod R. (1980a) Effective choice in the prisoner’s dilemma, Journal of Conflict Resolution, 24, pp. 3-25. Axelrod R. (1980b) More effective choice in the prisoner’s dilemma, Journal of Conflict Resolution, 24, pp. 379-403. Axelrod R. M. (1984). The Evolution of Cooperation (BASIC Books, New York). Axelrod R. (1987) The evolution of strategies in the iterated prisoner’s dilemma, In Davis L., Genetic Algorithms and Simulated Annealing, pp. 32-41. Axelrod R. (1999) The Complexity of Cooperation: Agent-based Models of Competition and Collaboration. University Press, Princeton, NJ. Axelrod R. and Dion D. (1988) The further evolution of cooperation, Science, 242, pp. 1385-1390. Axelrod R. and Hamilton W. (1981) The evolution of cooperation, Science, 211, 4489, pp. 1390-1396. Balkenborg D. and Schlag K. (2000) Evolutionarily stable sets, International Journal of Game Theory, 29, pp. 571-595. Batali J. and Kitcher P. (1994) Evolutionary dynamics of altruistic behaviour in optional and compulsory versions of the iterated prisoner’s dilemma, In Rodney A. and Maes P. Artificial Life IV. MIT Press, pp. 343-348. Beaufils B., Delahaye J., and Mathieu P. (1996) Our meeting with gradual: A good strategy for the iterated prisoner’s dilemma, Proceedings of the Artificial Life V, pp. 202-209. Becker N. and Cudd A. (1990) Indefinitely repeated games: a response to Carroll, Theory and Decision, 28, pp. 189-195. Bendor J. and Mookherjee D. (1990) Norms, third-party sanctions, and cooperation, Journal of Law, Economics, and Organization, 6, pp. 33-63. Bendor R., Kramer M., and Stout S. (1991) When in doubt: cooperation in a noisy prisoner’s dilemma, Journal of Conflict Resolution, 35, pp. 691-719. Ben-porath E. (1990) The complexity of computing a best response automaton in repeated games with mixed strategies, Games and Economic Behavior, 2, pp. 1-12. Berry D. and Fristedt B. (1985) Bandit problems: sequential allocation of experiments. Chapman and Hall, London. Binmore K. (1992) Fun and games. Lexington, MA: D.C. Heath and Company. Binmore K. (1994) Playing fair game theory and the social contract I. MIT Press. Binmore K. (1997) Rationality and backward induction, Journal of Economic Methodology, 4, pp. 23-41. Binmore K. (1998) Review of R. Axelrod’s ‘The complexity of cooperation: agent based models of competition and collaboration’, Journal of Artificial Societies and Social Simulation, 1, 1. Binmore K., McCarthy J., Ponti G., Samuelson L. and Shaked A. (2002) A backward induction experiment, Journal of Economic Theory, 104, pp. 48-88.
Iterated Prisoner’s Dilemma and Evolutionary Game Theory
51
Bishop, D. and Cannings, C. (1978) A generalized war of attrition, Journal of Theoretical Biology, 70, pp. 85-124. Bixenstine V., Potash H., and Wilson K. (1963) Effects of level of cooperative choice by the other player on choices in a Prisoner’s Dilemma game, Journal of Abnormal and Social Psychology, 66, pp. 308-313. Bolton G. (1991) A comparative model of bargaining: theory and evidence, The American Economic Review, 81, 5, pp. 1096-1136. Bolton G. and Ockenfels A. (2000) ERC: a theory of equity, reciprocity, and competition, The American Economic Review, 90, pp. 166-193. Bomze I. (1998) Uniform barriers and evolutionarily stable sets, Game Theory, Experience, Rationality, pp. 225-244. Bomze I. (2002) Regularity vs. degeneracy in dynamics, games, and optimization: a unified approach to different aspects, SIAM Review, 44, pp. 394-414. Boyd R. (1989) Mistakes allow evolutionary stability in the repeated prisoner’s dilemma game, Journal of Theoretical Biology, 136, 11, pp. 47-56. Boyd R. (1992) The evolution of reciprocity when conditions vary, Harcourt A. and Frans B. (eds.) Alliance formation among male baboons: shopping for profitable partners. Oxford: Oxford University Press, pp. 473-489. Boyd R. and Loberbaum J. (1987) No pure strategy is evolutionarily stable in the repeated Prisoner’s Dilemma game, Nature, 327, pp. 58-59. Boyd R. and Richerson P. (1988) The evolution reciprocity in sizable groups, Journal of Theoretical Biology, 132, pp. 337-356. Boyd R. and Richerson P. (1989) The evolution of indirect reciprocity, Social Networks, 11, pp. 213-236. Boyd R. and Richerson P. (1990) Group selection among alternative evolutionarily stable strategies. Journal of Theoretical Biology, 145, pp. 331-342. Boyd R. and Richerson P. (1992) Punishment allows the evolution of cooperation (or anything else) in sizable groups, Ethology and Sociobiology, 13, pp. 171195. Bovens L. (1997) The backward induction argument for the finite iterated prisoners dilemma and the surprise exam paradox, Analysis, 57, 3, pp. 179-186. Bragt D., Kemenade C. and Poutr´e H. (2001) The influence of evolutionary selection schemes on the iterated prisoner’s dilemma, Computational Economics, 17, pp. 253-263. Brauchli K., Killingback T. and Doebeli M. (1999) Evolution of cooperation in spatially structured populations, Journal of Theoretical Biology, 200, pp. 405-417. Brelis M. (1992) Reputed mobster defends his honor. Boston Globe, 1, pp. 23. Bunn G. and Payne R. (1988) Tit-for-tat and the negotiation of nuclear arms control, Arms Control, 9, pp. 207-233. Carmel D. and Markovitch S. (1996) Learning models of intelligent agents, Proceedings of the 13th National Conference on Artificial Intelligence and the 8th Innovative Applications of Artificial Intelligence Conference, 2, pp. 6267.
52
S. Y. Chong et al.
Carmel D. and Markovitch S. (1997) Model-based learning of interaction strategies in multi-agent systems, Journal of Experimental and Theoretical Artificial Intelligence, 10, 3, pp. 309-332. Carmel D. and Markovitch S. (1998) How to explore your opponent’s strategy (almost) optimally, Proceedings of the International Conference on Multi Agent Systems, pp. 64-71. Cohen L. and Machalek R. (1988) A general theory of expropriative crime: an evolutionary ecological approach, American Journal of Sociology, 94, 3, pp. 465-501. Cooper R., Jong D., Forsythe R., and Ross T. (1996) Cooperation without reputation: experimental evidence from prisoner’s dilemma games, Games and Economic Behavior, 12, 2, pp. 187–218. Cressman R., Garay J. and Hofbauer J. (2001) Evolutionary stability concepts for N-species frequency-dependent interactions, Journal of Theoretical Biology, 211, pp. 1-10. Croson R. (2000) Thinking like a game theorist: Factors affecting the frequency of equilibrium play, Journal of Economic Behavior and Organization, 41, 3, pp. 299–314. Crumbaugh C. and Evans G. (1967) Presentation format, other-person strategies, and cooperative behaviour in the prisoner’s dilemma, Psychological Reports, 20, pp. 895-902. Dacey R. and Pendegraft N. (1988) The optimality of Tit-For-Tat, International Interactions, 15, pp. 45-64. Darwen P. and Yao X. (1995) On evolving robust strategies for iterated prisoner’s dilemma, Progress in Evolutionary Computation, volume 956 in Lecture Notes in Artificial Intelligence, Springer, pp. 276-292. Darwen P. and Yao X. (1996) Automatic modularization by speciation, IEEE International Conference on Evolutionary Computation, pp. 88-93. Darwen P. and Yao X. (2001) Why more choices cause less cooperation in Iterated Prisoner’s Dilemma, Proceedings of the 2001 IEEE Congress on Evolutionary Computation. Darwen P. and Yao X. (2002) Coevolution in iterated prisoner’s dilemma with intermediate levels of cooperation: Application to missile defense, International Journal of Computational Intelligence and Applications, 2, 1, pp. 83107. Davis D. and Holt C. (1999) Equilibrium cooperation in two-stage games: Experimental evidence, International Journal of Game Theory, 28, 1, pp. 89-109. Delahaye J. and Mathieu P. (1996) Etude sur les dynamiques du Dilemme It´er´e des Prisonniers avec un petit nombre de strat´egies : Y a-t-il du chaos dans le Dilemme pur?, Publication Interne IT-294, Laboratoire d’Informatique Fondamentale de Lille. Doebeli M., Blarer A., and Ackermann M. (1997) Population dynamics, demographic stochasticity, and the evolution of cooperation, Proceedings of National Academy Society of USA, 94: 5167–5171. Doebeli M. and Knowlton N. (1998) The evolution of interspecific mutualisms, Proceedings of the National Academy of Sciences, 95(15): 8676-8680.
Iterated Prisoner’s Dilemma and Evolutionary Game Theory
53
Donninger C. (1986) Is it always efficient to be nice?, In Paradoxical effects of social behavior, edited by Dickmann A. and Mitter P., Heidelberg, Germany: Physica Verlag, pp. 123-134. Dugatkin L. (1989) N-person games and the evolution of cooperation: a model based on predator inspection in fish, Journal of Theoretical Biology, 142, pp. 123–135. Dugatkin L. (1990) N-person Games and the Evolution of Co-operation: A Model Based on Predator Inspection in Fish, Journal of Theoretical Biology, 142, pp. 123-135. Durrett R. and Levin S. (1998) Spatial aspects of interspecific competition, Theoretical Population Biology, 53, 1, pp. 30-43. Eckert D., Koch S., and Mitl¨ ohner J. (2005) Using the iterated prisoner’s dilemma for explaining the evolution of cooperation in open source communities, Proceedings of the First Conference on Open Source System, pp. 186-191. Ficici S., Melnik O., and Pollack J. (2000) A game-theoretic investigation of selection methods used in evolutionary algorithms, Proceedings of the 2000 Congress on Evolutionary Computation, 2, pp. 880-887. Ficici S. and Pollack J. (2000) Effects of finite populations on evolutionary stable strategies, Proceedings of the 2000 Genetic and Evolutionary Computation, pp. 927-934. Fogel D. (1993) Evolving behaviors in the iterated prisoners dilemma, Evolutionary Computation, 1, 1, pp. 77-97. Fogel D. and Fogel G. (1995) Evolutionary stable strategies are not always stable under evolutionary dynamics, Evolutionary Programming IV, pp. 565-577. Fogel D., Fogel G., and Andrew P. (1997) On the instability of evolutionary stable strategies, BioSystems, 44, pp. 135-152. Fogel G., Andrew P., and Fogel D. (1998) On the instability of evolutionary stable strategies in small populations, Ecological Modelling, 109, pp. 283-294. Frank R. (1988) Passions within reason. The strategic role of the emotions, New York: W.W. Norton & Co. Freund Y., Kearns M., Mansour Y., Ron D., Rubinfeled R., and Schapire R. (1995) Efficient algorithms for learning to play repeated games against computationally bounded adversaries, Proceedings of the Annual Symposium on the Foundations of Computer Science, pp. 332–341. Fudenberg D. and Maskin E. (1986) The Folk Theorem in repeated games with discounting and incomplete information, Econometrica, 54, pp. 533–554. Fudenberg D. and Maskin E. (1990) Evolution and cooperation in noisy repeated games, New Developments in Economic Theory, 80, pp. 274-279. Fudenberg D. and Levine D. (1998) The theory of learning in games. MIT Press. Garay B. and Hofbauer J. (2003) Robust permanence for ecological differential equations: minimax and discretizations, SIAM Journal on Mathematical Analysis, 34, pp. 1007-1093. Gaunersdorfer A. (1992) Time averages for heteroclinic attractors, SIAM Journal on Applied Mathematics, 52, pp. 1476-1489. Gaunersdorfer A., Hofbauer J., and Sigmund K. (1991) On the dynamics of asymmetric games, Theoretical Population Biology, 39, pp. 345-357.
54
S. Y. Chong et al.
Gilboa I. and Matsui A. (1991) Social stability and equilibrium, Econometrica, 59, pp. 859-867. Gilboa I. and Schmeidler D. (2001) A theory of case-based decisions. Cambridge University Press. Gittins J. (1989) Multi-armed bandit allocation indices. Wiley, Chichester, NY. Glance N. and Huberman B. (1993) The outbreak of cooperation, Journal of Mathematical sociology, 17, 4, pp. 281–302. Glance N. and Huberman B. (1994) The dynamics of social dilemmas, Scientific American, 270, pp. 76-81. Glomba M., Filak T., and Kwasnicka H. (2005) Discovering effective strategies for the iterated prisoner’s dilemma using genetic algorithms, 5th International Conference on Intelligent Systems Design and Applications, pp. 356-363. Godfray H. (1992) The evolution of forgiveness, Nature, 355, pp. 206-207. Goldstein J. and Freeman J. (1990) Three-Way Street: Strategic Reciprocity in World Politics. Chicago: University of Chicago Press. Grefenstette J., Ramsey C., and Schultz A. (1990) Learning sequential decision rules using simulation models and competition, Machine Learning, 5, pp. 355-381. Grim P. (1995) The greater generosity of the spatialized prisoner’s dilemma, Journal of Theoretical Biology, 173, pp. 242-248. Grossman W. (2004) New tack wins Prisoner’s Dilemma, Wired News, Lycos. Harborne S. (1997) Common belief of rationality in the finitely repeated prisoners’ dilemma, Games and Economic Behavior, 19, 1, pp. 133-143. Hardin G. (1968) The tragedy of the commons, Science, 162, pp. 1243-1248. Hargreaves H. and Varoufakis Y. (1995) Game theory: a critical introduction. Routledge, London. Harsanyi J. (1967) Games with incomplete information played by Bayesian players, Management Science, 14, 3, pp. 159-182. Harsanyi, J., and Selten, R. (1988) A General Theory of Equilibrium Selection in Games. Cambridge: MIT Press. Hauser M. (1992) Costs of deception: cheaters are punished in rhesus monkeys (Macaca mulatta). Proceedings of the National Academy of Sciences, 89, pp. 12137-12139. Heller J. (1967) The effects of racial prejudice, feedback strategy, and race on cooperative-competitive behaviour, Dissertation Abstracts, 27, pp. 25072508. Hemelrijk C. (1991) Interchange of ’Altruistic’ Acts as an Epiphenomenon. Journal of Theoretical Biology, 153, pp. 131-139. Hingston P. and Kendall G. (2004) Learning versus evolution in iterated prisoner’s dilemma, Proceedings of Congress on Evolutionary Computation, pp. 364372. Hirshleifer J. and Coll J. (1988) What strategies can support the evolutionary emergence of cooperation?, Journal of Conflict Resolution, 32, 2, pp. 367398. Hirshleifer D. and Rasmusen E. (1989) Cooperation in a repeated prisoner’s dilemma with ostracism, Journal of Economic Behavior and Organization, 12, pp. 87-106.
Iterated Prisoner’s Dilemma and Evolutionary Game Theory
55
Hofbauer J. (1981) On the occurrence of limit cycles in the Volterra-Lotka equation, Nonlinear Analysis, 5, pp. 1003-1007. Hofbauer J. (1984) A difference equation model for the hypercycle, SIAM Journal on Applied Mathematics, 44, pp. 762-772. Hofbauer J. (1996) Evolutionary dynamics for bimatrix games: a Hamiltonian system, Journal of Mathematical Biology, 34, pp. 675-688. Hofbauer J. and Sigmund K. (1998) Evolutionary games and population dynamics. Cambridge University Press. Hofbauer J. and Sigmund K. (2003) Evolutionary game dynamics, Bulletin of the American Mathematical Society, 40, pp. 479-519. Hoffmann R. (2001) The ecology of cooperation, Theory and Decision, 50, pp. 101-118. Holland J. (1975) Adaptation in Natural and Artificial Systems. University of Michigan Press, Ann Arbor. Holland J. (1992) Genetic algorithm, Scientific American, 267, 4, pp. 44-50. Holland J. (1995) Hidden Order - How adaptation builds complexity, Reading, Mass.: Addison-Wesley. Huberman B. and Glance N. (1993) Evolutionary games and computer simulations, Proceedings of the National Academy of Sciences, 90, pp. 7716-7718. Ifti M., Killingback T., and Doebeli M. (2004) Effects of neighborhood size and connectivity on the spatial continuous prisoner’s dilemma, Journal of Theoretical Biology, 231, pp. 97-106. Ikegami T. and Kaneko K. (1990) Computer symbiosis - emergence of symbiotic behavior through evolution, Physica D, 42, pp. 235-243. Insko C., Pinkley R., Hoyle R., Dalton B., Hong G., Slim R., Landry P., Holton B., Ruffin P., and Thibaut J. (1987) Individual-group discontinuity: the role of intergroup contact, Journal of Experimental Social Psychology, 23, pp. 250-267. Insko C., Hoyle R., Pinkley R., and Hong G. (1988) Individual-group discontinuity: the role of a consensus rule, Journal of Experimental Social Psychology, 24, pp. 505-519. Insko C., Schopler J., Hoyle R., Dardis G., and Graetz K. (1990) Individual-group discontinuity as a function of fear and greed, Journal of Personality and Social Psychology, 58, pp. 68-79. Insko C., Schopler J., Drigotas S., Graetz K., Kennedy J., Cox C., and Bornstein G. (1993) The role of communication in interindividual-intergroup discontinuity, Journal of Conflict Resolution, 37, pp. 108-138. Kaelbling L. (1993) Learning in embedded systems. The MIT Press, Cambridge, MA. Kaelbling L. and Moore A. (1996) Reinforcement learning: a survey, Journal of Artificial Intelligence Research, 4, pp. 237-285. Kagel J. and Roth A. (1995) The Handbook of Experimental Economics. Princeton University Press. Kahn L. and Murnighan J. (1993) Conjecture, uncertainty, and cooperation in Prisoners’ Dilemma games: Some Experimental Evidence, Journal of Economic Behavior and Organisms, 22, pp. 91–117.
56
S. Y. Chong et al.
Kalai E. and Lehrer E. (1993) Rational learning leads to Nash equilibrium Econometrica, 61, 5, pp. 1019-1045. Kandori M. (1992) Social norms and community enforcement, The Review of Economic Studies, 59, 1, pp. 63-80. Katok A. and Hasselblatt B. (1996) Introduction to the modern theory of dynamical systems. Cambridge ISBN 0521575575. Kavka G. (1986) Hobbesean Moral and Political Theory. Princeton: Princeton University Press. Kirchkamp O. (1995) Spatial Evolution of Automata in the Prisoners’ Dilemma. University of Bonn SFB 303, Discussion Paper B-330. Kitcher P. (1992) Evolution of altruism in repeated optional games, Working Paper of University of California at San Diego. Knapp W. and Podell J. (1968) Mental patients, prisoners, and students with simulated partners in a mixed-motive game, Journal of Conflict Resolution, 12, pp. 235-241. Komorita S., Sheposh J., and Braver S. (1968) Power, the use of power, and cooperative choice in a two-person game, Journal of Personality and Social Psychology, 8, pp. 134-142. Kraines D. and Kraines V. (1995) Evolution of learning among Pavlov strategies in a competitive environment with noise, The Journal of Conflict Resolution, 39, 3, pp. 439-466. Kraines D. and Kraines V. (2000) Natural selection of memory-one strategies for the iterated Prisoner’s Dilemma, Journal of Theoretical Biology, 203, pp. 335-355. Kreps D., Milgrom P., Roberts J., and Wilson R. (1982) Rational cooperation in the finitely repeated prisoner’s dilemma, Journal of Economic Theory, 27, pp. 245–252. Kreps, D., and Wilson R. (1982) Reputation and imperfect information, Journal of Economic Theory, 27, pp. 253–279. Krishna V. and Sj¨ ostr¨ om T. (1998) On the convergence of fictitious play, Mathematics Operations Research, 23, pp. 479-511. Lave L. (1965) Factors affecting cooperation in the prisoner’s dilemma, Behavioral Science, 10, pp. 26-38. Lindgren K. (1991) Evolutionary phenomena in simple dynamics, In Christopher G., et al. Santa Fe Institute Studies in the Sciences of Complexity. 10, pp. 295-312. Lindgren K. (1992) Evolutionary phenomena in simple dynamics, In Langton C. (ed.) Artificial Life II. Addison-Wesley. Lindgren K. (1995) Evolutionary dynamics in game-theoretic models, The economy as an evolving complex system II, Santa Fe Institute. Littman M. (1994) Markov games as a framework for multiagent reinforcement learning, Proceedings of the 11th International Conference on Machine Learning, pp. 157-163. Luce R. and Raiffa H. (1957) Games and decisions. New York: Wiley. Lynch G. (1968) Defense preference and cooperation and competition in a game, Dissertation Abstracts, 29, pp. 1174.
Iterated Prisoner’s Dilemma and Evolutionary Game Theory
57
Manarini S. (1998) The prisoner’s dilemma, experiments for the study of cooperation. Strategies, theories and mathematical models, Ph.D. thesis, University of Padova. Marinoff L. (1992) Maximizing expected utilities in the Prisoner’s Dilemma, Journal of Conflict Resolution, 36, 1, pp. 183-216. Martinez-Coll J. and Hirshleifer J. (1991) The limits of reciprocity, Rationality and Society, 3, pp. 35-64. Matsui A. (1992) Best response dynamics and socially stable strategies, Journal of Economic Theory, 57, pp. 343-362. May R. (1987) More evolution of cooperation, Nature, 327, pp. 15-17. Maynard Smith J. and Price G. (1973) The logic of animal conflict, Nature, 246, pp. 15-18. Maynard Smith J. (1982) Evolution and the Theory of Games, Cambridge University Press. McKelvey R. and Palfrey T. (1992) An experimental study of the centipede game, Econometrica, 60, pp. 803-836. Mealey L. (1995) The sociobiology of sociopathy: an integrated evolutionary model, Behavioral and Brain Sciences, 18, 3, pp. 523-599. Michalewicz Z. (1999) Genetic Algorithms + Data Structures = Evolution Programs, Springer-Verlag. Micko H. (1997) Benevolent tit for tat strategies with fixed intervals between offers of cooperation, Meeting of Experimental Psychologists, pp. 250-256. Micko H. (2000) Experimental Matrix games, In Open and Distance Learning-Mathematical Psychology, Institut f¨ ur Sozial- und Pers¨ onlichkeitspsychologie, Universit¨ at Bonn. Milgrom, P. and Roberts J. (1982): Predation, reputation and entry deterrence, Journal of Economic Theory, 27, pp. 280-312. Milinski M. (1993) Cooperation wins and stays, Nature, 364, pp. 12-13. Milinski M. and Wedekind C. (1998) Working memory constrains human cooperation in the prisoner’s dilemma, Proceedings of the National Academy of Sciences of the United States of America, 95, 23, pp. 13755-13758. Miller J. (1996) The coevolution of automata in the repeated prisoner’s dilemma, Journal of Economic Behavior and Organization, 29, pp. 87-112. Mitchell M. (1996) An introduction to Genetic Algorithms. The MIT Press, Cambridge MA. Molander P. (1985) The optimal level of generosity in a selfish, uncertain environment, Journal of Conflict Resolution, 29, pp. 611-618. Moore A. and Atkeson C. (1993) Prioritized sweeping: reinforcement learning with less data and less real time, Machine Learning, 13, pp. 103-130. Mueller U. (1988) Optimal retaliation for optimal cooperation, Journal of Conflict Resolution, 31, 4, pp. 692-724. Myerson R. (1991) Game Theory, Analysis of Conflict. Cambridge, Harvard University Press. Nachbar J. (1992) Evolution in the finitely repeated Prisoners’ Dilemma, Journal of Economic Behavior and Organization, 19, pp. 307-326.
58
S. Y. Chong et al.
Narendra K. and Thathachar M. (1989) Learning automata: an introduction. Prentice-Hall, Englewood Cliffs, NJ. Nash J. (1950) Equilibrium points in n-person games, Proceedings of the National Academy of the USA, 36, 1, pp. 48-49. Nash J. (1951) Non-cooperative games, The Annals of Mathematics, 54, 2, pp. 286-295. Nash J. (1996) Essays on Game Theory. Elgar. Cheltenham. Noldeke G. and Samuelson L. (1993) An evolutionary analysis of backward and forward induction, Games and Economic Behaviour, 5, pp. 425-454. Nowak M., Bonhoeffer S., and May R. (1994) More spatial games, International Journal of Bifurcation and Chaos, 4, 1, pp. 33-56. Nowak M. and May R. (1992) Evolutionary games and spatial chaos, Nature, 359, pp. 826-829. Nowak M. and May R. (1993) The spatial dilemmas of evolution, International Journal of Bifurcation and Chaos, 3, pp. 35-78. Nowak M. and Sigmund K. (1990) The evolution of stochastic strategies in the prisoner’s dilemma, Acta Applicandae Mathematicae, 20, pp. 247-265. Nowak M. and Sigmund K. (1992) Tit for tat in heterogeneous populations, Nature, 359, pp. 250-253. Nowak M. and Sigmund K. (1993) A strategy of win-stay lose-shift that outperforms Tit-for-Tat in the Prisoner’s Dilemma game, Nature, 364, pp. 56-58. Nowak M., Sigmund K. and El-Sedy E. (1995) Automata, repeated games, and noise, Journal of Mathematical Biology, 33, pp. 703-722. Orbell J., Kragt A., and Dawes R. (1988) Explaining discussion-induced cooperation, Journal of Personality and Social Psychology, 54, pp. 811-819. Orbell J. and Dawes R. (1993) Social welfare, cooperator’s advantage, and the option of not playing the game, American Sociological Review, pp. 787-800. Orbell J. and Robyn M. (1993) Social welfare, cooperators’ advantage, and the option of not playing the game. American Sociological Review, 58, pp. 787800. Oskamp S. (1971) Effects of programmed strategies on cooperation in the prisoner’s dilemma and other mixed-motive games, The Journal of Conflict Resolution, 15, 2, pp. 225-259. Oskamp S. and Perlman D. (1965) Factors affecting cooperation in a prisoner’s dilemma game, Journal of Conflict Resolution, 9, pp. 359-374. Plank M. (1997) Some qualitative differences between the replicator dynamics of two player and n player games, Nonlinear Analysis, 30, pp. 1411-1417. Pollock G. (1989) Evolutionary Stability of Reciprocity in a Viscous Lattice. Social Networks, 11, pp. 175-212. Posch M. (1997) Win Stay–Lose Shift: An Elementary Learning Rule for Normal Form Games, Working Paper of Santa Fe Institute, http://ideas.repec.org/p/wop/safire/97-06-056e.html. Prisoner’s dilemma tournament result (2004) http://www.prisoners-dilemma. com/results/cec04/ipd cec04 full run.html. Prisoner’s dilemma tournament result (2005) http://www.prisoners-dilemma. com/results/cig05/cig05.html.
Iterated Prisoner’s Dilemma and Evolutionary Game Theory
59
Radner R. (1980) Collusive behaviour in non-cooperative epsilon-equilibria in oligopolies with long but finite lives, Journal of Economic Theory, 22, pp. 136-154. Radner R. (1986) Can bounded rationality resolve the prisoner’s dilemma, In Mas- Colell A. and Hildenbrand W. Essays in Honor of Gerard Debreu, pp. 387-399. Rapoport A. (1966) Optimal policies for the prisoner’s dilemma, Technical Report No. 50 Psychometric Laboratory, University of North California, MH-10006. Rapoport A. (1999) Two-person Game Theory. Dover Publications, New York. Rapoport and Chammah (1965) Prisoner’s dilemma: a study in conflict and cooperation. Ann Arbor: University of Michigan Press. Rothbard M. (1997) Toward a Reconstruction of Utility and Welfare Economics, In The Logic of Action One: Method, Money, and the Austrian School, pp. 211-55. Rubinstein A. (1979) Equilibrium in super games with the overtaking criterion, Journal of Economic Theory, 21, pp. 1-9. Rubinstein A. (1998) Modeling bounded rationality. The MIT Press, 1998. Samuelson L. (2001) Introduction to the evolution of preferences, Journal of Economic Theory, 97, pp. 225-230. Sandholm T. and Crites R. (1996) Multiagent reinforcement learning in the iterated Prisoner’s Dilemma, Biosystems, 37, 1-2, pp. 147-66. Sarin R. (1999) Simple play in the prisoner’s dilemma, Journal of Economic Behavior and Organization, 40, 1, pp. 105–113. Sasaki A., Taylor C. and Fudenberg D. (2000) Emergence of cooperation and evolutionary stability in finite populations, Nature, 428, pp. 646-650. Schmidhuber J. (1996) A general method for multi-agent learning and incremental self-improvement in unrestricted environments, In Yao X. (ed.) Evolutionary Computation: Theory and Applications. Scientific Publications Co. Schmitt L. (2001) Theory of genetic algorithms, Theoretical Computer Science, 259, pp. 1-61. Schuessler R. (1989) Exit threats and cooperation under anonymity, Journal of Conflict Resolution, 33, pp. 728-749. Schweitzer F. (2002) Modeling Complexity in Economic and Social Systems. World Scientific, Singapore. Schweitzer F., Behera L., and M¨ uhlenbein H. (2002) Evolution of cooperation in a spatical prisoner’s dilemma, Advances in Complex Systems, 5, 2-3, pp. 269-299. Scodel A., Minas J., Ratoosh P., and Lipetz M. (1959) Some descriptive aspects of two-person non-zero sum games, Journal of Conflict Resolution, 3, pp. 114119. Selten, R. (1965) Spieltheoretische behandlung eines oligopolmodells mit nachfragetragheit, Zeitschrift fur die Gesamte Staatswissenschaft, 12, pp. 301324. Selten, R. (1975) Reexamination of the perfectness concept for equilibrium points in extensive games, International Journal of Game Theory, 4, pp. 25-55.
60
S. Y. Chong et al.
Selten R. (1983) Evolutionary stability in extensive two-person games, Mathematical Social Science, 5, pp. 269-363. Selten R. (1988) Evolutionary stability in extensive two-person games: correction and further development, Mathematical Social Science, 16, pp. 223-266. Selten R. and Stoecker R. (1986) End behaviour in sequences of finite Prisoner’s Dilemma supergames: a learning theory approach, Journal of Economic Behaviour and Organisation, 7, pp. 47-70. Sethi R. and Somanathan E. (1996) The evolution of social norms in common property resource use, The American Economic Review, 86, 4, pp. 766-788. Simon H. (1955) A behavioral model of rational choice, Quarterly Journal of Econometrics, 69, 1, pp. 99-118. Simon H. (1990) A mechanism for social selection and successful altruism, Science, 250, 4988, pp. 1665-1668. Sermat V. (1967) Cooperative behaviour in a mixed-motive game, Journal of Social Psychology, 62, pp. 217-239. Sigmund K. (1995) Games of Life: Explorations in Ecology, Evolution and Behaviour. Penguin, Harmondsworth. Skyrms B. (1990) The Dynamics of Rational Deliberation. Harvard UP. Smith H. (1995) Monotone dynamical systems: an introduction to the theory of competitive and cooperative systems, AMS Mathematical Surveys and Monographs, 41. Smith R. and Gray B. (1994) Co-adaptive genetic algorithms: an example in Othello strategy, Proceedings of the 1994 Florida Artificial Intelligence Research Symposium, pp. 259-264. Sobel J. (1975) Reexamination of the perfectness concept of equilibrium in extensive games, International Journal of Game Theory, 4, pp. 25-55. Sobel J. (1976) Utility maximization in iterated Prisoner’s Dilemmas, Dialogue, 15, pp. 38-53. Solomon L. (1960) The influence of some types of power relationships and game strategies upon the development of interpersonal trust, Journal of Abnormal and Social Psychology, 61, pp. 223-230. Spears W. (1992) Crossover or mutation? Foundations of Genetic Algorithms. 2, FOGA-92, edited by Whitley D., California: Morgan Kaufmann. Spiro D. (1988) The state of cooperation in theories of state cooperation: the evolution of a category mistake, Journal of International Affairs, 42, pp. 205225. Stanley E., Ashlock D., and Smucker M. (1995) Iterated prisoner’s dilemma with choice and refusal of partners: Evolutionary results, Lecture Notes in Artificial Intelligence, 929, pp. 490-502. Stanley E., Ashlock D., and Tesfatsion L. (1994) Iterated prisoner’s dilemma with choice and refusal of partners, In Christopher G. Artificial Life III. Addison-Wesley, pp. 131-176. Stephens D. (2000) Cumulative benefit games: achieving cooperation when players discount the future, Journal of Theoretical Biology, 205, 1, pp. 1-16. Stephens D., Mclinn C., and Stevens J. (2002) Discounting and Reciprocity in an Iterated Prisoner’s Dilemma, Science, 298, 5601, pp. 2216-2218.
Iterated Prisoner’s Dilemma and Evolutionary Game Theory
61
Sugden, R. (1986) The Economics of Cooperation, Rights and Welfare. Basil Blackwell. Surowiecki J. (2004) The Wisdom of Crowds: Why the Many Are Smarter Than the Few and How Collective Wisdom Shapes Business, Economies, Societies and Nations. Little, Brown. Sutton R. (1990) Integrated architectures for learning, planning, and reacting based on approximating dynamic programming, Proceedings of the 7th International Conference on Machine Learning, pp. 216-224. Swingle P. and Coady H. (1967) Effects of the partner’s abrupt strategy change upon subject’s responding in the prisoner’s dilemma, Journal of Personality and Social Psychology, 5, pp. 357-363. Swinkels J. (1993) Adjustment dynamics and rational play in games, Games and Economic Behavior, 5, pp. 455-84. Taylor, P. D. (1979). Evolutionarily stable strategies with two types of players, Journal of Applied Probability, 16, pp. 76-83. Taylor, P. and Jonker, L. (1978) Evolutionary stable strategies and game dynamics, Mathematical Biosciences, 40, pp. 145-156. Tekol Y. and Acan A. (2003) Ants can play Prisoner’s Dilemma, Proceedings of the 2003 Congress on Evolutionary Computation, pp. 1151-1157. Thierens D. (1997) Selection schemes, elitist recombination, and selection intensity, Proceedings of the 7th International Conference on Genetic Algorithms, pp. 152-159. Thierens D. and Goldberg D. (1994) Elitist recombination: an integrated selection recombination GA, Proceedings of the First IEEE Conference on Evolutionary Computation, pp. 508-512. Thomas B. (1985) On evolutionarily stable sets, Journal of Mathematical Biology, 22, pp. 105-115. Tzafestas E. (2000a) Toward adaptive cooperative behavior, Proceedings of the Simulation of Adaptive Behavior Conference, pp. 334-340. Tzafestas E. (2000b) Spatial games with adaptive tit-for-tats, Proceedings of the 6th Parallel Problem Solving from Nature (PPSN-VI), pp. 507-516. Young H. and Foster D. (1991) Cooperation in the Short and in the Long Run, Games and Economic Behavior, 3, pp. 145-156. Vegaredondo F. (1994) Bayesian boundedly rational agents play the finitely repeated prisoner’s dilemma, Theory and Decision, 36, 2, pp. 187–206. Von Neumann J. and Morgenstern O. (1944) Theory of Games and Economic Behavior. Princeton UP. Watkins C. (1989) Learning from delayed rewards. Ph.D. thesis, King’s College, Cambridge, UK. Watkins C. and Dayan P. (1992) Q-learning, Machine Learning, 8, 3, pp. 279-292. Wedekind C. and Milinski M. (1996) Human cooperation in the simultaneous and the alternating Prisoner’s Dilemma: Pavlov versus Generous Tit-forTat, Proceedings of the National Academy of Sciences of the United States of America, 93, 7, pp. 2686-2689. Weibull J. (1995) Evolutionary Game Theory. MIT Press, Cambridge, Mass.
62
S. Y. Chong et al.
Wilson W. (1969) Cooperation and the cooperativeness of the other player, Journal of Conflict Resolution, 13, pp. 110-117. Wilson W. (1987) Classifier systems and the animat problem, Machine Learning, 2, pp. 199-228. Whitworth R. and Lucker W. (1969) Effective manipulation of cooperation with college and culturally disadvantaged populations, Proceedings of 77th Annual Convention of American Psychological Association, 4, pp. 305-306. Wu J. and Axelrod R. (1995) How to cope with noise in the Iterated Prisoner’s Dilemma, Journal of Conflict Resolution, 39, pp. 183-189. Zeeman M. (1993) Hopf bifurcations in competitive three dimensional LotkaVolterra systems, Dynamics and Stability of Systems, 8, pp. 189-217 Zeeman E., Zeeman M. (2002) An n-dimensional competitive Lotka-Volterra system is generically determined by its edges, Nonlinearity, 15, pp. 2019-2032. Zeeman E., Zeeman M. (2003) From local to global behavior in competitive Lotka-Volterra systems, Transaction of American Mathematical Society, 355, pp. 713-734.
Chapter 3 Learning IPD Strategies Through Co-evolution
Siang Yew Chong1 , Jan Humble2 , Graham Kendall2 , Jiawei Li2,3 , Xin Yao1 University of Birmingham1 , University of Nottingham2 , Harbin Institute of Technology3
3.1. Introduction Complex behavioral interactions can be abstracted and modelled using a game. One particular aspect in modelling interactions that is of great interest is in understanding the specific conditions that lead to cooperation between selfish individuals. The iterated prisoner’s dilemma (IPD) game is one famous example. In its classical form, two players engaged in repeated interactions, are given two choices: cooperate and defect [Axelrod (1984)]. The dilemma of the game is captured by having both players who are better off mutually cooperating than mutually defecting being vulnerable to exploitation by one of the party who defects. Although the IPD game has become a popular model to study conditions for cooperation to occur among selfish individuals, which was due in large part to a series of tournaments reported in [Axelrod (1980a,b)], it has also received much attention in many other areas of study, and used to model social, economic, and biological interactions [Axelrod (1984)]. The classical IPD can be easily defined as a nonzero-sum, noncooperative, two-player game [Chellapilla and Fogel (1999)]. It is nonzero-sum because the benefits that a player obtains do not necessarily lead to similar penalties given to the other player. It is noncooperative because it assumes no preplay communication between the two players. The IPD game can be formulated by considering a predefined payoff matrix that specifies the payoff that a player receives for the choice it makes for a particular move given the choice that the opponent makes. Referring 63
S. Y. Chong et al.
64
to the payoff matrix given by figure 3.1, both players receive R (reward ) units of payoff if both cooperates. They both receive P (punishment) units of payoff if they both defect. However, when one player cooperates while the other defects, the cooperator will receive S (sucker ) units of payoff while the defector receives T (temptation) units of payoff. With the IPD game, the values R, S, T , and P must satisfy the constraints; T > R > P > S and R > (S + T )/2. Axelrod in [Axelrod (1980a,b)] used the following set of values: R = 3, S = 0, T = 5, and P = 1. However, any set of values can be used as long as they satisfy the IPD constraints. The game is played when both players choose between the two alternative choices over a series of moves (i.e., repeated interactions). Note that the game is fully symmetric, i.e., the same payoff matrix is applied to both players.
Cooperate
Defect R
T
Cooperate R
S S
P
Defect T
P
Fig. 3.1. The payoff matrix framework of a two-player, two-choice game. The payoff given in the lower left-hand corner is assigned to the player (row) choosing the move, while that of the upper right-hand corner is assigned to the opponent (column).
For the simple case of the one-shot prisoner’s dilemma (both players only get to make one move), the rational play will be to defect [Chellapilla and Fogel (1999)]. This can be viewed by considering the obtained payoff for a choice made by a player in light of the opponent’s. For example, a cooperating player will receive either R (opponent cooperates) or S (opponent defects). A defecting player will receive either T (opponent cooperates) or P (opponent defects). As such, from the player’s point of view (i.e., selfinterested), the rational play will be to defect because regardless of the opponent’s play, a higher payoff is obtained (T > R and P > S). However, when the game is iterated over many rounds of moves and that players can adopt game strategies where a response is based on what happened in the previous moves, defection is not necessarily the best choice of play. Instead, many studies have shown cooperative play to be a viable
Learning IPD Strategies Through Co-evolution
65
strategy, starting with the tournaments organized by Axelrod (reported in [Axelrod (1980a,b)]). More importantly, later studies (of which Axelrod himself is one of the early pioneers) showed that cooperative strategies can be learned from an initial, random population using evolutionary algorithms [Axelrod (1987); Fogel (1991, 1993); Darwen and Yao (1995)]. In particular, studies made in [Axelrod (1987); Fogel (1991, 1993); Darwen and Yao (1995)] (and many others) used a co-evolutionary learning approach. The motivation for the co-evolutionary learning approach is the learning of strategy behaviors through an adaptation process on strategy representations based solely on interactions (i.e., game-play). This approach is different compared to the classical evolutionary game approach (and also the ecological game approach used in [Axelrod (1980b); Axelrod and Hamilton (1981)]) that is mainly concerned with frequency dependent reproductions of fixed and predetermined strategies. As such, the use of co-evolutionary learning approach allows for one to construct a game (i.e., specifying the possible interactions between players, the rules that govern the interactions, and the payoffs) and then to search for effective game strategies without the need of human intervention (e.g., specify viable strategies) [Chellapilla and Fogel (1999)]. Within the framework of the co-evolutionary learning of game strategies, it is natural to explore more complex interactions that is closer to real-world interactions compared to highly abstracted models like the classical IPD. This review aims to provide a survey of studies using the co-evolutionary learning approach of more complex IPD games since the tournaments organized by Axelrod that were held almost 20 years ago. In particular, focus is placed on the motivations of certain extensions to the classical IPD and the general observations made when co-evolutionary learning systems are used. The following section describes the framework of co-evolutionary learning and the general issues of co-evolving IPD strategies. Section 3.2 surveys studies that extend the classical IPD with more choices, noise, N-players, and others. The review concludes with some remarks on the future directions for research in co-evolutionary learning of IPD strategies. It is emphasized again that this review focusses on the co-evolutionary learning approach to IPD games, rather than all possible work related to IPD games.
S. Y. Chong et al.
66
3.2. Co-evolving Strategies for the IPD Game 3.2.1. Co-evolutionary Learning Framework Co-evolutionary learning refers to a broad class of population-based, stochastic search algorithms that involves the simultaneous evolution of competing solutions (to a problem) with coupled fitness [Yao (1994)]. A coevolutionary learning system can be implemented using evolutionary algorithms (EAs) [Fogel (1994a); B¨ ack et al. (1997)]. That is, a co-evolutionary learning system iteratively apply the process of variation (e.g., mutation, crossovers, and others) and selection (e.g., choosing solutions to procreate in the next iterative step) on the competing solutions in the population. With this view, the framework of co-evolutionary learning (and also that of EAs) can be illustrated using figure 3.2.
(1) Initialize the population, X(t=0) (2) Evaluate the fitness of each individual through a comparison process with other individuals in X(t) (3) Select parents from X(t) based on their evaluated fitness (4) Generate offsprings from parents to produce X(t+1) (5) Repeat steps (2-4) until some termination criteria are reached
Fig. 3.2.
The general framework of co-evolutionary learning.
Co-evolutionary learning is different from EAs in the sense of assigning fitness, i.e., the quality or worth of a solution (Step 2 in Fig. 3.2). EAs are often viewed and constructed in terms of an optimization context, whereby an absolute fitness function is required to assign fitnesses to contending solutions. With co-evolutionary learning, the fitness of a solution is obtained through its interactions with other contending solutions in the population. That is, a solution fitness in a co-evolutionary learning system is relative and dynamic because a solution’s fitness not only depends on the population, but also changes as the composition of solutions in the population changes. Although the difference between co-evolutionary learning systems and traditional EAs appear to be small at first, from the contexts of certain problems, it can lead to significantly different outcomes. For example, consider the problem of searching for optimal solutions. In many real-world
Learning IPD Strategies Through Co-evolution
67
problems, designing a suitable fitness function that can lead to the search of solutions can be very difficult, if possible [Yao (1994)]. However, with co-evolutionary learning, this need of having a fitness function is essentially removed. Instead, a co-evolutionary learning system only needs to be able to rank contending solutions based on how they compared to one another. Here, games are well-suited, natural problem applications for coevolutionary learning systems. In particular, although games can be approached from an optimization context, it may not be possible to construct a fitness function that fully represent the problem of the game and fully discriminate solutions found through optimization algorithms. With coevolutionary learning, however, the search can be directed to find for better game strategies (e.g., defeat more strategies) as the evolutionary process continues [Chellapilla and Fogel (1999)]. In particular, for the IPD game, there have been many different approaches since Axelrod’s early study in [Axelrod (1987)] that investigated a particular co-evolutionary learning system. Like the study of EAs (commonly known as Evolutionary Computation) [Yao (1994); Fogel (1994a); B¨ ack et al. (1997); Fogel (1995); B¨ ack (1996)], there are a wide variety of specific strategy representations, selection and variation operators in the co-evolutionary learning approach used for the IPD game. A complete survey is beyond the scope of this chapter. Instead, the more popular choices will be reviewed here. The important thing to note is that all the coevolutionary learning systems used were based on the framework illustrated in figure 3.2, i.e., they involved an adaptation process on IPD strategies in some form of representations (involves variations and selection) based on interactions (game-play between strategies). For strategy representations, particularly on deterministic and reactive IPD strategies that were mostly studied, Axelrod and Lindgren [Axelrod (1987); Lindgren (1991)] were among the first few who used binary strings of ones (cooperation) and zeroes (defection) encoding for a look-up table (essentially a binary decision tree) representation. The look-up table in particular determines the outcome for the strategy based on the pairs of previous moves made by the strategy and the opponent. Since the strategies require histories of previous moves in order to make a response, they are encoded with the necessary histories for previous moves. We [Chong and Yao (2005)] recently introduced a look-up table representation that directly represents IPD strategies based on responses to previous moves. For the case of looking back the previous pair of moves made by the strategy and the opponent, direct look-up table represents the strategy responses as a
68
S. Y. Chong et al.
two-dimensional table. Each table element represents the response based on the pair of previous moves. Instead of some fictitious histories required to start the game, the direct look-up table specifies the first move directly. Fogel among many others [Fogel (1991, 1993, 1996); Miller (1989); Stanley et al. (1995)] used finite state machines (FSMs) for their capability of representing complex behaviors of IPD strategies. With FSMs, behavioral responses of an IPD strategy based on previous moves depend on the states and the next-state transitions. The motivation for using FSM compared to look-up table is to have a behavioral representation of IPD strategies instead of the look-up table representation of responses based on histories of previous moves (see [Fogel (1993)] for the full discussion on the origin of using FSM and evolution to simulate intelligent behaviors). In addition to the simple look-up table and FSM, neural network representations had also been experimented with and studied [Harrald and Fogel (1996); Darwen and Yao (2000); Chong and Yao (2005); Franken and Engelbrecht (2005)]. Although neural networks are primarily used for their ability of providing nonlinear input-output responses [Chellapilla and Fogel (1999)], the initial motivation to representing IPD strategies also include the capability of neural networks to process and represent a continuous range of behaviors [Harrald and Fogel (1996)]. After selecting a strategy representation, the next step is to consider the design of variation operators that are aimed at providing variations of IPD strategies in the population. In most cases, variation operators are dependent of the strategy representation considered. For example, look-up table encoded as binary strings can use crossovers and bit-flip mutation as in the case of standard genetic algorithms [Axelrod (1987)]. For the case of FSMs, variation operators may include altering a next-state transition, adding or removing states, and altering the output symbol (corresponding to making a choice). With neural networks, especially those that are realvalued representations, self-adapting mutations based on some probability distribution (i.e., Gaussian or Cauchy) can be used [Chong and Yao (2005)] (one of us has provided a comprehensive review on evolving neural networks in [Yao (1999)]). As for designing the process of selecting IPD strategies for the next generation, many other selection operators can be used (those found in EAs [Fogel (1994a); B¨ ack et al. (1997)]) and not just limited to proportional selection used by Axelrod in the first study of co-evolving IPD strategies [Axelrod (1987)]. For the case of obtaining the fitness for a particular IPD strategy in the population, payoffs obtained from the IPD game are usually
Learning IPD Strategies Through Co-evolution
69
used. In particular, many studies considered calculating the expected IPDpayoff-based-fitness using a round robin tournament whereby all pairs of strategies compete, including the pair where a strategy plays itself. 3.2.2. Shadow of the Future In the IPD game, the shadow of the future refers to the situation whereby the number of moves of a game is known in advance. In this situation, there is no incentive to cooperate in the last move because there is no risk of retaliation from the opponent. However, if every player defects on the last move, then there is no incentive to cooperate in the move prior to the last one. If every player defects in the last two moves, then there is no incentive to cooperate in the move before that, and so forth. As such, we would end up with mutual defection in all moves. One popular way to address this issue and to allow for cooperation to emerge is to have a fixed probability in ending the game on every move, thereby keeping the game length uncertain. Most of the studies that used the co-evolutionary learning approach considered a fixed game length (number of moves) in all game plays. For example, Axelrod [Axelrod (1987)] and others such as [Fogel (1991, 1993); Chong and Yao (2005)] used 150 moves (move start from 0). Other game lengths can be used, although the choice depends on the motivation of the study, e.g., a sufficiently long game length to allow for strategies to reciprocate cooperation. In any case, the fixed game length is used because the strategy representation cannot count the number of moves that have been played and how many more remain. 3.2.3. Issues for Co-evolutionary Learning of IPD Strategies For the IPD game, there are two main contexts in which co-evolutionary learning can be considered. First, co-evolutionary learning can be used to search for effective strategies, given the specific the rules of the game that govern the complexity of strategy interactions. Second, a co-evolutionary learning system can serve as a model for investigating how certain conditions (e.g., game rules, co-evolutionary learning system setup, or others) can lead to the evolution of certain behaviors. For the context of using co-evolutionary learning to search for effective strategies, the main issue is to evolve IPD strategies that perform well (e.g., defeat) against a large number of opponents. Axelrod [Axelrod (1987)] used a co-evolutionary learning system and compared the evolved strategies
70
S. Y. Chong et al.
with the representative strategies (e.g., tit for tat) obtained from his earlier tournaments that accounted for average performance of all strategies that participated the tournaments [Axelrod (1980a,b)]. He noted that some of the evolved strategies outperformed these representative strategies. Although results obtained from evolving effective IPD strategies were promising, the study in [Axelrod (1987)] had the important implication on specifying a principled method to determine the effectiveness (or robustness [Axelrod and Hamilton (1981)]) of evolved IPD strategies by testing them against some representative strategies. One of us (Yao) first framed this particular study in the context of generalization [Darwen and Yao (1995); Yao et al. (1996)]. In particular, co-evolutionary learning is a machine learning system that can be analyzed for its generalization performance. Here, the generalization performance of a co-evolutionary learning system for the IPD game can be thought of as the performance of the best strategy in the population or the population itself (e.g., using a gating algorithm that effectively combines different IPD strategies of the population as a single strategy entity [Darwen (1996); Darwen and Yao (1997)]) against a large number of IPD strategies, especially those that the evolved strategies have yet to play with during evolution. For the context of using co-evolutionary learning as a model to understand the conditions of how, why, and what IPD strategy behaviors are evolved, there are many issues that can be studied. First, one can consider the impact of specific IPD game specifications (e.g., payoff matrices [Fogel (1993)] and duration of interactions or game length [Fogel (1996)]) on evolved IPD strategy behaviors. Second, there are also studies that have focused on the impact of the interaction or game-play itself, which are not just limited to noisy interactions [Julstrom (1997)], continuous behavioral responses [Harrald and Fogel (1996)], and the possibility of refusal to interact [Stanley et al. (1995)]. Third, the specific the design of the co-evolutionary learning system itself can have an impact whereby certain IPD behaviors are favored and persist for a long period (e.g., investigating whether systems that provided genotypic diversity actually lead to a diverse population of IPD strategies with a variety of behaviors [Darwen and Yao (2000, 2001, 2002)]). 3.3. Extending the IPD Game The primary motivation in most studies that extend the classical IPD game is to model more complex IPD interactions that are closer to real-world
Learning IPD Strategies Through Co-evolution
71
interactions. This section describes some of the extended IPD games that have been investigated using the co-evolutionary learning approach. Each subsection starts with the motivation for extending the IPD game in a specific manner, and the important issues of studying the more complex IPD games. Each subsection discusses and concludes general observations obtained from the co-evolutionary learning of the particular extended IPD game. 3.3.1. Extending the IPD with More Choices Several studies have extended the classical IPD with more than two extreme choices that are available for play. That is, there are intermediate choices between full cooperation and full defection that strategies can response with. Fogel [Harrald and Fogel (1996)] investigated a continuous IPD game. We have investigated the IPD with multiple, discrete levels of cooperation [Darwen and Yao (2000, 2001, 2002); Chong and Yao (2005)], which could be use to approximate the continuous IPD game when the number of levels is sufficiently large. The main motivation of extending the IPD with more choices is to allow for the modelling of subtle behavioral interactions that are not possible with only two extreme choices. With the classical IPD game, the possible behaviors that strategies can exhibit are severely limited. For example, a strategy for the classical IPD game cannot play intermediate choices that allow for some degree of exploitation of the opponent without risking retaliation from an otherwise cooperative opponent [Harrald and Fogel (1996)]. The co-evolutionary learning approach usually considers a neural network strategy representation because it can be used to process a continuous range of behaviors (i.e., real numbers for representing the degree of cooperation) easily. Furthermore, for the case of IPD games with multiple, discrete levels of cooperation, a neural network is scalable to the number of levels considered. Fogel [Harrald and Fogel (1996)] showed that for the extended IPD a continuous range of choices, the evolution of cooperation is unstable, with fluctuations of average scores representing short periods of cooperation and defection. We have further shown that with increasingly higher number of choices to play in the IPD game with multiple, discrete levels of cooperation, evolution to cooperation are more difficult to achieve [Darwen and Yao (2000, 2001, 2002)]. From these studies, it appears that a co-evolving population of IPD
72
S. Y. Chong et al.
strategies has a higher tendency of evolving to play full defection. However, this does not mean that evolution to cooperation is not possible, or that cooperative behaviors that persist cannot be evolved. For example, it has been shown that evolving cooperative behaviors depends on the complexity of strategy representation that is used. In the case of neural networks, the number of nodes in the hidden layer can affect the co-evolutionary learning system to produce IPD strategies with cooperative responses [Harrald and Fogel (1996)]. In addition to the complexity of strategy representation, another important factor for evolving cooperative strategies is that of behavioral diversity. Early studies [Darwen and Yao (2000, 2001)] have shown that genetic diversity (i.e., variations at the genotypic level of strategy representations) does not equate to behavioral diversity (i.e., variations of IPD strategy responses) in the population. Without sufficient behavioral diversity, the co-evolving population can overspecialize to a specific strategy behavior that is vulnerable to invasion (e.g., cycles between tit for tat, naive cooperators, and defectors). As such, increasing the level of genetic diversity in the co-evolutionary learning system does not necessarily lead to an increase in behavioral diversity that can help with the evolution of cooperative strategies. We have recently further shown that strategy representation also plays an important factor in introducing behavioral diversity in the coevolutionary learning system [Chong and Yao (2005)]. We considered the n-choice IPD game, which was obtained based on the following linear interpolation: pA = 2.5 − 0.5cA + 2cB ,
− 1 ≤ cA , cB ≤ 1,
where pA is the payoff to player A, given that cA and cB are the cooperation levels of the choices that players A and B make, respectively. Fogel [Harrald and Fogel (1996)] also considered a similar interpolation process. However, we considered multiple, discrete levels of cooperation. For example, we used the four -choice IPD game, where the four cooperation levels are represented as +1 (full cooperation), +1/3, −1/3, and −1 (full defection). These choices can be used with the linear interpolation equation shown above to obtain the payoff. Figure 3.3 illustrates the payoff matrix of a four -choice IPD game that was used [Chong and Yao (2005)]. Note that in generating the payoff matrix for a n-choice IPD game, the following conditions must be satisfied [Chong and Yao (2005)]:
Learning IPD Strategies Through Co-evolution
73
PLAYER B 1
1
+1
+3
í3
+1
4
23
2
13
1
0
1 3
4
1 3
3
2 3
1 3
1
í1
PLAYER
+
A
í3
1
43
2
33
1
2
2 3
í1
5
33
2
23
1
1
Fig. 3.3. The payoff matrix for the two-player f our-choice IPD used in [Chong and Yao (2005)]. Each element of the matrix gives the payoff for Player A.
(1) For cA < c0A and constant cB : pA (cA , cB ) > pA (c0A , cB ), (2) For cA ≤ c0A and cB < c0B : pA (cA , cB ) < pA (c0A , c0B ), and (3) For cA < c0A and cB < c0B : pA (c0A , c0B ) > 21 (pA (cA , c0B ) + pA (c0A , cB )). These conditions are analogous to those for the classical IPD’s. The first condition ensures that defection always pays more. The second condition ensures that mutual cooperation has a higher payoff than mutual defection. The third condition ensures that alternating between cooperation and defection does not pay in comparison to just playing cooperation. We investigated two strategy representation: neural networks and direct look-up table. We considered these two strategy representations because they allow the investigation on the impact of strategy representation on the introduction and maintenance of variations of behavioral responses in the population of IPD strategies. On the one hand, the neural network indirectly represents the input-output response mappings of IPD strategies, with possibilities of many-to-one mappings between representations and actual behavioral responses [Fogel (1994b); Atmar (1994)]. On the other hand, the direct look-up table directly represents the input-output response mappings of IPD strategies. We hypothesized that a more direct representation of IPD strategies will allow more behavioral variations to be introduced and maintained in the population through co-evolution. For the neural network representation, we used a fixed-architecture feedforward multilayer perceptron (MLP) [Chong and Yao (2005)]. Specifically, the neural network consists of an input layer, a single hidden layer of ten nodes, and an output node. The network is fully connected and strictly layered (i.e., no short-cut connection from the input layer to the output node. The transfer (activation) function used for all nodes is the hyperbolic
74
S. Y. Chong et al.
tangent function, tanh(x). The input layer consists of the following four input nodes: (1) The neural network’s previous choice, i.e., level of cooperation, in [−1, +1]. (2) The opponent’s previous level of cooperation. (3) An input of +1 if the opponent played a lower cooperation level compared to the neural network, and 0 otherwise. (4) An input of +1 if the neural network played a lower cooperation level compared to the opponent, and 0 otherwise. The input layer is a function of two variables (e.g., neural network’s previous choice and the opponent’s previous choice) since the last two inputs are derived from the first two inputs. These additional inputs are to facilitate learning the recognition of being exploited and exploiting. Given the inputs, the neural network’s output determines the choice for its next move. The output is a real value between +1 and −1 that is discretized to either +1, +1/3, −1/3 or −1, depending on which discrete value the neural network output is closest to. We considered self-adaptive mutation for variation operators for the real-valued representation of neural networks that we used [Chong and Yao (2005)]. This approach associates a neural network with a self-adaptive parameter vector [σi (j)] that controls the mutation step size of the respective weights and biases of the neural network [wi (j)]. Offspring neural networks ([wi0 (j)] and [σi0 (j)]) are generated from parent neural networks ([wi (j)] and [σi (j)]) through mutations. Two different mutations based on Gaussian and Cauchy distributions were used in order to further investigate the impact of indirect strategy representation on variation operators that could increase genetic diversity but not necessarily lead to increase in behavioral diversity. For the self-adaptive Gaussian mutation, offspring neural networks are generated according to the following equations: σi0 (j) = σi (j) ∗ exp(τ ∗ Nj (0, 1)); i = 1 . . . 15, j = 1, . . . , Nw , wi0 (j) = wi (j) + σi0 (j) ∗ Nj (0, 1); i = 1 . . . 15, j = 1, . . . , Nw , where Nw = 63, τ = (2(Nw )0.5 )−0.5 = 0.251, and Nj (0, 1) is a Gaussian random variable (zero mean and standard deviation of one) resampled for every j. Nw is the total number of weights, biases, and the pre-game inputs required for an IPD strategy based on memory length of one.
Learning IPD Strategies Through Co-evolution
75
For the self-adaptive Cauchy mutation that is known to provide bigger changes to the neural network weights (i.e., provide more genetic diversity) [Yao et al. (1999)], the following equations are used: σi0 (j) = σi (j) ∗ exp(τ ∗ Nj (0, 1)); i = 1 . . . 15; j = 1, . . . , Nw , wi0 (j) = wi (j) + σi0 (j) ∗ Cj (0, 1); i = 1 . . . 15; j = 1, . . . , Nw , where Cj (0, 1) is a Cauchy random variable (centered at zero and with a scale parameter of 1) resampled for every j. All other variables remain the same as those in the self-adaptive Gaussian mutation. For the direct look-up table representation, the details can be illustrated by figure 3.4 [Chong and Yao (2005)], which shows the behavioral response of a four -choice IPD strategy. mij specifies the choice to be made, given the inputs i (player’s own previous choice) and j (opponent’s previous choice). Rather than using pre-game inputs (two for memory length one strategies), the first move is specified independently. Each of the table elements can take any of the possible four choices (+1, +1/3, −1/3, −1). Opponent’s Previous Move
Player’s Previous Move
1
1
+1
+3
í3
í1
+1
m11
m12
m13
m14
+
1 3
m21
m22
m23
m24
í
1 3
m31
m32
m33
m34
í1
m41
m42
m43
m44
Fig. 3.4. The look-up table representation for the two-player IPD with four choices and memory length one [Chong and Yao (2005)].
A simple mutation operator was used to generate offspring. Mutation replaces the original element, mij , by one of the other three possible choices with an equal probability. For example, if mutation occurs at m13 = +1/3, then the mutated element m013 can take either +1, −1/3, or −1 with an equal probability. Each table element has a fixed probability, pm , of being replaced by one of the remaining three choices. The value pm is not optimized. Crossover is not used in any of the experiments. With a direct representation of IPD strategy behaviors, a simple mutation is more than sufficient to provide behavioral diversity in the population.
76
S. Y. Chong et al.
The following co-evolutionary procedure was used [Chong and Yao (2005)]: (1) Generation step, t = 0: Initialize N/2 parent strategies, Pi , i = 1, 2, ..., N/2, randomly. (2) Generate N/2 offspring, Oi , i = 1, 2, ..., N/2, from N/2 parents using a variation. (3) All pairs of strategies compete, including the pair where a strategy plays itself (i.e., round-robin tournament). For N strategies in a population, every strategy competes a total of N games. (4) Select the best N/2 strategies based on total payoffs of all games played. Increment generation step, t = t + 1. (5) Step 2 to 4 are repeated until termination criterion (i.e., a fixed number of generation) is met. In particular, we used N = 30, and repeated the co-evolutionary process for 600 generations (which is sufficiently long to observe an evolutionary outcome, e.g., persistent cooperation). A fixed game length of 150 iterations is used for all games. Experiments are repeated for 30 independent runs. Note that additional steps were taken to ensure that the initial population has sufficient behavioral diversity in addition to genotypic diversity [Darwen and Yao (2000)] to avoid early convergence of results. All details are available in [Chong and Yao (2005)]. The procedure involves setting particular parameters for specific strategy representation and resampling for new strategies to make sure that the frequency at which each of the four choices (+1, +1/3, −1/3, −1) is played is approximately similar so that there is no bias to play a particular choice early in the evolution. Results showed that there were fewer number of runs where the population evolved to play mutual cooperation in experiments that used neural network representations [Chong and Yao (2005)]. For example, some runs had intermediate outcomes while a few had defection outcomes (Fig. 3.5). This is quite different from the case for classical IPD games [Axelrod (1987); Darwen and Yao (1995)] where each run converged to mutual cooperation quite consistently and quickly. Increasing genetic diversity (e.g., using selfadaptive Cauchy mutation) do not necessarily lead to more behavioral diversity in the population since some runs still evolved to intermediate or defection outcomes (Fig. 3.6). The results further illustrates that more choices have made cooperation more difficult to evolve. However, when direct look-up table representation was used, results
Learning IPD Strategies Through Co-evolution
77
4
Average Payoff
3.5
3
2.5
2
1.5
1 0
100
200
300
400
500
600
Generation
Fig. 3.5. Five sample runs of a co-evolutionary learning system that used neural network representation with a self-adaptive Gaussian mutation in the four -choice IPD [Chong and Yao (2005)].
4
Average Payoff
3.5
3
2.5
2
1.5
1 0
100
200
300
400
500
600
Generation
Fig. 3.6. Five sample runs of a co-evolutionary learning system that used neural network representation with a self-adaptive Cauchy mutation in the four -choice IPD [Chong and Yao (2005)].
showed that the evolution to cooperation was not difficult [Chong and Yao (2005)]. For example, results showed that even when a simple mutation with a low probability of mutation (e.g., pm = 0.05) was used, no run evolved to mutual defection even though intermediate outcomes were obtained (Fig. 3.7). However, increasing the probability of mutation resulted with all populations in all runs evolving to mutual cooperation play. The results showed that the choice of strategy representation can have an impact on the evolution of cooperation if it allows for greater behavioral diversity in the population. 3.3.2. IPD with Noise A natural extension to the classical IPD is to consider the impact of noisy interactions on the evolution of certain behaviors. Axelrod noted two types
S. Y. Chong et al.
78
4
Average Payoff
3.5
3
2.5
2
1.5
1 0
100
200
300
400
500
600
Generation
Fig. 3.7. Five sample runs of a co-evolutionary learning system that used direct look-up table representation with a simple mutation at pm = 0.05 in the four -choice IPD [Chong and Yao (2005)].
of noise, i.e., misimplementation and misperception, that can affect a strategy’s response to the opponent’s choice of play [Axelrod and Dion (1988)]. With misimplementation, the strategy knows a mistaken play but the opponent does not know. With misperception, one or both interacting strategies may not know that a different choice was made. The main motivation for this extension is to study the impact of noise on the learning of certain behaviors through co-evolution when interactions can be noisy. In particular, one issue that can be considered is whether cooperative strategies based on reciprocity (such as tit for tat) can still perform well when noise, which affects strategy behavioral response based on previous moves, is present. Julstrom [Julstrom (1997)] investigated the effects of noise in the twochoice IPD through a co-evolutionary learning system. In particular, noise was modelled as mistakes. That is, there is a probability that the choice played by a strategy is changed to the other choice (e.g., defection is played instead of the original cooperation, and vice versa). Results from the experiments showed that noise (starting around 2%) can reduce the level of cooperation in the population. Recently, we further extended the IPD game with more choices by introducing noise and used a co-evolutionary learning system as a model for investigations [Chong and Yao (2005)], which we have detailed in the earlier subsection. We also modelled noise as mistakes that a player makes. For the four -choice IPD game, there is a certain probability of occurrence, pn , and is fixed throughout a game where a strategy intends to play a particular choice but ends up with a different choice instead. For example, with pn = 0.05, there will be a 0.05 probability that if 1/3 is intended to be played, one of the other three possible cooperation levels, i.e., +1, −1/3,
Learning IPD Strategies Through Co-evolution
79
and −1, will be chosen uniformly at random. Results from experiments again showed the importance of behavioral diversity for the evolution of cooperation for noisy IPD games with more choices. For noise introduced at very low probabilities (less than 1.5% or pn = 0.0015), evolution to cooperation is more likely than the case when noise was not introduced. Strategies were observed to be more forgiving, confirming the predictions of other studies noted in [Axelrod and Dion (1988); Wu and Axelrod (1995)]. However, when noise was introduced at high probabilities (starting around 5% or pn = 0.05), evolution to cooperation was more difficult. The population was more likely to evolve to defection. Despite this, if the co-evolutionary learning system has sufficient behavioral diversity (e.g., using direct look-up table representation that allows for behavioral diversity to be introduced and maintained more easily and effectively), evolution of cooperation is not greatly affected [Chong and Yao (2005)]. Evolved strategies still played high levels of cooperation even when there are more choices to play and that the interactions can be noisy, both which can contribute to more difficulty of evolving cooperative behaviors. For example, table 3.1 compares different co-evolutionary learning system with different levels of behavioral diversity, e.g., C-CEP (neural network and self-adaptive Gaussian mutation), C-FEP (neural network and selfadaptive Cauchy mutation), C-PM05 (direct look-up table and mutation at pm = 0.05) for different noise levels (%) [Chong and Yao (2005)]. Results show the number of runs for each experiment that evolved to mutual defection, e.g., average payoff less than 1.5. The table showed that no runs evolved to mutual defection when direct look-up table representation was used in the co-evolutionary learning system [Chong and Yao (2005)]. Table 3.1. Comparison of results for three different co-evolutionary learning systems. Noise (%) C-CEP C-FEP C-PM05 0 4 1 0 5 4 9 0 10 7 11 0 15 8 17 0 20 18 26 0
It should be noted that although both mutation and noise can be considered as sources of behavioral variations in models that encourage coopera-
80
S. Y. Chong et al.
tion [Mcnamara et al. (2004)], they produce behavioral diversity differently. Mutation introduces strategies with different behaviors into the population. Noise allows other parts of a strategy’s behavior that are not played otherwise in a noiseless IPD game to be accessed. Our results [Chong and Yao (2005)] showed that noise does not necessarily promote behavioral diversity in the population that lead to a stable evolution to cooperation, although noise at low levels does help. With higher levels of noise, closer inspection of evolved strategies showed the population to overspecialize to a specific behavior that is vulnerable to invasion, leading to cyclic dynamics in the evolutionary process between cooperation and defection. In particular, noise and mutation have different impacts on the evolutionary process [Chong and Yao (2005)]. For example, increasingly higher levels of noise lead to mutual defection outcomes. Given a very noisy environment, strategies overspecialized to play defection only. This was not observed in the noiseless case of the IPD with increasingly more mutations. For example, increasingly higher mutation rates in the co-evolutionary learning system that used direct look-up table representation did not lead to mutual defection outcomes. Strategies were not observed to overspecialized to play defection, or any specific play. 3.3.3. N-Player IPD Real-world interactions may involve more than two players. One famous example is the “tragedy of the commons” [Hardin (1968)], which illustrates the problem of self-interested actions of players for a particular public goods for initial rewards leading to a situation where everyone loses out in the end. For the case of the IPD, N-player interactions can be extended to the original formulation of two-player game [Axelrod and Dion (1988)]. This allows for the study of whether cooperative behaviors are possible when interactions involve more than two players since strategies that are effective for the two-player case may not be effective (or worse, fail) in large group interactions [Glance and Huberman (1994)]. One of us formulated an N-player IPD or NIPD game for investigations using the co-evolutionary learning approach [Yao and Darwen (1994)] (other studies include [Bankes (1994); Lindgren and Johansson (2001)]). The NIPD game is defined by the following three properties [Colman (1982)] (page 159): • Each player faces two choices between cooperation and defection.
Learning IPD Strategies Through Co-evolution
81
• Defection is dominant for each player, i.e., each player is better off defecting than cooperating regardless of how many other players that cooperate. • The dominant defection strategies intersect in a deficit equilibrium. In particular, the outcome if all players choose their non-dominant cooperation strategies is preferable from every player’s point of view to the one in which everyone chooses defection, but no one is motivated to deviate unilaterally from defection. The payoff matrix (Fig. 3.8) for the NIPD game can then be constructed based on the following conditions that must be satisfied [Yao and Darwen (1994)]: • Di > Ci for 0 ≤ i ≤ n − 1. • Di+1 > Di and Ci+1 > Ci for 0 ≤ i ≤ n − 1. • Ci > (Di + Ci−1 )/2 for 0 ≤ i ≤ n − 1. A large number values satisfy these conditions. For the study in [Yao and Darwen (1994)], the values are chosen such that if nc is the number of cooperators in the NIPD game, then the payoff for cooperation is 2n c − 2 and the payoff for defection is 2nc + 1 (Fig. 3.9). For this payoff matrix, the average per-move payoff a can be calculated as follows if Nc cooperative moves are made out of N moves: a=1+
Nc (2n − 3), N
which will allow the measurement of how common cooperation was by examining the average per-round payoff.
Player A
Number of cooperators among the remaining n-1 players
0
1
2
n-1
C
C0
C1
C2
…
Cn-1
D
D0
D1
D2
…
Dn-1
Fig. 3.8. The payoff matrix for the NIPD game. The value in the table gives the payoff to the player based on its choice of play [Yao and Darwen (1994)].
S. Y. Chong et al.
82
Player A
Number of cooperators among the remaining n-1 players
Fig. 3.9.
0
1
2
n-1
C
0
2
4
…
2(n-1)
D
1
3
5
…
2(n-1)+1
An example of the payoff matrix for the NIPD game [Yao and Darwen (1994)].
NIPD game interactions were in the form of a large number of random selection of groups of N players with replacement (e.g., 1000 NIPD games for a population of 100 strategies). Results from the experiments in [Yao and Darwen (1994)] showed the group size (i.e., the value of N in the NIPD game) has a negative impact on the evolution of cooperation. As N increases, there are fewer number of runs where the population evolved to play cooperation. For example, in the case of memory two strategies, only one out of 20 runs had defection outcomes for 3IPD. However, the number of runs with defection outcomes increased to nine for 6IPD. Increasing N to 16 (i.e., 16IPD) resulted with all runs evolved to defection outcomes [Yao and Darwen (1994)]. 3.3.4. Other Extensions There are many other extensions to the classical IPD game, or even further extensions to already extended IPD games (such as the NIPD) that can be studied through a co-evolutionary learning approach. For example, we examined the impact of localized interactions of the NIPD games in [Seo et al. (1999, 2000)]. The earlier study for the NIPD [Yao and Darwen (1994)] showed that the evolution of cooperation is more difficult to achieve through a co-evolutionary learning process as N increases. However, in some real-world interactions, it is unlikely that a player interacts with everybody (or that it has equal probability of interacting with anyone in the population). Instead, a player might interact with other specific players (e.g., neighbours, relatives, or at the workplace). Such localized interactions may involve spatial models [Nowak and May (1992); Ishibuchi and Namikawa (2005)]. In particular, localized interactions can have a positive impact on the evolution of cooperation in the NIPD game. That is, population
Learning IPD Strategies Through Co-evolution
83
structured in a spatial model is more likely to evolve cooperation [Seo et al. (1999); Lindgren and Johansson (2001)]. Another extension that can be considered is to incorporate indirect interactions to the IPD game that originally only considers direct interactions between strategies. Most of the previous studies have focused on modelling direct interactions (e.g., cooperative behaviors through direct reciprocity that involves repeated encounters, i.e., IPD games [Axelrod (1984)]) or indirect interactions (e.g., cooperative behaviors through mechanisms of indirect reciprocity such as reputation where an individual receives cooperation from third parties due to the individual’s cooperative behaviors to others in the case of indirect reciprocity [Nowak and Sigmund (1998b)]). However, it has been suggested that complex real-world interactions involve both direct and indirect interations (although for simplicity for modelling and analysis, only one of the interactions is considered at one time) [Nowak and Sigmund (1998a)]. For this aspect, we have investigated a model with both direct and indirect interactions [Yao and Darwen (1999)]. In particular, each strategy is tagged with a reputation score, which is calculated based on payoffs received from a small random sample of pre-games. A co-evolutionary approach to show that with the addition of reputation, cooperative outcomes are possible and more likely even for the case of the IPD with more choices and shorter game durations [Yao and Darwen (1999)]. In addition to that, another extension will be to consider the adaptation of payoff matrices. We recently conducted a preliminary study on evolving strategy payoff matrices, and how such an adaptation process can affect the learning of strategy behaviors [Chong and Yao (2006)]. The motivation for the study is to relax the assumption of having fixed, symmetric payoff matrix for all evolving strategies. This assumption may not be realistic, considering that not all players are similar in real-world interactions. We focus specifically on an adaptation process of payoff matrix based on past behavioral interactions. In particular, a simple update rule that provides a reinforcement feedback process between strategy behaviors and payoff matrices during the co-evolutionary process is used. Results from experiments [Chong and Yao (2006)] showed that the evolutionary outcome is dependent on the adaptation process of both behaviors (i.e., strategy behavioral responses) and utility expectations that determine how behaviors are rewarded (i.e., strategy payoff matrices). Defection outcomes are more likely to be obtained if IPD-like update rules that favor the exploitation of opponents are used. However, cooperative outcomes can be easily obtained when mutualism-like update rules that favor mutual cooperation are used.
84
S. Y. Chong et al.
3.4. Conclusion and Future Directions
The greatest advantage and the most important feature of co-evolutionary learning is that of the process of adaptation on representation that is dependent on the interactions between members of the population. In this aspect, the co-evolutionary learning approach is well-suited to solving the problem of IPD games in two contexts. First, co-evolutionary learning can be used as a search algorithm for effective strategies without requiring human knowledge. All that is required is the rules of the game. Second, the adaptation process of strategy behaviors based on interactions in coevolution provides a natural way to investigate conditions that lead to the evolution of certain behaviors. In both of these contexts, the advantage of co-evolutionary learning to other approaches is that strategy behaviors are not fixed or predefined. Instead, co-evolutionary learning provides a means to realize strategy behavioral responses that are not necessarily bounded by expert human knowledge, thus providing new insight to the problem. Since the first study of co-evolutionary learning on the classical IPD by Axelrod [Axelrod (1987)], there had been a wide-range of studies that further extended the classical IPD game with additional features such as, but not limited to, continuous or multiple levels of cooperation, noisy interactions, N-player interactions, spatial interactions, and indirect interactions. The motivation in all of these studies is to bridge the gap between the abstract IPD interactions with the complex real-world interactions. As such, by understanding the specific conditions that lead to the evolution of specific IPD strategy behaviors, these studies have further helped to provide a more in-depth view on complex real-world interactions such as those found in the human society. There are still much more that can be explored using the co-evolutionary learning approach. One direction will be to further extend the more complex IPD games and investigate the impact of the additional extension. This is important because the extensions might interact with one another in some unknown and nonlinear fashion. Understanding these interactions will help to further unravel complex human interactions. Another direction will be to investigate a more rigorous approach to determine the robustness of evolved strategy behaviors. In this particular aspect, the notion of generalization might provide a more natural approach for co-evolutionary learning in addition to classical evolutionary game theory approach of the evolutionarily stable strategies.
Learning IPD Strategies Through Co-evolution
85
References Atmar, W. (1994). Notes on the simulation of evolution, IEEE Transactions on Neural Networks 5, 1, pp. 130–147. Axelrod, R. (1980a). Effective choice in the prisoner’s dilemma, The Journal of Conflict Resolution 24, 1, pp. 3–25. Axelrod, R. (1980b). More effective choice in the prisoner’s dilemma, The Journal of Conflict Resolution 24, 3, pp. 379–403. Axelrod, R. (1984). The Evolution of Cooperation (Basic Books, New York). Axelrod, R. (1987). The evolution of strategies in the iterated prisoner’s dilemma, in L. D. Davis (ed.), Genetic Algorithms and Simulated Annealing, chap. 3 (Morgan Kaufmann, New York), pp. 32–41. Axelrod, R. and Dion, D. (1988). The further evolution of cooperation, Science 242, 4884, pp. 1385–1390. Axelrod, R. and Hamilton, W. D. (1981). The evolution of cooperation, Science 211, pp. 1390–1396. B¨ ack, T. (1996). Evolutionary Algorithms in Theory and Practice (Oxford University Press, New York). B¨ ack, T., Hammel, U. and Schwefel, H. P. (1997). Evolutionary computation: Comments on the history and current state, IEEE Transactions on Evolutionary Computation 1, 1, pp. 3–17. Bankes, S. (1994). Exploring the foundations of artificial societies: Experiments in evolving solutions to iterated n-player prisoner’s dilemma, in R. Brookes and P. Maes (eds.), Artificial Life IV (Addison-Wesley), pp. 337–342. Chellapilla, K. and Fogel, D. B. (1999). Evolution, neural networks, games, and intelligence, Proc. IEEE 87, 9, pp. 1471–1496. Chong, S. Y. and Yao, X. (2005). Behavioral diversity, choices, and noise in the iterated prisoner’s dilemma, IEEE Transactions on Evolutionary Computation 9, 6, pp. 540–551. Chong, S. Y. and Yao, X. (2006). Self-adaptive payoff matrices in repeated interactions, in 2006 IEEE Symposium on Computational Intelligence and Games (CIG’06) (IEEE Press, Piscataway, NJ), pp. 103–110. Colman, A. M. (1982). Game Theory and Experimental Games (Pergamon Press, Oxford). Darwen, P. and Yao, X. (1995). On evolving robust strategies for iterated prisoner’s dilemma, in Progress in Evolutionary Computation, Lecture Notes in Artificial Intelligence, Vol. 956, pp. 276–292. Darwen, P. and Yao, X. (2000). Does extra genetic diversity maintain escalation in a co-evolutionary arms race, International Journal of Knowledge-Based Intelligent Engineering Systems 4, 3, pp. 191–200. Darwen, P. and Yao, X. (2001). Why more choices cause less cooperation in iterated prisoner’s dilemma, in Proc. 2001 Congress on Evolutionary Computation (CEC’01) (IEEE Press, Piscataway, NJ), pp. 987–994. Darwen, P. and Yao, X. (2002). Co-evolution in iterated prisoner’s dilemma with intermediate levels of cooperation: Application to missile defense, International Journal of Computational Intelligence and Applications 2, 1, pp.
86
S. Y. Chong et al.
83–107. Darwen, P. J. (1996). Co-evolutionary Learning by Automatic Modularization with Speciation, Ph.D. thesis, University of New South Wales, Sydney, Australia. Darwen, P. J. and Yao, X. (1997). Speciation as automatic categorical modularization, IEEE Transactions on Evolutionary Computation 1, 2, pp. 101–108. Fogel, D. B. (1991). The evolution of intelligent decision making in gaming, Cybernetics and Systems: An International Journal 22, pp. 223–236. Fogel, D. B. (1993). Evolving behaviors in the iterated prisoner’s dilemma, Evolutionary Computation 1, 1, pp. 77–97. Fogel, D. B. (1994a). An introduction to simulated evolutionary optimization, IEEE Transactions on Neural Networks 5, 1, pp. 3–14. Fogel, D. B. (1994b). An introduction to simulated evolutionary optimization, IEEE Transactions on Neural Networks 5, 1, pp. 3–14. Fogel, D. B. (1995). Evolutionary Computation: Toward a New Philosophy of Machine Intelligence (IEEE Press, Piscataway, NJ). Fogel, D. B. (1996). On the relationship between the duration of an encouter and the evolution of cooperation in the iterated prisoner’s dilemma, Evolutionary Computation 3, 3, pp. 349–363. Franken, N. and Engelbrecht, A. P. (2005). Particle swarm optimization approaches to coevolve strategies for the iterated prisoner’s dilemma, IEEE Transactions on Evolutionary Computation 9, 6, pp. 562–579. Glance, N. S. and Huberman, B. A. (1994). The dynamics of social dilemmas, Scientific American , pp. 58–63. Hardin, G. (1968). The tragedy of the commons, Science 162, pp. 1243–1248. Harrald, P. G. and Fogel, D. B. (1996). Evolving continuous behaviors in the iterated prisoner’s dilemma, BioSystems: Special Issue on the Prisoner’s Dilemma 37, pp. 135–145. Ishibuchi, H. and Namikawa, N. (2005). Evolution of iterated prisoner’s dilemma game strategies in structured demes under random pairing in game playing, IEEE Transactions on Evolutionary Computation 9, 6, pp. 552–561. Julstrom, B. A. (1997). Effects of contest length and noise on reciprocal altruism, cooperation, and payoffs in the iterated prisoner’s dilemma, in Proc. 7th International Conf. on Genetic Algorithms (ICGA’97) (Morgan Kauffman, San Francisco, CA), pp. 386–392. Lindgren, K. (1991). Evolutionary phenomena in simple dynamics, in C. G. Langton, C. Taylor, J. D. Farmer and S. Rasmussen (eds.), Artificial Life II (Addison-Wesley), pp. 295–312. Lindgren, K. and Johansson, J. (2001). Coevolution of strategies in n-person prisoner’s dilemma, in J. Crutchfield and P. Schuster (eds.), Evolutionary Dynamics - Exploring the Interplay of Selection, Neutrality, Accident, and Function (Addison-Wesley). Mcnamara, J. M., Barta, Z. and Houston, A. I. (2004). Variation in behaviour promotes cooperation in the prisoner’s dilemma, Nature 428, pp. 745–748. Miller, J. (1989). The coevolution of automata in the iterated prisoner’s dilemma, Tech. Rep. 89-003, Santa Fe Institute Report. Nowak, M. A. and May, R. M. (1992). Evolutionary games and spatial chaos,
Learning IPD Strategies Through Co-evolution
87
Nature 355, pp. 250–253. Nowak, M. A. and Sigmund, K. (1998a). The dynamics of indirect reciprocity, Journal of Theoretical Biology 194, pp. 561–574. Nowak, M. A. and Sigmund, K. (1998b). Evolution of indirect reciprocity by image scoring, Nature 393, pp. 573–577. Seo, Y. G., Cho, S. B. and Yao, X. (1999). Emergence of cooperative coalition in nipd game with localization of interaction and learning, in Proc. IEEE 1999 Congress on Evolutionary Computation (CEC’99) (IEEE Press, Piscataway, NJ), pp. 877–884. Seo, Y. G., Cho, S. B. and Yao, X. (2000). Exploiting coalition in co-evolutionary learning, in Proc. IEEE 2000 Congress on Evolutionary Computation (CEC’00) (IEEE Press, Piscataway, NJ), pp. 1268–1275. Stanley, E. A., Ashlock, D. and Smucker, M. D. (1995). Prisoner’s dilemma with choice and refusal of partners: Evolutionary results, in Proc. Third European Conf. on Advances in Artificial Life, pp. 490–502. Wu, J. and Axelrod, R. (1995). How to cope with noise in the iterated prisoner’s dilemma, The Journal of Conflict Resolution 39, 1, pp. 183–189. Yao, X. (1994). Introduction, Informatica (Special Issue on Evolutionary Computation) 18, pp. 375–376. Yao, X. (1999). Evolving artificial neural networks, Proc. IEEE 87, 9, pp. 1423– 1447. Yao, X. and Darwen, P. (1999). How important is your reputation in a multiagent environment, in Proc. 1999 Conf. on Systems, Man, and Cybernetics (SMC’99) (IEEE Press, Piscataway, NJ), pp. 575–580. Yao, X. and Darwen, P. J. (1994). An experimental study of n-person iterated prisoner’s dilemma games, Informatica 18, pp. 435–450. Yao, X., Liu, Y. and Darwen, P. J. (1996). How to make best use of evolutionary learning, in R. Stocker, H. Jelinck, B. Burnota and T. Bossomaier (eds.), Complex Systems - From Local Interactions to Global Phenomena (IOS Press, Amsterdam), pp. 229–242. Yao, X., Liu, Y. and Lin, G. (1999). Evolutionary programming made faster, IEEE Transactions on Evolutionary Computation 3, 2, pp. 82–102.
This page intentionally left blank
Chapter 4 How to Design a Strategy to Win an IPD Tournament
Jiawei Li University of Nottingham, Harbin Institute of Technology
4.1. Introduction Imagine that a player in an IPD tournament knows the strategy of each of his opponents; he will defect against opponents such as ALLC or ALLD and cooperate with opponents such as GRIM or TFT in order to maximize his payoff. This means that he can interact with each opponent optimally and receive higher payoffs. Although this information a priori is not possible, one can identify a strategy during the game. For example, if a strategy cooperated with its opponent in the previous 10 rounds while its opponent defected, it seems sensible to deduce that it will always cooperate. In fact, each strategy will gradually reveal itself through the IPD game; moreover, it is not after the game that we can identify the strategy but possibly after a few rounds. With an efficient identification mechanism, it is possible for a strategy to interact with most of its opponent optimally. However, two main problems must be solved in designing an efficient identification mechanism. Firstly, it is impossible, in theory, for a strategy to identify an opponent within a finite number of rounds because the number of possible strategies is huge. Only can the types of strategies belonging to a preconcerted finite set be identified, which may be just a small proportion of all those possible because identification will be of no use if it takes too long. Secondly, there exists a risk of exploring an opponent putting the player into a much worse position. In other words, such an action may have negative effect on future rewards. For example, in order to distinguish between ALLC and GRIM, a strategy has to defect at least once and loses the chance to cooperate with GRIM in the future. In this chapter we will discuss how to resolve these problems, how to 89
90
J. Li
design an identification mechanism for IPD games, and how the strategy of Adaptive Pavlov was designed, which was ranked first in Competition 4 of the 2005 IPD tournament. 4.2. Analysis of strategies involved in IPD games Every strategy may have its disadvantages as well as its advantages. A strategy may receive high payoffs when its opponent belongs to some set of strategies, and receive lesser payoffs when an opponent belongs to another set of strategies. However, some strategies always do better than others in IPD tournaments. The strategies involved in IPDs can be classified according to whether or not they respond to their opponents. One set of strategies is fixed and plays a predetermined action no matter what their opponent does. ALLD, ALLC and RAND are typical. Other strategies are more complicated and their actions depend on their opponent’s behavior. TFT, for example, starts with COOPERATE and then repeats his opponent’s last move. The second set is obviously superior to the former since the strategies like TFT, TFTT and GRIM have always performed better than ’fixed’ strategies in past IPD tournaments. Then, the question is what the optimal response to every opponent is. Is TFT’s imitation of opponent’s last move the best response? Although TFT has been shown to be superior to many other strategies, it is not good enough to win every IPD tournament. Let’s consider a simulation of IPD tournament with 9 players. These players are ALLC, ALLD, RAND, GRIM, TFT, STFT, TFTT, TTFT, and Pavlov. The descriptions of the strategies of these players are as shown in Table 4.1. These strategies are simple and representational, and have all appeared in past IPD tournaments. The rule of our simulation is that each strategy will play a 200-round IPD game with every strategy (including itself). The payoffs in a round are as shown in Fig. 4.1. The total payoff received by any given strategy is the summation of the payoffs throughout the tournament. The results of the tournaments vary because there are random choices in the strategies of Pavlov and RAND. In order to decrease the variability of the result, the tournament is repeated several times and the average score for each strategy is calculated. Simulation results show that TFT, TFTT and GRIM acquire higher scores than the others and their average scores across several tournaments are quite close. TFTT, however, wins
How to Design a Strategy to Win an IPD Tournament
Table 4.1.
Description of the players of the IPD simulation.
Players
Descriptions
ALLC ALLD RAND GRIM
This strategy always plays COOPERATE. This strategy always plays DEFECT. It plays DEFECT or COOPERATE with 1/2 probability. Starts with COOPERATE, but after one defection plays always DEFECT. Starts with COOPERATE, and then repeats opponent’s moves. Like TFT but it plays DEFECT after two consecutive defections. Like TFT but in first move it plays DEFECT. Like TFT but it plays two DEFECT after opponent’s defection. Result of each move is divided into two groups: SUCCESS (payoff 5 or 3) and DEFEAT (payoff 1 or 0). If the last result belongs to SUCCESS group it plays the same move, otherwise it plays the other move.
TFT TFTT STFT TTFT Pavlov
91
Player2’schoice
Player1’schoice
CO O PERATE D EFECT
CO O PERATE (3,3) (5,0)
D EFECT (0,5) (1,1)
Fig. 4.1. Payoffs table of the IPD tournament. The numbers in brackets denote the payoffs two players receive in a round of a game.
more times than the others in a single tournament. For example, TFTT wins 11 tournaments from a total of 20, while TFT wins 4 and GRIM wins 5. In addition, if Pavlov and RAND are removed TFTT will always win. One of the limitations of TFT is that it will inevitably run into the circle of defecting-defected (which means that TFT plays COOPERATE while its opponent defects; and then TFT plays DEFECT while its opponent cooperates) when its opponent happens to be STFT. However, cooperation will be achieved resulting in higher payoffs if TFT cooperates once after its opponent defects. TFTT is superior to TFT in this regard. And it is this reason why TFTT wins more tournaments than TFT in the above IPD simulation. It is easy to verify that TFT will not get lower scores than TFTT if STFT is removed from the simulation. Thus, we can improve the strategy of TFT in this way: when TFT enters a circle of defecting-defected (for example a sequence of 3 pairs of defecting-defected) it will choose COOPERATE in two continuous rounds. This modified TFT (MTFT) will achieve higher payoffs than TFT in the case that their opponents are STFT. By substituting MTFT for TFT, IPD
92
J. Li
experiments show that MTFT gets the highest average score and wins more single tournaments than the others. MTFT has used an identification technique. It identified STFT by detecting the defecting-defected circles in the process of an IPD game. When the opponent was considered to be STFT, optimal action (cooperates in two sequential rounds) would be carried out in order to maximize future payoffs. In this way, it is natural to deduce that MTFT can be further improved so that it can identify more strategies and then interact with them optimally. In the following sections, an approach to identify each strategy in a finite set will be introduced. A strategy can interact with the opponents almost optimally by using this identification mechanism. 4.3. Estimation of possible strategies in an IPD tournament In this section, we seek to define a finite set of types of strategies to be identified. Since the number of possible strategies for IPD are infinite, it is impossible to identify each of them in a finite number of rounds. For example, suppose that a strategy cooperated with its opponent in 10 sequential rounds while its opponent defected continuously. Although it is very likely to be ALLC, there are always other possibilities. It may be GRIM but the trigger is 11 defections; it may be RAND that has just happened to play 10 sequential COOPERATEs; or it may be a combination of ALLC and TFT and it will behave as TFT type in the following rounds. However, since only ALLC belongs to the set of identification, those other possibilities will be eliminated. How to choose the set of identification depends on prior knowledge and subjective estimation. Some strategies like TFT are likely to appear; while others are designated as default strategies. There are numerous strategies one can design for an IPD tournament. However, most of them seldom appear because their chances of winning are very small. For example, there may be such a strategy that it cooperates in the first two rounds and defects in the following two rounds, and then it cooperates and defects alternately. Few players will apply such a strategy because it is unlikely to win any IPD tournament. It is obvious that the strategies that usually win appear frequently and the others appear infrequently. We define two classifications of IPD strategies: cooperating and defecting. Cooperating strategies, for example TFT and TFTT, wish to coop-
How to Design a Strategy to Win an IPD Tournament
93
erate with their opponents and never start defecting. Defecting strategies, for example ALLD and Pavlov beginning with DEFECT (PavlovD), wish to defect in order to maximize their payoffs and they always start defecting. The cooperating strategies differ in the way of their responses to the opponent’s defections. For example, TFTT is more forgiving than TFT as it retaliates only if its opponent has defected twice. GRIM is sterner than TFT as it never forgives a defection. These strategies can be classified according to their responses to the opponent’s defections. The rules are the same as the one described in the previous simulation as shown in Fig. 4.2.
Stern
Forgiving
ALLC
TFTT Fig. 4.2.
TFT
TTFT
GRIM
The cooperating strategies.
The defecting strategies differ in the way they insist on defecting. PavlovD is a representative strategy in this set. It starts with DEFECT. If the opponent is too forgiving to retaliate, it defects forever. Otherwise, it tries to cooperate with the opponent.a The defecting strategies can be classified as shown in Fig. 4.3. Defect more
Defect less
PavlovD
STFT Fig. 4.3.
ALLD
The defecting strategies.
Other simple strategies which lack a clear objective differ from the cooperating and defecting strategies and hardly ever get high scores in IPD tournaments. Most of the players of an IPD tournament will be cooperating strategies at the present time since cooperating strategies have been dominant in most of the tournaments. There will also be a small quantity of dea Although
PavlovD tries to cooperate with an opponent when the opponent retaliates upon its defection, it seldom succeeds. For example, even if PavlovD meets a forgiving strategy like TFTT they cannot keep cooperating in the game. In fact, if only PavlovD cooperates one more time cooperating can be achieved. We have examined a modified PavlovD (MPavlovD) strategy that starts with DEFECT and cooperates twice when the opponent retaliates. The results of simulation show that MPavlovD always gains more scores than PavlovD.
J. Li
94
fecting strategies. Based on the above idea, we have designed the Adaptive Pavlov strategy that applies a simple mechanism to distinguish cooperating strategies and several representative defecting strategies. 4.4. Interaction with a strategy optimally For any strategy there must be another strategy that optimally deals with it. Because the strategies of ALLC, ALLD and RAND are independent of the opponent’s behavior, ALLD is the optimal strategy. Because GRIM, TFT, STFT and TTFT retaliate as soon as their opponent defects, the optimal strategy for its opponent is to always cooperate but defect in the last round. TFTT is more charitable and forgives a single defection; therefore, its opponent can maximize the payoff by alternately choosing DEFECT and COOPERATE. If Pavlov starts with COOPERATE its opponent should always cooperate except in the last round; Otherwise, its opponent should start with DEFECT, then always cooperate except in the last round. Table 4.2 shows the optimal strategies to deal with each strategy shown in Table 4.1. Table 4.2. Strategies ALLC ALLD RAND GRIM TFT TFTT STFT TTFT Pavlov
Optimal strategies to interact with a known strategy. Optimal strategy of opponent
It always plays DEFECT. It always plays DEFECT. It always plays DEFECT. It always plays COOPERATE except DEFECT in the last move. It always plays COOPERATE except DEFECT in the last move. It starts with DEFECT, and then plays COOPERATE and DEFECT in turn. It always plays COOPERATE except DEFECT in the last move. It always plays COOPERATE except DEFECT in the last move. If Pavlov starts with DEFECT it starts with DEFECT, and then always plays COOPERATE except that it plays DEFECT in the last round; If Pavlov starts with COOPERATE it always plays COOPERATE except that it plays DEFECT in the last round.
Given an IPD tournament with n players, a player will win the tournament if it interacts with each of its opponent optimally. For example, a unique ALLD will win when the other n − 1 players in a IPD tournament are all ALLC. Hence, the winning strategy of an IPD tournament must be optimal in interacting with most of the others. Although the strategy of a player is unknown to his opponent before
How to Design a Strategy to Win an IPD Tournament
95
a game, the strategy gradually emerges as the game progresses. It is not difficult for a human player to identify the strategy of his opponent but it is more difficult for a computer program to possess the ability of identification. To make this feasible, there is a need for a method to distinguish each type of strategy from the others, and then a computer program can interact with different types of strategies with a relevant response. Under the assumption that every player belongs to a pre-defined finite set of strategies, an example is given to show how the method of identification is realized and how the winning strategy is designed . Consider an IPD tournament with 10 players. Besides the players shown in Table 4.1, let us add a new player MyStrategy (MS) which applies an identification mechanism to identify its opponent. The rules are the same as those described in the previous simulation. MS starts with DEFECT. If its opponent chooses DEFECT in the first round, MS chooses COOPERATE in round two, otherwise MS chooses DEFECT. MS always chooses COOPERATE in the third round. In this way, most of the strategies can be identified after just three rounds. For example, suppose that the choices of MS and its opponent in the first 3 rounds are as shown in Fig. 4.4. The strategy of the opponent can be confirmed to be RAND. Because the opponent starts with DEFECT it must be one of the strategies of ALLD, STFT, RAND and Pavlov. Since MS defects in the first round and the opponent cooperates in round two, it is impossible to be ALLD or STFT. Since MS and the opponent cooperate in the second round, the opponent should not defect in the third round if it were Pavlov. Therefore, the opponent must be RAND. The optimal strategy is ALLD in interacting with RAND, and MS will behave as ALLD in the following rounds of the game.
Round 1
Round 2
Round 3
M S’sm oves
D efect
Cooperate
Cooperate
O pponent’sm oves
D efect
Cooperate
D efect
Fig. 4.4.
A possible process of a game (shows that the opponent is RAND).
Some possible results of identification for the 9 strategies are listed in Table 4.3, where ’C’ denotes COOPERATE and ’D’ denotes DEFECT. Because the strategy RAND chooses its move randomly it may behave like
J. Li
96
any other strategy during a short period; therefore, there needs to be more rounds to distinguish RAND from other strategies. If there is a process different from that of as shown in Table 4.3, the strategy of the opponent must be RAND. Table 4.3. Players
Identification of the 9 strategies.
Possible moves of two players
Identification result
MyStrategy The opponent
D C C D C C
Pavlov (RAND)
MyStrategy The opponent
D C C D D D
ALLD (RAND)
MyStrategy The opponent
D C C D D D C
STFT (RAND)
MyStrategy The opponent
D D C C C C C
ALLC (RAND)
MyStrategy The opponent
D D C C C C D
TFTT (RAND)
MyStrategy The opponent
D D C C C D C
Pavlov (RAND)
MyStrategy The opponent
D D C C C C D D C
TFT (RAND)
MyStrategy The opponent
D D C C C C D D D C
TTFT (RAND)
MyStrategy The opponent
D D C C C C D D D D
GRIM (RAND)
In this way, a strategy can be identified after several rounds of game, and then the optimal strategy can be applied. Ten IPD tournaments with the above 10 players are carried out.b The simulation results are as shown in Fig. 4.5. It shows that MS gains the highest average payoffs when compared to the other strategies and achieves the highest score in each tournament. The reason for MS’s success is that b How
many rounds an IPD game commits is usually not fixed in order to avoid the players’ knowing when the end of the game is due. The simulation applies a fixed number of rounds in order to decrease complexity of computation. However, the strategy of MS does not make use of this to get extra payoff; that is to say, MS does not purposely choose DEFECT in the last round of a game.
Points in 10 tournaments
Average Rank
MS
6134
6213
6179
6127
6202
6175
6152
6172
6212
6187
6175.3
1
TFTT
5957
5996
5970
6003
5994
5959
5965
5969
5966
5976
5975.5
2
TFT
5961
5936
5919
5946
5959
5938
5940
5929
5954
5978
5946.0
3
Pavlov
5718
5691
5725
5775
5816
5763
5748
5763
5733
5745
5747.7
4
TTFT
5725
5723
5725
5717
5719
5725
5746
5732
5722
5716
5725.0
5
GRIM
5404
5394
5416
5410
5440
5468
5322
5400
5390
5384
5402.8
6
ALLC
5115
5091
5103
5127
5103
5103
5103
5082
5109
5091
5102.7
7
RAND
4339
4349
4254
4340
4216
4219
4258
4241
4228
4274
4271.8
8
STFT
4165
4187
4160
4169
4179
4144
4173
4158
4142
4158
4163.5
9
ALLD
3800
3792
3852
3792
3848
3856
3832
3864
3832
3832
3830.0
10
Fig. 4.5.
How to Design a Strategy to Win an IPD Tournament
Players
Simulation results of 10 IPD tournaments.
97
98
J. Li
it has almost optimally interacted with most of the strategies in this IPD tournament. Most IPD strategies, such as TFT or Pavlov, are memory-one strategies which can only respond to the opponent’s last move; however, the past process of the game contains more information. The identification mechanism of MS uses information about the opponent’s strategy, thus MS responds to not just the opponent’s past moves but the opponent’s strategy. By identifying different opponents, MS makes use of more information than the simple strategies. This is the reason MS is able to win IPD tournaments. Different identification approaches may lead to different results for MS. For example, all of the strategies GRIM, TFT and ALLC start with COOPERATE, and they will not defect if their opponents don’t. To identify each of these strategies, MS starts with DEFECT and loses the chance to cooperate with GRIM. On the other hand, if MS doesn’t firstly defect, it cannot distinguish the 3 strategies and cannot interact with ALLC optimally. The risk involved in exploring the opponent must be considered in order to choose an efficient or payoff-maximizing identification approach. 4.5. Escape from the trap of defection When a player begins to explore the opponent, there is a risk of the identifying process’s putting the player into a much worse position. Some strategies, especially those with trigger mechanism such as GRIM, will change their behaviors at the trigger point. For example, the strategy MS described in the above section defects at the beginning of IPD games in order to distinguish each of the cooperating strategies ALLC, TFT and GRIM; however, the chance to cooperate with GRIM is lost. In IPD games, the risk of identification is mainly the trap of defection, which means an identifying process leading the opponent to keep defecting with nothing that can be done to rescue the situation. It appears that a strategy will not run into the trap of defection if it never defects first. But this is not the case. Suppose a strategy keeps playing COOPERATE if its opponent defects, and defects forever once its opponent cooperates; then, any cooperating strategy will be defected against in interacting with it while most of defecting strategies will keep cooperating. If there is a equal possibility of this reverse-GRIM strategy appearing in a game to that of GRIM, to cooperate or to defect has equal risk to invoke future defection. This means that there always exists the risk of the defection trap whether or not an identification mechanism is applied.
How to Design a Strategy to Win an IPD Tournament
99
One may argue that the reverse-GRIM type of strategies will not appear as frequently as GRIMs in IPDs, so to cooperate is safer than to defect and the MS strategy is more likely to run into the defection trap than TFT. That is right. But it is not enough to testify that a defection trap is not inevitable for a strategy with an identification mechanism because many identification approaches can be applied. For example, a simple way to avoid retaliation from GRIM is not to defect first. The identification mechanism that Adaptive Pavlov used in 2005 IPD tournament only explored defecting strategies in order to keep cooperation with each of those cooperating strategies. Again, what kind of identification mechanisms should be applied depends on prior knowledge and subjective estimation. If there are enough ALLC strategies in an IPD game, it is worth identifying them from other cooperating strategies. But if GRIMs are prevailing, it is better not to defect first. Generally speaking, we can compare different identification approaches to choose the most efficient one although uncertainty still exists. 4.6. Adaptive Pavlov and Competition 4 of 2005 IPD tournament The 2005 IPD tournament comprised 4 competitions. Competition 4 mirrored the original competition of Axelrod. There were a total of 50 players including 8 default strategies. The strategy of Adaptive Pavlov (AP) that was ranked first in Competition 4 will be analyzed in this section. The strategy of AP combines 6 continuous rounds to a period and applies different tactics in different periods. AP behaves as a TFT strategy in the first period, and then changes its strategy according to the identification of its opponent. AP classifies the possible opponents into 5 categories: cooperating strategies, STFT, PavlovD, ALLD and RAND.c By identifying the opponent’s strategy at the end of a period, AP shift its strategy in the new period in order to deal with each opponent optimally. AP is never the first to defect, and thus it will cooperate with each cooperating strategy. AP tries to cooperate with the strategies of STFT and PavlovD, and defect to the strategies such as ALLD or RAND. The processes of AP’s interacting with cooperating strategies, ALLD, STFT, and PavlovD in the first 6 rounds are shown in Fig. 4.6 (AP behaves as TFT). For example, when a process of interaction as shown in Fig. 4.6(c) c RAND
is claimed to be a default strategy.
100
1
2
3
4
5
6
AP
C
C
C
C
C
C
Co-op
C
C
C
C
C
C
1
2
3
4
5
6
AP
C
D
D
D
D
D
A LLD
D
D
D
D
D
D
(a)
(b)
2
3
4
5
6
AP
C
D
C
D
C
D
STFT
D
C
D
C
D
C
(c)
1
2
3
4
5
6
AP
C
D
D
C
D
D
PavlovD
D
D
C
D
D
C
(d)
Fig. 4.6. Identifying the opponent according to the process of interaction in six rounds. (a) AP cooperates with any cooperating strategy. (b) ALLD strategy always defects. (c) If a strategy alternately plays D and C when interacting with TFT, it is identified to be STFT. (d) If a strategy periodically plays D-D-C when interacting with TFT, it is identified to be PavlovD.
J. Li
1
How to Design a Strategy to Win an IPD Tournament
101
occurs, the opponent will be identified to be STFT and AP will cooperate twice in the next period in order to achieve cooperation. If the opponent is determined to be PavlovD, AP will defect once and then always cooperate in the next period. If there is a process of interaction different from that of as shown in Fig. 4.6, the opponent will be identified as RAND. In this way, any strategy that is not defined in identification set is likely to be identified as RAND. Once cooperation has been established, AP will always cooperate unless a defection occurs. Identification of the opponent is performed in each period throughout the IPD tournament in order to correct misidentification and to deal with those players who change their strategies during a game. As we have mentioned, most of the players will be cooperating strategies. The results show that there are 34 cooperating strategies in Competition 4 (including 4 default strategies of TFT, TFTT, GRIM and ALLC). With the exception of the default strategies, there are still 3 strategies that behave like ALLD, 5 strategies that behave like STFT, and 2 strategies that behave like NEG. As shown in Table 4.4, AP can identify most of the strategies involved in Competition 4.d Table 4.4.
Categories of the strategies in Competition 4.
Categories
Number of the strategies
Cooperating strategies Strategies like STFT Strategies like ALLD Strategies like NEG Strategies like RAND Others
34 6 4 3 1 2
4.7. Discussion and conclusion AP belongs to the type of adaptive automata for IPD. However, it differs from other adaptive strategies in respect of how adaptation is achieved. The approach of AP exactly belongs to the set of artificial intelligence approaches. Rather than adjusting some parameters in computing responses as most of the adaptive strategies do, AP uses an identification mechanism d AP
regards NEG as RAND. It maximizes the scores when interacting with the strategies like NEG because either of the optimal strategies to interact with NEG and RAND are ALLD.
102
J. Li
which acts as an expert system. Knowledge about different opponents is expressed in the form of ’If..., then...”, for example, if the opponent cooperates in 6 rounds then it is determined to be ALLC. In this way, information acquired and used can be transparently expressed and thus AP can tell which strategy the opponent is using. Recent years have seen many AI approaches applied in evolutionary game theory and IPD, for example reinforcement learning, artificial neural networks, and fuzzy logic [Sandholm and Crites (1996); Macy and Carley (1996); Fort and Perez (2005)]. To solve the problem of computing a best response to an unknown strategy has been one of the objectives of those AI approaches. The problem is, in general, intractable because of the computational complexity, and finding the best response for an arbitrary strategy can be non-computable [Papadimitriou (1992); Nachbar and Zame (1996)]. Reinforcement learning which is based on the idea that the tendency to produce an action should be reinforced if it produces favourable results, and weakened if it produces unfavourable results [Gilboa (1988); Gilboa and Zemel (1989)] is widely used for the automata to learn from the interaction with others. With respect to IPD, several approaches have been developed to learn optimal responses to a deterministic or mixed strategy [Carmel and Markovitch (1998); Darwen and Yao (2002)]. However, computational complexity is still the main difficulty in the application of these approaches in real IPD tournaments. AP’s identification mechanism is implemented in a simple way by making use of a priori knowledge, which greatly reduces the computational complexity and makes it practical for AP to respond to the opponent almost optimally. First, a priori knowledge about what strategies are more likely to appear in the IPD tournament is used in determining the identification set. The size of the identification set is restricted in order to reduce computational complexity. Second, a priori knowledge about how well different identification approaches will work in a certain environment is used in selecting an efficient identification approach, with which AP can avoid the risk of identification and maximize the payoffs. Third, a priori knowledge about how to identify the opponent according to the process of interaction is used in constructing the identification rules. With these simple rules, the AP strategy is easy to understand. It is obvious that the identification set can be extended in order to include more strategies that can be identified; however, more calculations will be involved as the size of identification set increases. We have to make a tradeoff between the wish to identify any strategy and the wish to develop a less complicated strategy. Compared to the NP-completeness
How to Design a Strategy to Win an IPD Tournament
103
of those reinforcement learning approaches [Papadimitriou (1992)], AP’s √ computational complexity is between O( n) and O(n), which depends on the similarities of those strategies to be identified. Therefore, the algorithm of AP is suitable for real IPD tournaments. An identification mechanism can also work in the environment with noise, where each strategy might, with a possibility, misunderstand the outcome of game. Noise blurs the boundaries between different strategies. However, identification can still be applicable by admitting a small identification error. In this circumstance, we can set a threshold value that the opponent is considered to be identified if the probability of misidentification is smaller than this value. Just as the case of identifying the strategy of RAND, the probability of mistakenly identifying a strategy will decrease to zero as the process of computation and identification repeats. Information plays a key role in intelligent activities. The individuals with more information consequentially gain the advantage over others in most circumstances. With an identification mechanism, strategies such as AP acquire information about their opponents and they are more intelligent than any known strategies such as TFT or Pavlov. These type of strategies are suitable in modeling the decision-making process of human beings, where learning and improvement frequently happens.
References Carmel, D. and Markovitch, S. (1998). How to explore your opponent’s strategy (almost) optimally, in Proceedings of the International Conference on Multi Agent Systems, pp. 64–71. Darwen, P. and Yao, X. (2002). Co-evolution in iterated prisoner’s dilemma with intermediate levels of cooperation: Application to missile defense, International Journal of Computational Intelligence and Applications 2, 1, pp. 83–107. Fort, H. and Perez, N. (2005). The fate of spatial dilemmas with different fuzzy measures of success, Journal of Artificial Societies and Social Simulation 8, 3. Gilboa, I. (1988). The complexity of computing best response automata in repeated games, Journal of Economic Theory 45, pp. 342–352. Gilboa, I. and Zemel, E. (1989). Nash and correlated equilibria: some complexity considerations, Games and Economic Behavior 1, pp. 80–93. Macy, M. and Carley, K. (1996). Natural selection and social learning in prisoner’s dilemma: co-adaptation with genetic algorithms and artificial neural networks, Sociological Methods and Research 25, 1, pp. 103–137.
104
J. Li
Nachbar, J. and Zame, W. (1996). Non-computable strategies and discounted repeated games, Economic Theory 8, pp. 103–122. Papadimitriou, C. (1992). On players with bounded number of states, Games and Economic Behavior 4, pp. 122–131. Sandholm, T. and Crites, R. (1996). Multiagent reinforcement learning in the iterated prisoner’s dilemma, Biosystems 37, 1-2, pp. 147–166.
Chapter 5 An Immune Adaptive Agent for the Iterated Prisoner’s Dilemma
Oscar Alonso, Fernando Ni˜ no National University of Colombia
5.1. Introduction The Prisoner’s Dilemma [Tucker (1950)] is a game in which two players have to decide between two options: cooperate, doing something that is good for both players, and defect, doing something that is worse for the other player but better for oneself. No pre-play communication is permitted between the players. The dilemma arises because no matter what the other does, each player will do better defecting than cooperating, but as both players defect, both will do worse than if both had cooperated [Alonso et al.]. The payoff obtained by each player is given by a payoff matrix, as shown in table 5.1. The first number in each cell represents the payoff for the row player, and the second value represents the payoff for the column player. Table 5.1. Payoff matrix C D C 3,3 0,5 D 5,0 1,1
When the game is played several times between the same players, and the players are able to remember past interactions, it is called the Iterated Prisoner’s Dilemma (IPD). Each player is said to have a strategy, i.e., a way to decide its next move depending on previous interactions. Accordingly, complex patterns of strategic interactions may emerge, which may lead to exploitation, retaliation or mutual cooperation. The Iterated Prisoner’s Dilemma game has attracted the interest of many researchers in a wide set of fields, including game theorists, social 105
106
O. Alonso and F. Ni˜ no
scientists, economists and computer scientists [Axelrod (1984); Angeline; Hofstadter (1985); Yao and Darwen (1994)]. From the computational point of view, there has been a deep interest in the development of effective strategies for the IPD game [Yao and Darwen (1994); Axelrod (1984); Delahaye and Mathieu (1995)]. Most well-known IPD strategies have been proposed by humans, specifying the decision rules that a player will follow depending on the opponent’s behaviour[Beaufils et al. (1997); Nowak and Sigmund (1993)]. Clearly, this has mainly depended on the researcher’s assumptions about the game. In a first computational approach, Axelrod explored human designed strategies by confronting them through a tournament [Axelrod (1984)]. Conversely, there has been also some interest in obtaining IPD strategies using evolutionary computation, coevolution, reinforcement learning and other computational techniques, without explicitly specifying the decision rules [Sandholm and Crites (1995); Darwen and Yao (1995)]. These methods have found good IPD strategies, requiring little or no intervention from a human. For instance, in Axelrod’s work, human-designed strategies were compared to strategies obtained through evolution and coevolution [Axelrod (1984)]. Further research has been done towards finding strategies that generalise well without human intervention. Studies have focused on coevolutionary approaches, since no human intervention is required in the evaluation process. For instance, Darwen and Yao [Darwen and Yao (1996)] proposed a speciation scheme in order to get a modular system that played the IPD, in which coevolution and fitness sharing were used in order to get a diverse population that played as a whole against the opponent. The scheme showed a significant degree of generalisation. The model proposed in this work falls into the second kind of method. Thus, the main goal of this research is to generate an agent which will learn to play the IPD game and will be able to adapt to the opponent’s behaviour. Learning, memory and adaptation capabilities are argued to be desirable to be present in an IPD agent, and consequently, the agent implementation is accomplished using artificial immune networks, a computational technique inspired in the natural immune system that presents such capabilities. The rest of this chapter is organised as follows. First, some fundamentals about artificial immune systems, namely, immune networks are summarised. Subsequently, a general model for an adaptive agent is introduced. Then, a specific immune-based model of this agent is explained in detail. An implementation of the immune model was developed and some experiments were carried out to validate the agent capabilities. The imple-
An Immune Adaptive Agent for the Iterated Prisoner’s Dilemma
107
mented agent showed adaptation and learning; however, in some cases, the immune agent exhibited a poor performance. 5.2. Immune network fundamentals Antigens are substances capable of inducing a specific immune response. They may be viruses, bacteria, fungi, or other protozoa. They are invaders assumed to cause harm in the body. However, an antigen may be harmless, such as grass pollen [Jonathan (2001)]. On the other hand, antibodies are proteins found in the blood, produced by specialised white blood cells, called B-cells. B-cells make antibodies when the body recognises that something foreign (antigen) is present. Antibodies are the antigen-binding proteins that are present on the B-cell membrane. They are also secreted by plasma cells. The affinity between an antigen and an antibody is given by the complementarity of their binding proteins. If the antigen/antibody affinity is higher than an affinity threshold, the corresponding B-cell becomes stimulated. In the early stages of the immune response, the affinity between the antibodies and antigens may be low, but as the B-cells undergo clonal selection, the binding B-cells mutate and clone again and again to improve the affinity of the binding between a particular antigen and a B-cell. Then, the mature and activated B-cells produce plasma cells, which differentiate into antibodies with a high affinity of the antigen/antibody bonds. The Immune Network Theory tries to explain the way in which a natural immune system achieves immunological memory [Perelson and Weisbuch (1997)]. Jerne [Jerne (1974)] hypothesised that the immune system is a regulated network of molecules and cells that recognise one another even in the absence of antigens, rather than being a set of isolated cells that respond only when stimulated by antigens. Though in immune network theory the main elements are B-cells, most models only consider the antibodies attached to the B-cell membranes. Therefore, here only antibodies will be considered. The basic idea behind immune network theory is that antibodies are stimulated not only by antigens, but also by other antibodies, allowing the generated antibodies to be preserved over time for future encounters with the same or similar antigens. Therefore, when the same antigen reappears, the immune response is faster, since the immune system already contains suitable antibodies to deal with such antigen. This is known as secondary response, which is depicted in figure 5.1. [Jonathan (2001)].
108
O. Alonso and F. Ni˜ no
Fig. 5.1. Secondary Response. The amount of antibodies is greater and the response time is shorter when the antigen is presented for the second time to the immune system
Even though antibodies stimulate each other, there is also a suppression relation between them, which controls the size of the network. Thus, the network structure is a result of the interactions among antibodies. A graphical representation of an immune network model is shown in figure 5.2. An Artificial Immune Network (AIN) is a computational model based on immune network theory. In a broad sense, immune networks are mainly suitable to solve clustering and classification problems, due to their natural dynamics by which affine antibodies stimulate each other, thus forming clusters of antibodies with similar features. Typically, an immune network is stimulated by a set of antigens, corresponding to input data to a problem, and the resulting structure of the immune network will give the solution to the related problem [Castro and Zuben (2000)]. When an antigen is presented to the AIN, the internal dynamics of the AIN develops antibodies with high affinity to the antigen, through a process called affinity maturation. This process implies selection of high affinity antibodies and a mutation process called somatic hypermutation; this is
An Immune Adaptive Agent for the Iterated Prisoner’s Dilemma
Fig. 5.2.
109
Immune Network Theory
an evolutionary process that in a short period of time evolve antibodies capable of deal with the presented antigen. Several computational models for immune networks have been proposed, which are mainly derived from aiNet, a model used for optimisation and data clustering proposed by de Castro, and RAIN, a model proposed by Timmis, also used for data analysis [Castro and Zuben (2000); Castro (2003)]. In the RAIN model, the resulting set of antibodies exhibits an spatial distribution that reflects the data concentration in the data space. On the other hand, the result of the aiNet model does not present this behaviour, as highly concentrated data are considered redundant and then eliminated. In the aiNet model the interaction among antibodies leads to network suppression, i.e., antibodies that are affine (close) will suppress each other in order to control the size of the network and eliminate redundant information. Consequently, in this work the aiNet model will be used. Notice that this model does not consider stimulation among antibodies. When using an immune network to solve a problem, it is necessary to specify the following aspects:
110
O. Alonso and F. Ni˜ no
(1) Identify the entities of the problem and find the corresponding elements in an immune network, i.e., antibodies and antigens; (2) define an appropriate representation of such elements, (3) define an affinity measure between antigens and antibodies, and among antibodies themselves; and (4) establish the algorithms that model the behaviour of the immune network. 5.3. A general adaptive agent model In this section, a general adaptive agent model to play the IPD game is proposed. The model is based on trying to figure out the opponent’s strategy, which is further used to determine the next move of the agent. The information about the recent history of the game is used to model the opponent’s strategy. Accordingly, in order to decide the next move, the IPD agent will accomplish the following three phases: (1) Recognition of the opponent’s strategy (2) Development of a good strategy to face the opponent (3) Selection of the next move to play In the first phase, the Agent attempts to guess the strategy the opponent is playing, based on the recent history of moves from both players. As a result of this phase, an IPD strategy which resembles the behaviour of the opponent is obtained, which will be used in the next stage. In the second phase, the Agent generates a strategy which obtains a good score when it is faced to the strategy generated in the first phase. Finally, in the third phase, the Agent uses the strategy obtained in the second phase to decide its next move. The adaptive IPD agent consists of a memory, a recognition module, strategy generation module and a decision module (see figure 5.3), which are explained next. • The memory is responsible for storing the recent history of moves played by both, the agent and the opponent. • The recognition module is responsible for recognising the opponent’s strategy based on the recent history; it produces a strategy that resembles the one of the opponent • The strategy generation module is responsible for generating a strategy which obtains a good score when faced to the strategy that resembles
An Immune Adaptive Agent for the Iterated Prisoner’s Dilemma
Fig. 5.3.
111
General model of the IPD agent
the opponent’s • The decision module is responsible for using the strategy obtained by the strategy generation module in order to decide the next move that the agent will play Though, the model may look simple at first, it should be emphasised that the implementation of each one of the modules is not trivial. The recognition module should try to infer the strategy that the opponent is playing, which may be a difficult task. Also, the strategy generation module should be able to adapt to the changes in the opponent’s strategy. 5.4. Immune agent model The definition of a particular agent based on the general model presented above requires the stipulation of each module, as well as the representation that will be used for the strategies. The recognition of the opponent and the generation of a good strategy against it require adaptability and learning. Additionally, it would be desirable to preserve the strategies generated, which implies a memory
O. Alonso and F. Ni˜ no
112
mechanism. For these reasons, Artificial Immune Networks are used to implement the recognition and strategy generation modules. The structure of the general IPD agent can be seen in figure 5.4, and the global IPD decision making process is described in algorithm 5.1.
Fig. 5.4.
Structure of the immune agent
Algorithm 5.1. Decision making algorithm Decision making 1 while playing 2 do 3 Present history to the recognition AIN 4 Find recognised strategy from the recognition AIN 5 Present recognised strategy to strategy generation AIN 6 Find best payoff strategy from strategy generation AIN 7 Obtain suggested next move from best payoff strategy 8 Play next move
An Immune Adaptive Agent for the Iterated Prisoner’s Dilemma
113
5.4.1. Strategy representation First, each strategy is represented using a look up table [Axelrod (1984)]. This representation indicates the next move to play, based on the n previous moves of both players. The representation consists of a vector of moves, where each position in the vector indicates the next move to be played given a specific history of the game. Thus, there are 22n possible histories given a memory of n previous moves. Additionally, since there is no initial history, this representation requires 2n assumed pre-game moves at the beginning of the game. Hence, the total length of the vector of moves will be 22n +2n, and given that each position of the vector has 2 possible values, cooperate 2n and defect, the number of strategies that can be represented is 22 +2n . An example of a look up table is shown in figure 5.5.
Fig. 5.5.
Example of a Look up Table representing the strategy TFT
5.4.2. Memory The memory of the agent is represented by 2 vectors, containing the last k moves played by the agent and the opponent. 5.4.3. Recognition module An antibody of the Recognition AIN is represented by an IPD strategy. The Recognition AIN will receive as an antigen the history of recent moves of both, the opponent and the agent itself. As the agent should obtain a strategy similar to the opponent’s, the an-
114
O. Alonso and F. Ni˜ no
tibodies are stimulated according to its similarity with the opponent. This was measured presenting the moves played by the agent to each strategy, and comparing its response with the one of the opponent. Such measure is given by the Hamming distance between the move sequences of the strategy and the opponent. Additionally, the AIN model requires a measure of stimulation between antibodies. Such measure was given by the similarity between the strategies. The similarity between two strategies is measured indirectly as follows: both strategies play against a randomly generated sequence of moves. Then, the moves of the strategies are compared using the Hamming distance and the percentage of coincidences determines the similarity of the strategies. The interaction between antibodies leads to suppression, i.e., similar strategies suppress each other. After presenting the history of recent moves to the Recognition AIN, the most stimulated antibody is taken, because it represents the strategy which better resembles the opponent. A summary of the representation of the elements in the recognition AIN is given in table 5.2. Table 5.2. Recognition immune network representation Immune network Representation Antigen History of moves Antibody IPD strategy Antibody/Antigen affinity Similarity between the strategy an the opponent’s Antibody/Antibody affinity Similarity between the strategies
In addition, the process of affinity maturation requires the strategies to be mutated. Particularly, strategies will be mutated in two fashions. The first one consists of changing the number of previous interactions remembered by the strategy (memory length), and the second one consists of mutating each position of the vector that defines the strategy according to the mutation rate. The process of changing the memory length is performed as follows: the new length is selected randomly between one and the maximum allowed memory length. If the new length of the strategy is same as the old one, nothing has to be done. If it is longer, the new positions of the vector are filled in such a way that the strategy presents the same decision rules as before. This operation is shown in figure 5.6. If the new history length is shorter than before, the process is done as
An Immune Adaptive Agent for the Iterated Prisoner’s Dilemma
Fig. 5.6.
115
Example of mutation when new LuT memory length is larger
follows: notice that there are four histories which are different in only the last move. Thus, removing the last move will cause those four histories to be compressed into one. Therefore, the corresponding value of the compressed history will be the value that has the majority in the correspondent histories of the original vector. If there is a tie, it is resolved as Defect. This operation is shown in figure 5.7. 5.4.3.1. Immune network model In the aiNet model, all the antigens are known a priori and they are presented to the network many times until the structure of the network adapts to the antigen set. In contrast, the proposed IPD agent, the opponents are not known a priori, and the agent will have to be adapted to the opponents as they appear. Accordingly, to deal with such problem, a slightly modified version of the aiNet algorithm will be used. The main modification of the aiNet algorithm is introduced in the mechanism used by the network to add antibodies to the memory. An antibody interacts with the antibodies that have been already memorised. If the suppression it receives from memorised antibodies is less than the suppression threshold, it is added to the memory and will never be removed. Notice that if an antibody is suppressed by the memorised ones, it means that an antibody capable of recognising such antigen is already present in the
O. Alonso and F. Ni˜ no
116
Fig. 5.7.
Example of mutation when new LuT memory length is shorter
memory. Thus, in order to avoid redundancy, this new antibody is not added to the memory. When a new opponent starts playing a game, there is not enough information to consider that the recognised antibodies correspond to the opponent, therefore adding the antibody in the very early beginning of the game is not a good idea. Additionally, since the agent confronts the same opponent during various moves, it is not necessary to add antibodies to the memory in each movement given that the history of moves does not change significantly with only one new movement. Thus, in this situation, it is more efficient to add the antibodies that have been periodically generated every k movements. The modified version of the aiNet algorithm is summarised in algorithm 5.2.
Algorithm 5.2. Modified aiNet algorithm
An Immune Adaptive Agent for the Iterated Prisoner’s Dilemma
117
Modified aiNet 1 for each antigen 2 do 3 Add new random antibodies to the network 4 Calculate antigen/antibody affinity 5 Select the n antibodies with highest affinity 6 Clone and hypermutate selected antibodies 7 Re-calculate antigen/antibody affinity 8 Re-select a percentage of highest affinity antibodies 9 Remove low affinity antibodies 10 Calculate suppression among antibodies 11 Remove highly suppressed antibodies 12 Add resultant antibodies to the memory In the algorithm, the affinity (suppression) of the antibodies is normalised in the interval [0,1]. After that, it is considered low (high) in relation to an affinity (suppression) threshold, which is a parameter of the algorithm. Additionally, in the hypermutation process, the mutation rate is inversally related to the affinity of the antigen. Particularly, it was defined as 1- affinity. This means that high affinity antibodies are mutated less than low affinity antibodies, which helps keeping good antibodies while exploring new regions of the search space. When an new antigen is presented, the network dynamics develops antibodies with high affinity (similarity) with it. 5.4.4. Strategy generation module For this module, the antibodies are also represented as game strategies. In this case, the strategy obtained in the phase one is presented as an antigen for the second AIN. As the agent is interested in obtaining a good strategy against the one obtained in the first phase, the antibodies are stimulated according to the result of a short IPD game between the antigen and each antibody, beginning from the current history of the game between the agent and the opponent. The affinity between antibodies, the mutation operator and the immune network algorithm are defined in the same way as in recognition AIN. Therefore, the most stimulated antibody corresponds to the best strategy against the one that resembles the opponent, and is selected as the output of this phase.
118
O. Alonso and F. Ni˜ no
A summary of the representation in the strategy generation AIN is shown in table 5.3. Table 5.3. Strategy generation immune network representation Immune network Representation Antigen IPD strategy Antibody IPD strategy Antibody/Antigen affinity Payoff of a short IPD game Antibody/Antibody affinity Similarity between the strategies
5.4.5. Decision module Once a good strategy against the opponent has been found, it is used to look up the next move that the agent will play, given the recent history of the game.
5.5. Experimental results Some experiments were carried out in order to explore the capabilities of the proposed agent. All the experiments used a payoff matrix where Temptation=5, Punishment=1, Sucker’s Payoff=0 and Reward=4. The values of the parameters of an immune network affect some aspects of it, such as the number of antibodies of the network and the performance of the affinity maturation process. After testing several values for the parameters, the following were found to provide a good behaviour to the agent: the suppression threshold was 0.8, and the affinity threshold was 0.9; the number of stimulated antibodies that were selected in each iteration was 5, and the percentage of stimulated antibodies that were selected after being cloned and hypermutated was 20%. In each iteration of the immune networks, four new random antibodies were added to the network. In clonal selection, the minimum number of clones that a stimulated antibodies could generate was 5, and the maximum amount was 10. New antibodies were added to the memory of each network every 20 moves. In the recognition process, the length of the history of moves was 10, and the maximum length of the memory of the lookup table representation was set to 3 previous moves. The experiments were designed to answer some key questions about the agent’s capabilities, which are addressed in the following subsections.
An Immune Adaptive Agent for the Iterated Prisoner’s Dilemma
119
5.5.1. Can the agent adapt to a new opponent? In order to test the adaptability of the immune agent when confronting one opponent, it was faced to opponents playing the well-known strategies TFT, ALLD, Pavlov and GRIM. The length of the game was 100 moves, and there were 100 repetitions for each opponent. The average score obtained in this experiment is shown in figure 5.8. a)
b) Adaptation for TFT
Adaptation for ALLD
6
6 Agent Optimal
5
5
4
4 Average Payoff
Average Payoff
Agent Optimal
3
3
2
2
1
1
0
0 0
10
20
30
40
50 Move Number
60
70
80
90
0
100
c)
10
20
30
40
50 Move Number
60
70
80
100
90
d) Adaptation for Pavlov
Adaptation for GRIM
6
6 Agent Optimal
5
5
4
4 Average Payoff
Average Payoff
Agent Optimal
3
3
2
2
1
1
0
0 0
10
20
30
40
50 Move Number
60
70
80
90
100
0
10
20
30
40
50 Move Number
60
70
80
90
100
Fig. 5.8. Adaptability Tests. Optimal is obtained from mutual cooperation in a), c and d), and mutual defection in b).
As it can be seen, the agent adapts its behaviour to the one of the opponent, which leads to an increase of the mean payoff over the first 20 moves, and then it stabilises. 5.5.2. Can the agent adapt to consecutive opponents? The agent was confronted with two opponents one after the other, in order to evaluate the adaptability of the agent to further opponents (i.e. not only the first opponent it confronts). The results for consecutive opponents playing ALLD-TFT and PAVLOV-GRIM can be seen in figure 5.9.
O. Alonso and F. Ni˜ no
120
Adaptation for Consecutive Opponents (ALLD-TFT)
6 Agent Optimal
5
Average Payoff
4
3
2
1
0 0
20
40
60
80
120 100 Move Number
140
180
160
200
Adaptation for Consecutive Opponents (Pavlov-GRIM)
6 Agent Optimal
5
Average Payoff
4
3
2
1
0 0
20
40
Fig. 5.9.
60
80
120 100 Move Number
140
160
180
200
Tests of adaptation to consecutive opponents
Experimental results showed that the immune agent adapts to every new opponent it confront. Moreover, the curves described by the mean
An Immune Adaptive Agent for the Iterated Prisoner’s Dilemma
121
payoff is very similar to the one found in the first experiment, which shows that the agent preserves its adaptability through multiple games. 5.5.3. Can the agent remember previous opponents? Since immune networks possess a memory mechanism, this was evaluated in the agent. In this setup, the agent first confronts an opponent, then it is faced with another opponent and once again it is confronted by the first opponent. In this case two experiments were performed, the first confronted TFT-ALLD-TFT, and the second one confronted Pavlov-GRIM-PavlovGRIM. Also 100 repetitions of the experiment were carried out and the length of every game was 100 moves. The average payoff can be seen in figure 5.10. The results showed that the mean average curves stabilised faster the second time the agent faced an opponent, as a result of the memory capability of the agent. However, the mean in which the payoff stabilises did not increase. 5.5.4. Results from the IPD competition The agent proposed in this chapter participated in the IPD competitions held at CEC 2004 and CIG 2005, under the name of ”Immune Based Agent”. It competed twice in the first competition and it was ranked 126 out of 223 and 160 out of 223. In the second competition, it participated in the category # 4 (one entry per participant), and it was ranked 40 out of 50. 5.6. Discussion Experimental results show that the proposed agent presents the expected behaviour: it adapts its behaviour to the opponents it confronts in order to increase its payoff, and is also able to remember its interactions with opponents in order to recognise them faster in future encounters. However, the following was observed in the agents behaviour: • The payoff stabilises in a mean value which is less than the best possible payoff. • For some opponents such as GRIM, the performance of the agent is very poor: it obtains a payoff much lower than the best possible
O. Alonso and F. Ni˜ no
122
Memory of previous opponents (TFT-ALLD-TFT)
6 Agent Optimal
5
Average Payoff
4
3
2
1
0 300
250
200
150 Move Number
100
50
0
Memory of previous opponents (Pavlov-GRIM-Pavlov-GRIM)
6 Agent Optimal
5
Average Payoff
4
3
2
1
0 0
50
Fig. 5.10.
100
150
200 Move Number
250
300
350
400
Test of memory of previously met opponents
Some explanations to the agent’s behaviour could be hypothesised as follows:
An Immune Adaptive Agent for the Iterated Prisoner’s Dilemma
123
• The recognition module finds a strategy similar to the opponent’s, which is good enough for most cases. However, a history of moves could correspond to several opponent strategies, which makes very difficult for the recognition module to find the exact strategy that the opponent is playing. For instance, a history where all previous moves are COOPERATE could correspond to players playing TFT or ALLC, and the best response is different in every case. • Since the recognition process is imperfect, the strategy found by the strategy generation module may not be the most adequate, and could lead the agent to make bad decisions. This produces non-optimum payoff, and with some strongly retaliative opponents, it may lead to mutual defection and low payoffs. This explains why the immune agent does not obtain a good payoff confronting strategies such as GRIM, since it tries to take advantage of the opponent and, consequently, it receives a strong retaliation from GRIM. • The model does not implements a feedback mechanism which may help to determine how good a strategy selected is. Notice that the agent knows the history of moves, but it does not analyse if the strategy it is currently using is good or bad in order to change it if the strategy is performing badly. The experiments also show that since the agent does not reach the best possible payoff, it is slightly exploited by very uncooperative strategies, such as ALLD. An analysis of the performance of the strategy during the competition showed that the agent frequently evolved to mutual defection with opponents that were not fully uncooperative, such as go by majority. There were also some cases where the agent became exploited by some opponents, probably due to the perception limitations of the agent exposed above. As a consequence, the agent performed poorly in the competition. 5.7. Conclusions This work presented an agent model that played the IPD game. The model is based on artificial immune systems in order to achieve adaptability, learning and memory. Some experiments were carried out in order to evaluate the behaviour of the proposed agent. The results showed that the agent presents the expected capabilities: it adapted its own behaviour to suit the opponent’s
124
O. Alonso and F. Ni˜ no
one, going through a learning process which produced an increase of the mean payoff until it reached a stable value. Additionally, the learning process was faster when the agent met the opponent for the second time, which evidenced a memory mechanism. However, although the mean payoff increased and stabilised due to the learning process, it did not reach the optimum value. Additionally, for some strategies such as GRIM, the agent did not even obtain a payoff close to the best possible. This shows that although the agent perform as expected, it still needs to be tuned in order to avoid poor performance in some special cases. Particularly, the proposed model could be modified by using different computational techniques, such as evolutionary algorithms, to implement some of the modules. It may also be extended to include multiple levels of cooperation and multiple opponents. Additionally, the agent could be endowed with a feedback mechanism, such as reinforcement learning. References Alonso, O., Ni˜ no, F. and Velez, M. (2004). A robust immune based approach to the iterated prisoner’s dilemma, in Proceedings of the 3rd International Conference on Artificial Immune Systems, pp. 290–301. Angeline, P. J. (1994). An alternate interpretation of the iterated prisoner’s dilemma and the evolution of non-mutual cooperation, in Proceedings 4th Artificial Life Conference, pp. 353–358. Axelrod, R. (1984). The Evolution of Cooperation (Basic Books, New York, USA). Beaufils, B., Delahaye, J.-P. and Mathieu, P. (1997). Our meeting with gradual: A good strategy for the iterated prisoner’s dilemma, in Artificial Life V (Proceedings of the Fifth Int’l Workshop on the Synthesis and Simulation of Living Systems) (MIT Press), pp. 202–209. Castro, L. N. D. (2003). The immune response of an artificial immune network (ainet), in Congress on Evolutionary Computation (CEC’03) (Canberra), pp. 146–153. Castro, L. N. D. and Zuben, F. J. V. (2000). An evolutionary immune network for data clustering, in IEEE Brazilian Symposium on Artificial Neural Networks (Rio de Janeiro), pp. 84–89. Darwen, P. and Yao, X. (1995). On evolving robust strategies for iterated prisoner’s dilemma, in Progress in Evolutionary Computation, Lecture Notes in Artificial Intelligence, Vol. 956, pp. 276–292. Darwen, P. and Yao, X. (1996). Automatic modularization by speciation, in Proc. of the 1996 IEEE Int’l Conf. on Evolutionary Computation (ICEC’96) (IEEE Press, Nagoya, Japan), pp. 88–93. Delahaye, J.-P. and Mathieu, P. (1995). Complex strategies in the iterated prisoner’s dilemma, in A. Albert (ed.), Chaos and Society, Frontiers in Arti-
An Immune Adaptive Agent for the Iterated Prisoner’s Dilemma
125
ficial Intelligence and Applications, Vol. 29 (IOS Press, Amsterdam), pp. 283–292. Hofstadter, D. R. (1985). The prisoner’s dilemma computer tournaments and the evolution of cooperation, in Metamagical Themas: Questing for the essence of mind and pattern (Basic Books, New York). Jerne, N. K. (1974). Towards a network theory of the immune system, Ann. Immunol. 125, pp. 373–389. Jonathan, T. (2001). Artificial Immune Systems: A novel data analysis technique inspired by the immune network theory, Ph.D. thesis, University of Wales, Aberystwyth, Wales. Nowak, M. A. and Sigmund, K. (1993). A strategy of win-stay lose-shift that outperforms tit-for-tat in the prisoner’s dilemma game, Nature 364, pp. 56–58. Perelson, A. S. and Weisbuch, R. (1997). Immunology for physicists, Rev. Modern Physics 69, pp. 1219–1267. Sandholm, T. and Crites, R. (1995). Multiagent reinforcement learning in the iterated prisoner’s dilemma, BioSystems: Special Issue on the Prisoner’s Dilemma 37, pp. 147–166. Tucker, A. W. (1950). A two person dilemma, . Yao, X. and Darwen, P. J. (1994). An experimental study of n-person iterated prisoner’s dilemma games, Informatica 18, pp. 435–450.
This page intentionally left blank
Chapter 6 Exponential Smoothed Tit-for-Tat
Michael Filzmoser University of Vienna Reciprocating strategies, as for instance Tit-for-Tat, have been shown to be very successful in IPD Tournaments without noise, while other tournaments and analytical studies show that they perform rather poor in noisy environments. The implementation of generosity or contrition into reciprocating strategies was proposed as a solution for this poor performance. We propose a third possibility, a relief of the provocability property of reciprocating strategies, which we design by exponential smoothing. This chapter explores how exponential smoothing and Tit-for-Tat can be combined in ’Exponential Smoothed Tit-for-Tat’ strategies for the Iterated Prisoners’ Dilemma and how the strategies perform in competitions with and without noise compared to Tit-for-Tat 6.1. Introduction Robert Axelrod (1980a,b, 1984) was the first to perform computer tournaments of the Iterated Prisoners’ Dilemma (IPD). In these tournaments strategies played the Prisoners’ Dilemma repeatedly with additional information about the history of their own moves as well as of the moves of the opponent strategy. In two tournaments with 14 and 62 entries respectively the winner both times was Tit-for-Tat (TFT), submitted by Anatol Rapoport, the simplest of all participating strategies. TFT starts with cooperation and afterwards mirrors the opponent’s move of the previous round. Niceness and provocability were identified to be important properties of successful strategies in the IPD and both are embodied in TFT. Niceness in this context denotes that a strategy never should be the first to defect, while provocability denotes that an ’uncalled for’ defection of the opponent should be punished by a defection immediately (Axelrod, 1980b). 127
128
M. Filzmoser
In recent years the original IPD has been extended by the integration of ’noise’. Noise in the context of IPD can either denote measurement errors — a strategy receives incorrect information that its opponent defected while it actually cooperated and vice versa — or implementation errors — a strategy which is intended to cooperate in a given situation erroneously defects and vice versa (Bendor, 1993).a Axelrod and Wu (1995) state that noise is an important feature of real-world interaction as errors in the implementation of choice can never be completely excluded. It has been shown analytically (Molander, 1985; Bendor, 1993) as well as by further IPD tournaments which incorporated noise (Donninger, 1986; Bendor et al., 1991) that the existence of noise undermines the performance of reciprocating strategies like TFT dramatically. Bendor, Kramer and Stout (1991) argue that a main reason for the poor performance of TFT in noisy environments is the unintended involvement in vendettas of mutual or alternating defection with other nice and provocable strategies, which can be caused by one single implementation error on either side. For coping with noise Axelrod and Wu (1995) propose to make reciprocating strategies more generous or more contrite. Generosity denotes that some of the opponent’s defections are not punished as they could be the result of noise. Such generosity of course can be exploited easily but prevents from an echoing of a single error throughout the whole game and therefore can maintain mutual cooperation among reciprocating strategies. Contrition on the other hand means that a defection as a reaction to a defection of the opponent in the last round, which in turn occurred as an answer to one’s own implementation error in the round before last, should be avoided. While generosity can be conceived as a correction of the opponent’s implementation errors, contrition can be interpreted as the correction of one’s own implementation errors in a noisy environment. However both of these further-developments of reciprocating strategies for noisy environments are one-sided insofar as they focus on correcting their own or the opponent’s implementation errors only, none of these concepts attempts to correct both kinds. Moreover the mitigation of the provocability property of reciprocating strategies takes place in a rather indiscriminate way by an increase of generosity or the implementation of contrition. In an effort to improve the a We
focus exclusively on implementation error as this was the category of noise implemented in the IPD tournament of G. Kendal, P. Darwen, and X. Yao performed in April 2005 on which this study bases (see http://www.prisoners-dilemma.com).
Exponential Smoothed Tit-for-Tat
129
performance of reciprocating strategies if they play against other reciprocating strategies one must not neglect the existence of non-reciprocating strategies. Such strategies could capitalize on the combination of noise and generosity by infrequent but intentional defections. Moreover if the history of the opponent’s moves consists of a series of defections, a single cooperation, which could be an implementation error of the opponent, should not induce a reciprocating strategy to switch from defection to cooperation. In such a case of continuous defection of the opponent an increase of generosity will only reduce the performance of a reciprocating strategy. We share the opinion that a mitigation of the provocability property is essential to overcome the comparatively poor performance of reciprocating strategies like TFT in IPD tournaments with noise. To do so we propose a third alternative beside generosity and contrition. We hold the view that the whole history of the opponent’s moves as well as the misperceptions should be taken into consideration by a reciprocating strategy in the decision to cooperate or defect. Generous or contrite reciprocating strategies as proposed by Axelrod and Wu (1995) only take into account the last move of the opponent and use some additional modification rules to adapt to the situation of noise. The analysis of the entire series of moves of the opponent should allow filtering out reactions to our own implementation errors as well as the opponent’s implementation errors, which in turn should improve the performance of a so-designed reciprocating strategy. In section 6.2 we present exponential smoothing which we suggest as a method to implement the concept of considering the whole history of moves in the decision making process of reciprocating IPD strategies. Furthermore ’Exponential Smoothed Tit-for-Tat’ (ESTFT) strategies are developed. Section 6.3 reports on the performance of the ESTFT strategies in an IPD tournament in competitions with and without noise, and in comparison to TFT. Section 6.4 summarizes the main results and concludes. 6.2. Exponential Smoothed Tit-for-Tat The intention of ESTFT is to incorporate the two properties of TFT, niceness and provocability, which have demonstrated to be important ingredients of successful strategies in the IPD without noise, and mitigate provocability to adjust to the existence of noise. To do so ESTFT uses exponential smoothing. Tzafestas (2000) used exponential smoothing as the basis for the development of his meta-regulated adaptive TFT (a strategy that drops the cooperation rate when the opponent is conceived cooperative and increases
M. Filzmoser
130
it otherwise) and by Ashlock et al. (1996) for memory weighting in a study on partner selection for the IPD. However exponential smoothing has not yet been applied to cope with the problem of noise in the IPD, which is the focus of this study. In the next two subsections, first the concept of exponential smoothing will be briefly presented and afterwards applied for the design of exponential smoothed Tit-for-Tat strategies for competitions with and without noise. 6.2.1. Exponential Smoothing Exponential smoothing was originally a time series analysis approach, which can be used for the analysis of time series that neither exhibit trend nor seasonal components. It allows for weighting past – possibly not so important – observations differently than the recent ones. From the original time series Xt the exponential smoothed time series St can be calculated by (6.1). For the calculations it is necessary to indicate a starting value S 0 as for the first period no observations of the original time series exist.
St =
( S0
(1 − α)St−1 + αXt−1
if t = 0, else
(6.1)
In (6.1) α is the smoothing parameter that indicates the weight assigned to the last observation.b The higher α, the lower the smoothing of the time series, so for α = 1 exponential smoothing reproduces the original time series while for α = 0 the smoothed time series is a constant of St = S0 . Exponential smoothing can be customized for the design of reciprocating or simple deterministic IPD strategies. We conceive the series of the transformations of the opponent’s moves mt as the observations that are to be smoothed. Where the opponent’s moves are transformed into discrete numbers applying (6.2).
mt =
( 1 if opponent move in t is ’cooperate’ 0 if opponent move in t is ’defect’
(6.2)
With the adapted exponential smoothing formula (6.3) different kinds of simple deterministic strategies can be designed, that either defect if St = 0 b The
notion ’smoothing parameter’ is somewhat misleading, as a higher value for this parameter leads to a stronger consideration of currently observed values and therefore results in a less smoothed time series.
Exponential Smoothed Tit-for-Tat
131
or cooperate if St = 1. The parameter combination S0 = 1 and α = 1 exactly equals the TFT strategy (Tzafestas, 2000), with α = 0 and S 0 = 1 (respectively S0 = 0) a constant series of cooperations (respectively defections) and therefore an ALLC (respectively ALLD) strategy can be modelled. Many other combinations of the two variables, starting value S0 and smoothing parameter α, are possible which allow modelling a large number of strategies.
St =
( S0
if t = 0
(1 − α)St−1 + αmt−1
else
(6.3)
We refer to the internal register St as the ’mood’ of the strategy. This mood is a continuous variable ranging from 0 — in case of total defection of the opponent — to 1 — for total cooperation of the opponent. Intermediate values between these two extremes represent different degrees of cooperation (closer to 1) and defection (closer to 0). In the spirit of TFT the next own move (either cooperate or defect) is derived from this mood by a threshold rule (see section 6.2.2). Furthermore we need an initial mood I — an expectation about the opponent’s behavior — to calculate the bounds on the smoothing parameter α for the ESTFT strategies designed for the competition with noise (see section 6.2.2). We derive I from the optimistic assumption that the opponent strategy is cooperative or reciprocating and therefore will cooperate if it plays against ESTFT strategies except the expected 10% of implementation errors due to noise (i.e. I = 0.9). 6.2.2. Strategies for Competitions with and without Noise The ESTFT strategies were actually a two parameter family of IPD strategiesc where the two decision parameters are i) the smoothing parameter α and ii) the threshold rule for which values of St the strategy should cooperate or defect. For all ESTFT strategies the threshold rule for cooperation and defection in round t is determined as follows: for St ≥ 0.5 ESTFT cooperate otherwise defect (see 6.4).
move in t =
( ’cooperate’ if 0.5 ≤ St ≤ 1 ’defect’
if 0 ≤ St < 0.5
(6.4)
In defining one threshold rule for all ESTFT strategies the only variable parameter of these strategies is the α-value. For the ESTFT strategies c The
starting value S0 can be derived from α
132
M. Filzmoser
designed for the competition with noise we demand two additional characteristics to cope with the problem of noise in the IPD: i) they should never defect in return to a single defection of the opponent as this single defection could be an implementation error by the opponent or a reaction to an implementation error of the ESTFT strategy itself, and ii) they should react with defection in return to two consecutive defections of the opponent to avoid exploitation by the opponent. These additional requirements restrict the area of possible values for the smoothing parameter α. The possible area according to restrictions i) and ii) is calculated in (6.5) and (6.6) respectively for an initial mood of I = 0.9. Restriction i): ESTFT should cooperate after a single defection St = (1 − α)I + αmt−1 ≥ 0.5 for mt−1 = 0 and I = 0.9 0.5 = 0.44444 α ≤ 1 − 0.9
(6.5)
Restriction ii): ESTFT should defect after two consecutive defections St−1 = (1 − α)I + αmt−2 St = (1 − α)St−1 + αmt−1 < 0.5 for mt−2 = mq t−1 = 0 and I = 0.9 α>1−
0.5 0.9
(6.6)
= 0.25464
From (6.5) we derive that the ESTFT strategy will not defect after a single defection (mt−1 = 0) of assumed cooperative or reciprocating strategies (I = 0.9) when α ≤ 0.44444 as for this value of the smoothing parameter S t remains above the threshold for defection of 0.5. An α > 0.25464 guarantees that for two consecutive defections of the opponent (mt−2 = mt−1 = 0) the St lies below the threshold value and the ESTFT strategy therefore defects (6.6). Three ESTFT strategies lowESTFT_noise, mediumESTFT_noise, and highESTFT_noise were designed for the competition with noise using αvalues that represent the upper bound, the lower bound, and the average between these extremes (α ≈ 0.34) respectively. As large numbers for the smoothing parameter α lead to higher weighting of the current observations and to a less smoothed value the border induced by (6.5) is applied in the lowESTFT_noise, the border induced by (6.6) in the highESTFT_noise and the average between these borders in the mediumESTFT_noise ESTFT strategy. For the competition without noise neither of the two restrictions
Exponential Smoothed Tit-for-Tat
133
mentioned above is necessary as the true move of the opponent can be observed with certainty. We determine smoothing parameters of α = 0.2 for the highESTFT_classic , α = 0.35 for the mediumESTFT_classic , and α = 0.5 for the lowESTFT_classic strategy respectively and start with cooperation. Due to the decision rule mentioned above values above 0.5 would not change the result and are therefore omitted. The results these strategies achieved compared to TFT in competitions with and without noise are summarized in the next section.
6.3. Tournament Results The IPD computer tournament organized by Graham Kendal, Paul Darwen, and Xin Yao in April 2005 offered an excellent possibility to test the ESTFT strategies and to compare it to TFT in situations with and without noise. In addition to the classical competition without noise (competition 1) – a re-run of Robert Axelrod’s original tournaments – a competition with a 10% chance of noise in the form of implementation error (competition 2) was conducted. The Java Applet that was used to run the tournament, as well as the entries, based on the Java IPDLX software library, in addition simple deterministic strategies with a history of maximal three rounds could be entered via a web-interface. In each of the competitions five runs were performed, each of these runs lasted 200 rounds. For each competition we calculate the average of the payoffs of all five runs the ESTFT strategies and TFT reached if it plays against a specific opponent. In using the average we can filter out random effects induced by noise or by the strategies themselves (e.g. RAND). Furthermore we consider only the payoffs against the 141 strategies that are represented in both the classical competition and the competition with noise. This establishes a common basis for analysis that allows us to perform paired tests on the difference between the average payoffs against each of these 141 reference strategies. Figure 6.3 presents box-wisker diagrams for the six ESTFT strategies and TFT (the according data can be taken from Table 6.1). First we apply a non-parametric paired Wilcoxon test to test the difference in the payoffs of the ESTFT strategies and TFT between the competition without noise and the competition with noise. The alternative hypothesis that payoffs are higher in the competition without noise than in the competition with noise can be accepted for all seven reciprocating
M. Filzmoser
500 400 300
payoff (averaged over the five runs)
600
134
0.20
0.26
0.34
0.35
0.44
0.5
1 (TFT)
soomthing parameter
Fig. 6.1. Box-Wisher plot of the average payoff of ESTFT and TFT strategies for the competition with noise
strategies. The results are highly significant as can be seen from Table 6.2.d Next we test the difference in the payoff between the ESTFT strategies and TFT in the competitions without and with noise. Again we use a non-parametric paired Wilcoxon test. The alternative hypothesis that the payoffs of the focal ESTFT strategy are greater than the payoffs of TFT can be accepted only for the lowESTFT_classic and mediumESTFT_classic (p < 0.05). From Table 6.2 we see that the tournament results reproduce what has been argued analytically and already shown in previous IPD tournaments already (Molander, 1985; Donninger, 1986; Bendor et al., 1991; Bendor, 1993), reciprocating strategies like TFT or ESTFT are less successful in noisy d In
Tables 6.2 and 6.3 the column α represents the specific value of the smoothing parameter for this strategy, µ the mean of the payoff averaged over the five runs per competition, ± the standard deviation of payoffs, V the test statistic and p the significance of the non-parametric paired Wilcoxon test.
Exponential Smoothed Tit-for-Tat
135
Table 6.1. Data for the Box-Wisher plot of the average payoff of ESTFT and TFT strategies of the competition with noise strategy min 1. quartile median 3. quartile max highESTFT_classic 232.6 255.4 432.8 500.2 613.4 highESTFT_noise 242.0 268.8 435.4 515.6 594.2 mediumESTFT_noise 241.0 260.2 430.6 514.0 585.6 mediumESTFT_classic 243.0 270.6 430.4 505.0 593.2 lowESTFT_noise 240.0 264.2 435.4 513.0 597.6 lowESTFT_classic 245.8 300.6 428.0 512.2 586.4 TFT 223.4 269.0 430.6 502.6 614.2
Table 6.2. Performance of the ESTFT and TFT strategies in environments without noise without noise with noise strategy α µ ± µ ± V highESTFT_classic 0.20 467.22 181.14 397.73 123.93 7,817.0 highESTFT_noise 0.26 470.97 180.20 399.73 117.57 8,131.0 mediumESTFT_noise 0.34 470.43 180.45 399.45 120.43 8,115.5 mediumESTFT_classic 0.35 468.47 179.45 404.26 114.16 7,733.5 lowESTFT_noise 0.44 469.43 181.72 401.94 120.24 8,000.5 lowESTFT_classic 0.50 469.76 179.64 408.43 109.95 7,734.0 TFT 1.00 467.49 181.06 400.39 121.78 7,615.0
with and
p < 0.0001 < 0.0001 < 0.0001 < 0.0001 < 0.0001 < 0.0001 < 0.0001
Table 6.3. ments
Comparison of the TFT and ESTFT strategies in environwith and without noise without noise with noise strategy α V p V p highESTFT_classic 0.20 3.0 0.6054 4,779.0 0.6798 highESTFT_noise 0.26 449.0 0.9980 5,342.5 0.2443 mediumESTFT_noise 0.34 366.0 0.9998 4,962.0 0.5361 mediumESTFT_classic 0.35 500.5 0.6830 6,071.0 0.0142 lowESTFT_noise 0.44 282.0 1.0000 5,308.5 0.1758 lowESTFT_classic 0.50 628.5 0.2342 5,961.0 0.0247
environments than in environments without noise. The comparison between the performance of ESTFT strategies compared to TFT for the competition without and the competition with noise summarized in Table 6.3 shows two noteworthy results. First while there are no significant differences in performance between ESTFT strategies and TFT in the case of no noise, the ESTFT strategies lowESTFT_classic and mediumESTFT_classic are significantly better (p < 0.05) in the presence of noise. Moreover the three ESTFT strategies designed for the competition with noise (lowESTFT_noise,
136
M. Filzmoser
mediumESTFT_noise, and highESTFT_noise) where, though still less successful than TFT, able to reduce the distance to TFT in the competition with noise compared to the one without. Above we mentioned that the α-value is the only parameter that varies across the six ESTFT strategies. In the competition without noise the average performance of all ESTFT strategies except highESTFT-noise – the strategy with the lowest α-value – exceeded the performance of TFT, however these results are not significant according to the non-parametric paired Wilcoxon tests (see Table 6.3). From Table 6.2 one can see that in the competition with noise the three ESTFT strategies with the higher α-values (mediumESTFT_classic, lowESTFT_noise, and lowESTFT_classic) reach higher average payoffs than TFT while the three ESTFT strategies with lower α-values (lowESTFT_classic, lowESTFT_noise, and mediumESTFT_noise) reach lower average payoffs. Generally the performance of the ESTFT strategies in the competition with noise increases with the smoothing parameter. That two ESTFT strategies designed for the classical competition without noise outperformed TFT in the competition with noise while the ESTFT strategies designed for the competitions with noise did rather poorly, does not necessarily contradict the statements made above. We stated that an unbalanced mitigation of the provocability property of reciprocating strategies or too much generosity is insufficient to improve the performance of reciprocating strategies in the IPD with noise. A one-sided reduction of provocability which just focuses on not punishing some of the opponent’s defections as they could be the direct or indirect result of implementation errors neglects the possibility of implementation errors in combination with opponent’s cooperation, while too much generosity could cause exploitation. On the one hand the ESTFT strategies for the competition with noise were more generous as they only defect when two consecutive defections of the opponent occur or the smoothed value declines below a limit for cooperation. On the other hand highESTFT_classic probably smoothed too much which reduces its performance. Two strategies that outperformed TFT used higher values for the smoothing parameter that leads to a higher weighting of the currently observed opponent’s moves and reduces the smoothing effect. 6.4. Conclusions Based on the shortfalls of existing approaches that attempt to improve the poor performance of reciprocating strategies for the IPD with noise, we
Exponential Smoothed Tit-for-Tat
137
suggest that exponential smoothing is an approach that allows an outbalanced mitigation of the provocability property of reciprocating strategies. By exponential smoothing the whole series of the opponent’s moves rather than only the previous move of the opponent can be taken into consideration in the decision of cooperation or defection. Six ESTFT strategies were designed and participated in an IPD tournament in competitions with and without noise. The results of the tournament show that in noisy environments the performance of ESTFT strategies increases with the smoothing parameter and that low exponential smoothing improves the performance of reciprocating strategies. While exponential smoothing improves the ability of TFT to deal with noise in the IPD, it still does not deal with it very well. Moreover the results indicate that our design concept for determining smoothing parameters for strategies for the competition with noise seem to be inadequate, as in the competition with noise strategies designed for the competition without noise outperformed strategies designed especially for this environment. The optimistic assumptions concerning the initial mood of the ESTFT strategies and the simplistic restrictions for the smoothing parameter α for strategies for the competition with noise may be the cause of this weaker than expected performance. While we found a seemingly promising way to improve the performance of reciprocating strategies for noisy environments, obviously further research in this direction is necessary. Moreover we used TFT — as the most important representative of reciprocating strategies — as a benchmark, clearly ESTFT strategies have to be compared to other (reciprocating) strategies as well.
References Ashlock, D., Smucker, M. D., Stanley, E. A. and Tesfatsion, L. (1996). Preferential partner selection in an evolutionary study of prisoner’s dilemma, BioSystems 37, pp. 99–125. Axelrod, R. (1980a). Effective choice in the prisoner’s dilemma, Journal of Conflict Resolution 24, 2, pp. 3–25. Axelrod, R. (1980b). More effective choice in the prisoner’s dilemma, Journal of Conflict Resolution 24, 3, pp. 379–403. Axelrod, R. (1984). Genetic algorithms and simulated annealing, chap. The evolution of strategies in the iterated prisoner’s dilemma (Pitman, London), pp. 32–41. Axelrod, R. and Wu, J. (1995). How to cope with noise in the iterated prisoner’s dilemma, Journal of Conflict Resolution 39, 1, pp. 183–189.
138
M. Filzmoser
Bendor, J. (1993). Uncertainty and the evolution of cooperation, Journal of Conflict Resolution 37, 4, pp. 709–734. Bendor, J., Kramer, R. M. and Stout, S. (1991). When in doubt. cooperation in a noisy prisoner’s dilemma, Journal of Conflict Resolution 35, 4, pp. 691–719. Donninger, C. (1986). Paradoxical effects of social behavior. Essays in honor of Anatol Rapoport, chap. Is it always efficient to be nice? A computer simulation of Axelrod’s computer tournament (Physica, Heidelberg). Molander, P. (1985). The optimal level of generosity in a selfish, uncertain environment, Journal of Conflict Resolution 29, 4, pp. 611–618. Tzafestas, E. S. (2000). Toward adaptive cooperative behavior, in Proceedings of the simulation of Adaptive behavior conference (Paris).
Chapter 7 Opponent Modelling, Evolution, and the Iterated Prisoner’s Dilemma Philip Hingston1 , Dan Dyer2 , Luigi Barone2 , Tim French2 , Graham Kendall3 Edith Cowan University1 , The University of Western Australia2 , The University of Nottingham3 In this chapter, we report on a series of studies exploring the interplay between evolution and intelligence. The evolutionary setting is a population of agents playing Iterated Prisoners Dilemma, a setting which provides choice between cooperative and selfish behaviour in interactions between agents. Intelligence is represented using opponent modelling agents. Our studies show that, while opponent modellers can survive in such a setting, an evolving population of less intelligent agents can limit their success. We also report on the performance of our opponent modelling agent, which competed in the CIG’05 IPD competition. 7.1. Introduction IPD has served as a model for cooperation between self-interested individuals for 40 years. Sometimes, these individuals are taken to be animals, sometimes humans, and sometimes some other kind of agency, such as a corporation or a nation. A useful way to categorise studies based on the IPD model is by what is assumed about the cognitive abilities of the players. On an increasing scale of rationality, well-studied assumptions include • That populations of players can evolve good strategies. This is the traditional evolutionary computation approach. • That the players can learn good strategies. This is the traditional machine learning approach. • That the players are perfectly rational. This is the traditional mathematical game theory approach. 139
140
P. Hingston et al.
But there is another point on this scale, somewhere between the last two points, that has, surprisingly, been largely neglected — the assumption that players adapt their play based on a learned model of their opponents’ play. In this chapter, we will argue the merits of opponent modelling as a realistic approach to the study of IPD, and present some results of experiments designed to explore this approach. The expanded, four point scale, corresponds roughly to some theories concerning stages in the evolution of intelligence. A recent and controversial example is the Machiavellian intelligence hypothesis – “that apes and humans have evolved special cognitive adaptations for predicting and manipulating the behaviour of other individuals” [Miller (1997), p313]. There are various stronger or weaker interpretations of this hypothesis. A strong version postulates a “theory of mind”, a module that attributes beliefs and desires to others in order to better predict their behaviour. In other words, in our terms, apes and humans use opponent modelling. Researchers disagree about whether or not, and to what degree, various primates have such a module. Everyone seems to agree that humans do, but there is evidence supporting both sides of the argu