A game theoretical approach
Matteo Venanzi
bvenanzi@gmail.com

Utilities and preferences

Multiagent encounters
◦ Solution concepts

The Prisoner’s Dilemma
◦ IPD in a social network


Self-interested agents:
Each agent has its own preferences and desires over the
states of the world (non-cooperative game theory).
Modelling preferences:
◦ Outcomes (states of the world):
Ω = {w1, w2, ...}
◦ Utility function:
ui: Ω → R
Utility functions lead to preference orderings over outcomes
•
•
Preference over w
ui ( w)  ui ( w' )  ww'
Strict preference over w ui ( w)  ui ( w)  w  w'
Properties
• Reflexivity
w   : wi w
• Transitivity
if wi w'w' i w' '  wi w' '
• Comparability
w, w'  : wi w'  w' i w

Interaction as a game:
the states of the world can be seen as the outcomes of a
game.



Assume we have just two agents (players) Ag={i,j},
The final outcome in Ω depends on the combination of
actions selected by each agent.
State transformer function:
:
Ac

agent i 's action

Ac

agent j 's action
 

Normal-form game (or strategic-form game):
A strategic interaction is familiarly represented in game theory as
a tuple (N, A, u), where:
◦ N is (finite) the set of player.
◦ A = A1 x A2 x ... x An where Ai is the set of actions available to
player i
◦ U = (u1, u2, ... , un) utility functions of each player

Payoff matrix:
n-dimentional matrix with cells enclosing the values of the utility
functions for each player
i cooperates
i defects
j cooperates
1, 1
1, 4
j defects
4,1
4, 4

Rational agent’s strategy:
Given a game scenario, how a rational agent will act?
• Apparently: “just” maximize the expected payoff
(single-agent point of view).
– In most cases unfeasible because the individual best strategy
depends on the choices of others (multi-agent point of view)
• In practice: answer in solution concepts:
– Dominant strategies
– Pareto optimality
– Nash equilibrium

Best response: given the player j’s strategy sj, the player i’s
best response to sj is the strategy si that gives the highest
payoff for player i.
i: C
i: D
j: C
1, 1
1, 4
j: D
4,4
4, 1
Example:
j plays C
j plays D
----->
----->
i’s best response = D
i’s best response = C

Dominant strategy: A strategy si* is dominant for player i if
no matter what strategy sj agent j chooses, i will do at least
as well playing si* as it would doing anything else.
(si* is dominant if it is the best response to all of agent j’s
strategies.)
i: C
i: D
j: C
1, 4
1, 1
j: D
4,1
4, 4
Example:
- D is the dominant strategy for player j
-There are not dominant strategies for player i
Dominated strategy can be removed from the table

Dominant strategy: A strategy si* is dominant for player i if
no matter what strategy sj agent j chooses, i will do at least
as well playing si* as it would doing anything else.
(si* is dominant if it is the best response to all of agent j’s
strategies.)
i: C
i: D
j: C
1, 4
1, 1
j: D
4,1
4, 4
Example:
- D is the dominant strategy for player j
-There are not dominant strategies for player i
Dominated strategy can be removed from the table

Dominant strategy: A strategy si* is dominant for player i if
no matter what strategy sj agent j chooses, i will do at least
as well playing si* as it would doing anything else.
(si* is dominant if it is the best response to all of agent j’s
strategies.)
j: D
i: C
i: D
4,1
4, 4
Example:
- D is the dominant strategy for player j
-There are not dominant strategies for player i
Dominated strategy can be removed from the table

Pareto optimality (or Pareto efficiency): An outcome for the game
is said to be Pareto optimal (or Pareto efficient) if there is no
other outcome that makes one agent better off without making
another agent worse off.
If an outcome w is not Pareto optimal, then there is another outcome w’ that makes
everyone as happy, if not happier, than w. “Reasonable” agents would agree to move to
w’ in this case. (Even if I don’t directly benefit from w’, you can benefit without me
suffering.)

Pareto optimality (or Pareto efficiency): An outcome for the game
is said to be Pareto optimal (or Pareto efficient) if there is no
other outcome that makes one agent better off without making
another agent worse off.
Girl: whole cake
Girl: half cake
Girl: nothing
Boy: whole cake
Boy: Half cake
Boy: Nothing
If an outcome w is not Pareto optimal, then there is another outcome w’ that makes
everyone as happy, if not happier, than w. “Reasonable” agents would agree to move to
w’ in this case. (Even if I don’t directly benefit from w’, you can benefit without me
suffering.)

Pareto optimality (or Pareto efficiency): An outcome for the game
is said to be Pareto optimal (or Pareto efficient) if there is no
other outcome that makes one agent better off without making
another agent worse off.
Boy: whole cake
Boy: Half cake
Boy: Nothing
Girl: whole cake
Girl: half cake
Girl: nothing
Not Pareto efficient
Not Pareto efficient
Pareto efficient
Not Pareto efficient
Pareto efficient
Not Pareto efficient
Pareto efficient
Not Pareto efficient
Not Pareto efficient
If an outcome w is not Pareto optimal, then there is another outcome w’ that makes
everyone as happy, if not happier, than w. “Reasonable” agents would agree to move to
w’ in this case. (Even if I don’t directly benefit from w’, you can benefit without me
suffering.)
Pareto optimality is a nice proberty but not very
useful for selecting strategies
Nash equilibrium for pure strategies:
Two strategies s1 and s2 are in Nash equilibrium if:
1. under the assumption that agent i plays s1, agent j can do
no better than play s2;
AND
2. under the assumption that agent j plays s2, agent i can do
no better than play s1.


Neither agent has any incentive to deviate from a Nash equilibrium.
Nash equilibrium represents the “Rational” outcome of a game played
by self-interested agents.
Unfortunately:
◦ Not every interaction scenario has a Nash equilibrium.
◦ Some interaction scenarios have more than one Nash equilibrium.
But:
Nash’s Theorem (1951): Every game with a finite number of players and
a finite set of possible strategies has at least one Nash equilibrium in
mixed strategies.
However:
“The complexity of finding a Nash equilibrium is the most important
concrete open question on the boundary of P today”
[Papadimitriou, 2001]
John Forbes Nash, Jr.
Bluefield. 1928
Nobel Prize for
Economics, 1994
“A beautiful mind”

Battle of the sexes game: husband and wife wish to go to the movies and
they are undecided between “lethal Weapon (LW)” and “Wondrous Love
(WL)”. They much prefer to go together rather than separate to the
movie, altought the wife prefers LW and the husband prefers (WL)”.
husband:
husband:
wife: LW
2, 1
0, 0
wife: WL
0,0
1, 2
LW
WL
Easy way to find pure-strategy Nash equilibria in a payoff matrix:
Take the cell where the first number is the maximum of the column
and check if the second number is the maximum of that row.

Battle of the sexes game: husband and wife wish to go to the movies and
they are undecided between “lethal Weapon (LW)” and “Wondrous Love
(WL)”. They much prefer to go together rather than separate to the
movie, altought the wife prefers LW and the husband prefers (WL)”.
husband:
husband:
wife: LW
2, 1
0, 0
wife: WL
0,0
1, 2
LW
WL
Easy way to find pure-strategy Nash equilibria in a payoff matrix:
Take the cell where the first number is the maximum of the column
and check if the second number is the maximum of that row.

Max-min strategy: the maxmin strategy for player i is the
strategy si that maximizes the i’s payoff in the worst case.
(Minmax is the dual strategy)
Example:
(zero-sum game: the sum of the utilities is always zero)
Player 2: A
Player 2: B
Player 1: A
2, -2
3, -3
Player 1: B
0,0
4, -4
From the player 1’s point of view:
- If he plays A, the min payoff is 2.
- If he plays B, the min payoff is 0.
Player 1 chooses A.
With the same reasoning:
 player 2 chooses A too.
• The “maxmin solution” is (A, A).
• It is also the (only) Nash equilibrium of the game. In fact:
Minmax Theorem ( von Neuman, 1928 ) : In any finite, two-players, zero-sum
game maxmin strategies, minmax strategies and Nash equilibria coincide.
Two men are collectively charged with a crime and
held in separate cells, with no way of meeting or
communicating. They are told that:
if one confesses and the other does not, the
confessor will be freed, and the other will be jailed
for five years;
◦ if both confess, then each will be jailed for two
years.
◦ Both prisoners know that if neither confesses, then
they will each be jailed for one year.
◦


Well-known (and ubiquitous) puzzle in game theory.
The fundamental paradox of multi-agent interactions.
Payoff matrix for prisoner’s dilemma:
Player 2 confesses
Player 2 does not confess
Player 1 confesses
1, 1
5, 0
Player 1 does not confess
0,5
3, 3
Player 2 Defects
Player 2 Cooperates
Player 1 Defects
1, 1
5, 0
Player 1 Cooperates
0,5
3, 3
(Confession = defection, No confession = cooperation)
•
Solution concepts:
◦
◦
◦
•
D is a dominant strategy.
(D, D) is the only Nash equilibrium.
All outcomes except (D;D) are Pareto optimal.
Game theorist’s conclusions:
◦
In the worst case Defection guarantees a highest payoff so Defection is for both players
the rational move to make.

•
Both agents defects and get payoff = 2
Why dilemma..?
◦
They both know they could make better off by both cooperating and each get payoff =3.
• Cooperation is too dangerous!!
 In the one-shot prisoner’s dilemma the individual rational choice is defection


Observation: If the PD is repeated a number of time (so, I will
meet my opponent again) then the incentive to Defect appears to
decrease and Cooperation can finally emerge (Iterated Prisoner’s
Dilemma).
Axelrod in 1985 organized two computer tournament,
inviting participants to submit their strategies to play
the IPD.
◦ The length of the game was unknown to the players.
◦ 15 strategies submitted.
◦ Best strategies:



SPITEFUL: Cooperate until receive a defection, then always defect
MISTRUST: Defect, then play the last opponent’s move.
TIT FOR TAT: Cooperate on 1st round, then play what the opponent played on the
previous round.
◦ TIT_FOR_TAT won both the tournaments!!!
(Still today, nobody was able to “clearly” defeat TFT)
Even though TFT was the best strategy in the Axelrod’s tournament,
it doesn’t mean it the Optimal Strategy for the IPD. In fact:
Axelrod’s Theorem: If the game is long enough and the players
do care about their future encounters, a universal optimal
strategy for the Iterated Prisoner’s Dilemma does not exist.

Axelrod also suggests the following rules for a successful
strategy for the IPD:
 Don’t be envious: Remember the PD is not a zero-sum game. Don’t play as you
would strike your opponent!
 Be nice: Never be the first to defect.
 Retaliate appropriately: There must be a proper measure in rewarding
cooperation and punishing defection.
 Don’t be too clever: The performance of a strategy are not necessarily related
to its complexity.

Imagine agents playing the IPD, arranged as a graph where the
edges represents rating relationships.

The agent can use the knowledge and experience of his neighbours for
retrieving ratings about his opponent and inferring his reputation.
Rating propagation:
Mechanism to propagate ratings from one node to another within the
network.
◦
possible solution: the propagated rating the product
of the weights on the edges along the path
connecting the two players. In case of Multiple Path,
take the one with the highest weight product.
P max  max p   wPath w

Strategy: Cooperate with probability Pmax
Random Network
 TrustRate-Agents get all around
the same score
 The ranking keeps the features of
the Axelrod’s Tournament
Dense Graph
 Increases the number of ratings.
 There is always a nice path between
any couple of players.
Get new agents into the game
in progress
Agent 11 inserted at round 55/100
 The network helps the new player to
immediately recognize the opponents
with who cooperate or to avoid.
 Agent 11 scores the same points than
the starting players.

Conclusions:
◦ Acquaintance’s network allows to promote
cooperation in the IPD with a distributed approach.
 No data overload on a single agent.
 Information spread through the whole network.
◦ Incoming agents are immediately in conditions to
get into the logics of cooperation.
 If surrounded by some nice neighbourhood, the agent is able
to immediately distinguish cooperative players from defective
players, even without any prior experience.


An introduction to Multiagent Systems (2nd edition).
M. Wooldridge, John Wiley, 2009.
[Cap. 11]
Multiagent Systems. Algorithmic, Game-Theoretic, and Logical
Foundations.
Y. Shoham, K. Leyton-Brown, Cambridge, 2009.
[Cap. 3]
(Free download at http://www.masfoundations.org)

Trust, Kinship and Locality in the Iterated Prisoner’s Dilemma.
M. Venanzi. MSc Thesis.
[Cap. 5]
(PDF available here)