ART Testbed
 Join the discussion group at:
http://www.art-testbed.net
© 2006 THE UNIVERSITY OF TEXAS AT AUSTIN
1
ART Testbed Questions









Can agents request reputations about themselves?
Can an agent produce an appraisal without purchasing opinions?
Does the Testbed assume a common representation for reputations?
Does the Testbed prevent agents from winning via action-planning
skills, as opposed to trust-modeling skills?
What if an agent can’t or won’t give a reputation value?
Why does it cost more to generate an accurate opinion than an
inaccurate one?
Why not have a centralized reputation broker?
Isn’t it unrealistic to assume a true value of a painting can be
known? Is art appraisal a realistic domain?
Why not design an incentive-compatible mechanism to enforce
truth-telling?
© 2006 THE UNIVERSITY OF TEXAS AT AUSTIN
2
“Really Good” ART Testbed Questions
 Is there a consensus on the definitions of “trustworthiness” and
“reputation”?
 How can collusion be avoided?
 Is truth-telling a dominant strategy?
 Will the system reach equilibrium, at which point reputations are no
longer useful?
 What happens if client fee (100), opinion cost (10), and reputation
cost (1) are changed?
 Do any equilibria exist?
 What happens when agents enter or leave the system?
 When will agents seek out reputations?
 Space of experiments is underexplored—that’s a good thing!
© 2006 THE UNIVERSITY OF TEXAS AT AUSTIN
3
Questions about the Paper
 What is a “trust model”?
 How does q-learning work? How related to reinforcement learning?
How do rewards tie in?
 What is lambda?
 How can experience- and reputation-based learning be combined to
overcome the weaknesses of each (intermediate lambda values)?
 What about different combinations of (more sophisticated) agents in
a game?
 Why the assumptions chosen? They seem too extreme.
 Reputation decisions weren’t examined very well.
© 2006 THE UNIVERSITY OF TEXAS AT AUSTIN
4
The Agent Reputation and Trust Testbed:
Experimentation and Competition
for Trust in Agent Societies
Karen K. Fullam1, Tomas B. Klos2, Guillaume Muller3, Jordi Sabater4, Andreas Schlosser5, Zvi
Topol6, K. Suzanne Barber1, Jeffrey S. Rosenschein6, Laurent Vercouter3, and Marco Voss5
1Laboratory
for Intelligent Processes and Systems, University of Texas at Austin, USA
for Mathematics and Computer Science (CWI), Amsterdam, The Netherlands
3 Ecole Nationale Superieure des Mines, Saint-Etienne, France
4Institute of Cognitive Science and Technology (ISTC), National Research Council (CNR), Rome, Italy
5IT Transfer Office, Darmstadt University of Technology, Darmstadt, Germany
6Multiagent Systems Research Group—Critical MAS, Hebrew University, Jerusalem, Israel
2Center
Testbed Game Rules
If an appraiser is not
very knowledgeable
For a fixed price, clients
Agents functionabout
as a painting, it can
ask appraisers to provide
purchase "opinions"
art appraisers with
appraisals of paintings
from
varying expertise
in other appraisers.
from various eras.
different
Appraiserartistic eras.
Agent
Client
Client
Appraiser
Agent
Opinions and
Reputations
Client
Clientwhose
Share
Appraisers
appraisals are more
accurate receive larger
shares of the client base
in the future.
The Agent Reputation and Trust Testbed, 2006
Appraiser
Agent
Appraiser
Agent
Appraisers can also buy
and sell reputation
information about other
Appraisers compete toappraisers.
achieve the highest
earnings by the end of the game.
Appraiser
Agent
Step 1: Client and Expertise Assignments



Appraisers receive clients
who pay a fixed price to
request appraisals
Client paintings are randomly
distributed across eras
As game progresses, more
accurate appraisers receive
more clients (thus more profit)
The Agent Reputation and Trust Testbed, 2006
Step 2: Reputation Transactions




Appraisers know their own
level of expertise for each era
Appraisers are not informed
(by the simulation) of the
expertise levels of other
appraisers
Appraisers may purchase
reputations, for a fixed fee,
from other appraisers
Reputations are values
between zero and one
• Might not correspond to
appraiser’s internal trust
model
• Serves as standardized
format for inter-agent
communication
The Agent Reputation and Trust Testbed, 2006
Step 2: Reputation Transactions
Requester sends
request message to a
potential reputation
provider, identifying
appraiser whose
reputation is
requested
Requester
Requester sends fixed
payment to the
provider
The Agent Reputation and Trust Testbed, 2006
Provider
Request
Accept
Payment
Reputation
Potential reputation
provider sends
“accept” message
Provider sends
reputation information,
which may not be
truthful
Step 3: Opinion Transactions




For a single painting, an
appraiser may request
opinions (each at a fixed
price) from as many other
appraisers as desired
The simulation “generates”
opinions about paintings for
opinion-providing appraisers
Accuracy of opinion is
proportional to opinion
provider’s expertise for the
era and cost it is willing to pay
to generate opinion
Appraisers are not required to
truthfully reveal opinions to
requesting appraisers
The Agent Reputation and Trust Testbed, 2006
Step 3: Opinion Transactions
Requester sends
request message to a
potential opinion
provider, identifying
painting
Requester
Requester sends fixed
payment to the
provider
The Agent Reputation and Trust Testbed, 2006
Provider
Request
Certainty
Potential provider
sends a certainty
assessment about the
opinion it can provide
- Real number (0 – 1)
- Not required to
truthfully report certainty
assessment
Payment
Opinion
Provider sends
opinion, which may not
be truthful
Step 4: Appraisal Calculation





Upon paying providers and
before receiving opinions,
requesting appraiser submits
to simulation a weight (selfassessed reputation) for each
other appraiser
Simulation collects opinions
sent to appraiser (appraisers
may not alter weights or
received opinions)
Simulation calculates “final
appraisal” as weighted
average of received opinions
True value of painting and
calculated final appraisal are
revealed to appraiser
Appraiser may use revealed
information to revise trust
models of other appraisers
The Agent Reputation and Trust Testbed, 2006
2006 ART Testbed
Competition Results
Karen K. Fullam
The Laboratory for Intelligent Processes and Systems
Electrical and Computer Engineering
The University of Texas at Austin
http://www.lips.utexas.edu
Competition Organization
 “Practice” Competition
• Spanish Agent School, Madrid, April 2006
• 12 participants
 International Competition
• AAMAS, Hakodate, May 2006
• Preliminary Round
13 Participants
5 games each
• Final Round
5 Finalists
10 games with all finalists participating
© 2006 THE UNIVERSITY OF TEXAS AT AUSTIN
14
Bank Balances
Iam achieves
highest bank
balances
© 2006 THE UNIVERSITY OF TEXAS AT AUSTIN
15
Opinion Purchases
Joey and
Neil do
not
purchases
opinions
Sabatini
purchases
the most
opinions
© 2006 THE UNIVERSITY OF TEXAS AT AUSTIN
16
Opinion Earnings
Sabatini
and Iam
provide
the most
opinions
Neil and
Frost do
not provide
many
opinions
© 2006 THE UNIVERSITY OF TEXAS AT AUSTIN
17
Opinion Sensing Costs
Iam
invests
the most
in
opinions it
generates
© 2006 THE UNIVERSITY OF TEXAS AT AUSTIN
18
Expertise vs. Bank Balance
Iam’s average
expertise was
not significantly
higher than
others’
Greater Expertise
© 2006 THE UNIVERSITY OF TEXAS AT AUSTIN
19
Learning Trust Strategies in
Reputation Exchange Networks
Karen K. Fullam
K. Suzanne Barber
The Laboratory for Intelligent Processes and Systems
Electrical and Computer Engineering
The University of Texas at Austin
http://www.lips.utexas.edu
Trust Decisions in Reputation Exchange Networks
 Agents perform transactions to obtain needed resources
• Transactions have risk because partners may be untrustworthy
• Agents must learn whom to trust and how trustworthy to be
 When agents can exchange reputations
• Agents must also learn when to request reputations and what
reputations to tell
• Agents’ trust decisions affect each other
If I cheat A, and
 Difficult to learn each decision independently
Resources
A tells
B, trustworthy
will it hurt
How
my interactions
should I with
be?
Should I B?
trust?
(goods, services,
information)
IfWhat
I lie toreputations
others that
Cshould
is bad,I tell?
can I
monopolize C’s
interactions?
Which
reputations
should I listen to?
Reputations
© 2006 THE UNIVERSITY OF TEXAS AT AUSTIN
21
Enumerating Decisions in a Trust Strategy
Num agents = a
Num transaction types = e
Num choices/decision = n
Trustee
Truster
Trustee
Truster
Agent Role
Transaction
Fundamental
Reputation
How trustworthy
should I be?
Should I tell an
accurate
reputation?
combinations  n
ae
combinations  n
a 2e
Should I believe
this reputation?
Should I trust?
combinations  n
ae
combinations  n
a 2e
© 2006 THE UNIVERSITY OF TEXAS AT AUSTIN
If these
decisions affect
each other,
there are
n
2 ae1 a  
possible
strategies!
How to learn the
best strategy
with so many
choices?
22
Reinforcement Learning
Strategy
Expected Reward
Select a
strategy
A
B
C
D
Strategies with
higher expected
rewards are more
likely to be selected
© 2006 THE UNIVERSITY OF TEXAS AT AUSTIN
Strategy
feedback
influences
expected
reward
23
Learning In Reputation Exchange Networks
Strategy
Expected Reward
Decision
Tr(A),Tr(B),Tr(C)…
Tr(A)
⌐Tr(A),Tr(B),Tr(C)…
⌐Tr(A)
Expected Reward
Tr(A),⌐Tr(B),Tr(C)…
Decision
⌐Tr(A),⌐Tr(B),Tr(C)…
Expected Reward
Tr(B)
Tr(A),Tr(B),⌐Tr(C)…
⌐Tr(B)
⌐Tr(A),Tr(B),⌐Tr(C)…
Tr(A),⌐Tr(B),⌐Tr(C)…
Decision
⌐Tr(A),⌐Tr(B),⌐Tr(C)…
Expected Reward
Tr(C)
...
⌐Tr(C)
Because decisions
are interdependent,
 2 ae1 a  
there are n
.
possible strategies!
Use the ART
Testbed as a
case study
© 2006 THE UNIVERSITY OF TEXAS AT AUSTIN
Removing
interdependencies makes
each decision
in the strategy
learnable
24
Many Interdependent Decisions
Reputation
Requester’s
reputation costs
Opinion
Requester’s
opinion costs
Opinion
Provider’s
opinion order
costs
Reputation
Provider
Reputation
Requester
Opinion
Requester
Opinion
Provider
Number of
requests received
by Reputation
Provider
Accuracy of
Reputation
Requester’s trust
models
Accuracy of
Opinion
Requester’s
appraisals
Number of
requests received
by Opinion
Provider
Opinion
Requester’s
client revenue
Opinion
Provider’s
opinion revenue
Reputation
Provider’s
reputation
revenue
When Reputation
Requester is
Opinion Requester
Other
Appraisers’
client revenue
© 2006 THE UNIVERSITY OF TEXAS AT AUSTIN
25
Opinion Requester Feedback
Opinion
Client
Reward =
– Purchase
Revenue
Costs
Opinion
Requester’s
opinion costs
Opinion
Requester
Assume: Client
revenue feedback is
wholly attributed to
Opinion Requester
decision
Divide revenue
(client revenue)
among opinions
based on opinion
accuracy
Opinion
Requester’s
client revenue
© 2006 THE UNIVERSITY OF TEXAS AT AUSTIN
26
Opinion Provider Feedback
Opinion
Provider’s
opinion order
costs
Opinion
Opinion
Reward = Selling – Generating
Revenue
Costs
Opinion
Provider
Assume: Client
revenue is not related
to Opinion Provider
decision
Opinion
Provider’s
opinion revenue
Other
Appraisers’
client revenue
© 2006 THE UNIVERSITY OF TEXAS AT AUSTIN
27
Reputation Provider Feedback
Reputation
Reward =
Selling
Revenue
Reputation
Provider
Reputation
Provider’s
reputation
revenue
Assume: Client
revenue is not
related to Reputation
Provider decision
Other
Appraisers’
client revenue
© 2006 THE UNIVERSITY OF TEXAS AT AUSTIN
28
Reputation Requester Feedback
Reward =
l(
Opinion
Requester
Reward
)
Reputation
– Purchase
Costs
Reputation
Requester’s
reputation costs
Opinion
Requester’s
opinion costs
Reputation
Requester
Opinion
Requester
l = 0: Past experience only
 Opinion-requesting decision
 No reward for requesting reputations
l = 1: Reputations only
 Reputation-requesting decision
 Full reward for requesting reputations
Opinion
Requester’s
client revenue
© 2006 THE UNIVERSITY OF TEXAS AT AUSTIN
l determines
influence of:
past experience
vs. reputations
in deciding to
purchase
opinions
29