Reputation Systems
Guest Lecture
Paul Resnick
Associate Professor
Univ. of Michigan
School of Information
presnick@umich.edu
Learning Objectives

Understand
– What a reputation system is
– Theory about when and why it should work
– Open research questions

Participate in design
– Recognize situations when it might be helpful
– Raise some of the difficult design challenges
si.umich.edu
SCHOOL OF INFORMATION
UNIVERSITY OF MICHIGAN
Outline





What is a reputation system?
Theory: when/why they should work
Empirical results
Design space
Case study: NPAssist recommender
si.umich.edu
SCHOOL OF INFORMATION
UNIVERSITY OF MICHIGAN
Definition

A Reputation System…
– Collects
– Distributes
– Aggregates

…information about behavior
si.umich.edu
SCHOOL OF INFORMATION
UNIVERSITY OF MICHIGAN
Examples




BBB
Bizrate
eBay
Expertise sites
– Epinions “top reviewers”
– Slashdot karma system
si.umich.edu
SCHOOL OF INFORMATION
UNIVERSITY OF MICHIGAN
Why Reputation Systems


Interacting with strangers
Sellers (Exchange Partners) Vary
– Skill
– Effort
– Ethics
si.umich.edu
SCHOOL OF INFORMATION
UNIVERSITY OF MICHIGAN
Other Trust-Inducing Mechanisms
in E-commerce



Insurance
Escrow
Fraud Prosecution
si.umich.edu
SCHOOL OF INFORMATION
UNIVERSITY OF MICHIGAN
How Reputation Systems Should
Work



Information
Incentive
Self-selection
si.umich.edu
SCHOOL OF INFORMATION
UNIVERSITY OF MICHIGAN
Some Issues






Anonymity
Name changes
Name trades
Lending reputations
Eliciting evaluation
Honesty of evaluations
si.umich.edu
SCHOOL OF INFORMATION
UNIVERSITY OF MICHIGAN
Anonymity Analysis
Interaction
Type
ID
changes
Anonymous
every
xaction
Pseudonyms
at will
Identified
never
Interaction
type
ID
Reputation
Trust/
changes
Sharing cooperation
Anonymous
every
xaction
Pseudonyms
at will
1L Pseudonyms
each
arena
Identified
never
Interaction
type
ID
changes
Reputation
Trust/
Sharing
cooperation
Anonymous
every
xaction
none
none
Pseudonyms
at will
+ only
+ only
1L Pseudonyms
each
arena
Identified
never
+ and –
+ and 0
1L Pseudonyms

Third-party issues pseudonyms
– No cost
– Not replaceable
– Reveal name to third party
– Don’t reveal mapping of name to
pseudonym
si.umich.edu
SCHOOL OF INFORMATION
UNIVERSITY OF MICHIGAN
Interaction
type
ID
Reputation
changes
Sharing
Trust/
cooperation
Anonymous
every
xaction
none
none
Pseudonyms
at will
+ only
+ only
1L
Pseudonyms
each
arena
Identified
never
+ and –
+ and 0
within arena within arena
+ and –
+ and 0
Empirical Results: eBay




Feedback is provided
It’s almost all positive
Reputations are informative
Reputation benefits
– Effect on probability of sale
– Effect on price
si.umich.edu
SCHOOL OF INFORMATION
UNIVERSITY OF MICHIGAN
Provision of Feedback
negative
neutral
positive
none
Total


Buyer of Seller
Frequency Percent
111
0.3
62
0.2
18,569
51.2
17,491
48.3
36,233
Seller of Buyer
Frequency
Percent
353
1.0
60
0.2
21,560
59.5
14,260
39.4
36,233
Negatives: paid but did not receive;
seller cancelled; not as advertised;
communication
Neutrals: slow shipping, not as
advertised, communication
si.umich.edu
SCHOOL OF INFORMATION
UNIVERSITY OF MICHIGAN
Feedback Profiles of Buyers and
Sellers
Group
0-9 positive
10-49 positive
50-199 positive
200-999 positive
1000+
All
si.umich.edu
N
Percent neutral
N
(Sellers) and negative (Buyers)
(Sellers)
4,018
2.83%
13,306
3,932
1.25%
7,366
3,728
0.95%
3,678
1,895
0.79%
738
122
1.18%
15
13,695
0.93%
25,103
Percent neutral
and negative
(Buyers)
1.99%
1.09%
0.76%
0.60%
0.92%
0.83%
SCHOOL OF INFORMATION
UNIVERSITY OF MICHIGAN
Predicting Problematic
Transactions

Logistic Regression
N = 36233
Beginning Block Number 0. Initial Log Likelihood Function
-2 Log Likelihood
2194.3468
-2 Log Likelihood
2075.420
Dependent Variable..
NEGNEUT
---------------------- Variables in the Equation ----------------------Variable
B
LNNPOS
LNPOS
Constant
.7712
-.5137
-3.9399
S.E.
Wald
df
Sig
R
Exp(B)
.1179 42.7907
.0475 116.8293
.1291 931.3828
1
1
1
.0000
.0000
.0000
.1363
-.2288
2.1624
.5983
f(0,0) = 1.91% f(100,0) = .18% f(100,3) = .53%
si.umich.edu
SCHOOL OF INFORMATION
UNIVERSITY OF MICHIGAN
Predictive Value
Predicting Problem Transactions
1.00
.75
Sensitivity
.50
.25
0.00
0.00
.25
.50
.75
1.00
1 - Specificity
1-specificity
Sensitivity
Cutoff
(% of unproblematic (% of problematic
predicted
transactions rejected) transactions rejected) probability
75%
94.2%
.20%
50%
81.5%
.31%
25%
57.2%
.54%
10%
32.4%
1.09%
0%
0%
Accept all
si.umich.edu
% of accepted
transactions that
are problematic
.11%
.18%
.27%
.36%
.48%
SCHOOL OF INFORMATION
UNIVERSITY OF MICHIGAN
Some Recently Completed Work

Experiment: does reputation affect
profit?
– Many positives: Yes, but only a little (8.1%)
– One or two negatives: No

Incentives for quality feedback provision
– Can pay based on agreement among
raters
si.umich.edu
SCHOOL OF INFORMATION
UNIVERSITY OF MICHIGAN
Studies Currently Underway

Feedback provision (empirical)
– Reciprocation, altruism, and free riding


Dynamics: learning and selection
(empirical)
Geography: trust and trustworthiness by
state
si.umich.edu
SCHOOL OF INFORMATION
UNIVERSITY OF MICHIGAN
Design Space





Rating scales
Aggregation of ratings
Who rates?
Incentives for raters
Identification/Anonymity
– Exchange partners
– Evaluation providers
si.umich.edu
SCHOOL OF INFORMATION
UNIVERSITY OF MICHIGAN
Case Study


Goal: help Michigan non-profits select
consultants and other service providers
Is this a good candidate for a reputation
system?
si.umich.edu
SCHOOL OF INFORMATION
UNIVERSITY OF MICHIGAN
Case Study


Goal: help Michigan non-profits select
consultants and other service providers
Is this a good candidate for a reputation
system?
Interacting with strangers
Sellers (Exchange Partners) Vary
Skill
Effort
Ethics
si.umich.edu
SCHOOL OF INFORMATION
UNIVERSITY OF MICHIGAN
Case Study Design Choices





Rating scales
Aggregation of ratings
Who rates?
Incentives for raters
Identification/Anonymity
– Exchange partners
– Evaluation providers
si.umich.edu
SCHOOL OF INFORMATION
UNIVERSITY OF MICHIGAN
si.umich.edu
SCHOOL OF INFORMATION
UNIVERSITY OF MICHIGAN
si.umich.edu
SCHOOL OF INFORMATION
UNIVERSITY OF MICHIGAN
si.umich.edu
SCHOOL OF INFORMATION
UNIVERSITY OF MICHIGAN
Summary



RS inform, incent, select
Opportunity for RS: interactions with
strangers
Design space
– Scales, aggregation, raters, incentives,
anonymity
si.umich.edu
SCHOOL OF INFORMATION
UNIVERSITY OF MICHIGAN