Information Aggregation:
Experiments and Industrial
Applications
Kay-Yut Chen
HP Labs
Agenda
•
Lessons from HP Information Markets
(Chen and Plott 2002)
•
Scoring Rules and Identification of Experts
(Chen, Fine and Huberman 2004)
(Chen and Hogg 2004)
•
Public Information
(Chen, Fine and Huberman 2004)
Experimental Economics Program
HP Information Markets (Chen and Plott)
•
•
Summary of Events
–
12 events, from 1996 to 1999
–
11 events sales related
–
8 events had official forecasts
Methodology & Procedures
–
Contingent state asset (i.e. winning ticket pays $1, others $0)
–
Sales amount (unit/revenue) divided into (8-10) finite intervals
–
Web-based real time double-auction
–
15-20 min phone training for EVERY subject
–
Market open for one week at restricted time of the day (typically lunch and after hours)
–
Market size: 10-25 people
Experimental Economics Program
Event 2
0.3
0.25
IAM Distribution
P
0.2
Actual Outcome
0.15
0.1
HP Official
Forecast
0.05
IAM Prediction
0
0
100
200
$
Experimental Economics Program
300
Results
Abs % Errors of IAM Predictions
Last Interval Ignored
Event
2
3
4
5
6
7
8
9
Absolute %
errors of HP
forecasts
13.18%
59.55%
8.64%
32.08%
29.69%
4.10%
0.11%
28.31%
T-test P-value
Average
last 60%
trade
4.61%
57.48%
7.84%
30.93%
24.23%
7.33%
2.00%
23.85%
Average
last 50%
trade
4.57%
55.72%
8.15%
31.57%
24.54%
7.02%
2.35%
24.85%
Average
last 40%
trade
4.68%
54.60%
8.52%
31.83%
25.30%
6.71%
1.83%
24.39%
Average
last 60%
trade
5.63%
59.25%
6.45%
29.74%
22.94%
5.35%
1.53%
17.55%
0.079
0.084
0.071
0.034
Random variable x=official error – market error
H0: mean of x=0 Alternate:
mean of x>0
Experimental Economics Program
Last Interval Mass at
Lower Bound
Average Average
last 50% last 40%
trade
trade
5.68%
5.80%
57.46%
56.32%
6.77%
7.13%
30.33%
30.48%
23.22%
23.93%
4.91%
4.55%
1.39%
1.00%
17.32%
16.54%
0.026
0.022
Business Constraints and Research Issues
•
Not allowed to “bet” players’ own money -> stakes limited to an average of $50 per person
•
Time horizon constraints -> 3 months to be useful
•
Recruit the “right” people
•
Asset design affects the results (How to set the intervals?)
•
Thin markets (sum of price ~ $1.11 to $1.31 over the dollar)
–
Few players
–
Not enough participation
Experimental Economics Program
Reporting with Scoring Rule
Outcome
A
B
C
Reports of Probability Distribution
p1
p2
Pays C1+C2*Log(p3)
Experimental Economics Program
p3
Information Aggregation Function
If reports are independent, Bayes Law applies …
Ps | I  
p s1 p s2 ... p s N
p
s
s1
p s2 ... p s N
Experimental Economics Program
Two Complications
•
Non-Risk Neutral Behavior
•
Public Information
Experimental Economics Program
Dealing with Risks Attitudes:
Two-Stage Mechanism
Event 1
Stage 1: Information Market
Event 2
Event 3
Call Market to Solicit Risk Attitudes
Time
Event 4
Event 5
Event 6
Event 7
Stage 2: Probability Reporting & Aggregation
Individual Report of Probability Distribution
Nonlinear Aggregated Function
Event 8
Experimental Economics Program
Second Stage: Aggregation Function
Bayes Law with Behavioral Correction
1
1s
2
2s
N
Ns
N
Ns
p p ...p
Ps | I  
1  2
 p1s p2s ...p
s
Normalizing constant
for individual risks
 i=r(V i / i)c
Holding value/Risk
- measure relative risk of individuals
“market” risk
~sum of prices/winning payoff
Experimental Economics Program
Experiments:
Inducing Diverse Information
Outcome
A
B
Box of Balls
A
C
C
B
C
C
* In actual experiments, there are TEN states
Experimental Economics Program
Random Draws
Provide Info
Comparison To All Information Probability
Kullback-Leibler = 1.453
0.900
0.9
Omniscient
No Info
0.8
0.800
0.7
0.700
Probability
0.6
0.600
0.5
0.500
0.4
0.400
0.3
0.300
0.2
0.200
0.1
0.100
0.0
0.000
Series1
Series2
A
1
B
2
C
3
D
4
5
E
6 States
7
F
G
8
Experiment 4, Period 17
No Information
9
H
10
I
J
Kullback-Leibler Measure
•
Relative entropy
•
Always >=0
•
=0 if two distributions are identical
•
Addictive for independent events
Experimental Economics Program
Comparison To All Information Probability
Kullback-Leibler = 1.337
0.900
0.9
Omniscient
IA Mechanism
0.8
0.800
Probability
0.7
0.700
0.6
0.600
0.5
0.500
0.4
0.400
0.3
0.300
0.2
0.200
0.1
Series1
Series2
0.100
0.0
0.000
A
11
B
2 2 3
C
34
D
45
E
F
65 States
7 6
8
Experiment 4, Period 17
1 Player
G
H
I
J
79
8
10
9
10
Comparison To All Information Probability
Kullback-Leibler = 1.448
0.900
0.9
Omniscient
IA Mechanism
0.8
0.800
Probability
0.7
0.700
0.6
0.600
0.5
0.500
0.4
0.400
0.3
0.300
0.2
0.200
0.1
0.100
0.0
Series1
Series2
A
0.000
1
B
2
C
3
4
D
5
E
6
F
States
7
G
8
Experiment 4, Period 17
2 Players Aggregated
9
H
10
I
J
Comparison To All Information Probability
Kullback-Leibler = 1.606
0.900
0.9
Omniscient
IA Mechanism
0.8
0.800
Probability
0.7
0.700
0.6
0.600
0.5
0.500
0.4
0.400
0.3
0.300
0.2
0.200
0.1
Series1
Series2
0.100
0.0
A
0.000
1
B
2
C
3
4
D
5
E
6
F
States
7
G
8
Experiment 4, Period 17
3 Players Aggregated
H
9
10
I
J
Comparison To All Information Probability
Kullback-Leibler = 1.362
0.900
0.9
Omniscient
IA Mechanism
0.8
0.800
Probability
0.7
0.700
0.6
0.600
0.5
0.500
0.4
0.400
0.3
0.300
0.2
0.200
0.1
Series1
Series2
0.100
0.0
A
0.000
1
B
2
C
3
4
D
5
E
6
F
States
7
G
8
Experiment 4, Period 17
4 Players Aggregated
9
H
10
I
J
Comparison To All Information Probability
Kullback-Leibler = 0.905
0.900
0.9
Omniscient
IA Mechanism
0.8
0.800
Probability
0.7
0.700
0.6
0.600
0.5
0.500
0.4
0.400
0.3
0.300
0.2
0.200
0.1
Series1
Series2
0.100
0.0
A
0.000
1
B
2
C
3
4
D
5
E
6
F
States
7
G
8
Experiment 4, Period 17
5 Players Aggregated
H
9
10
I
J
Comparison To All Information Probability
Kullback-Leibler = 1.042
0.900
0.9
Omniscient
IA Mechanism
0.8
0.800
Probability
0.7
0.700
0.6
0.600
0.5
0.500
0.4
0.400
0.3
0.300
0.2
0.200
0.1
0.100
0.0
Series1
Series2
A
0.000
1
B
2
C
3
4
D
5
E
6
F
States
7
G
8
Experiment 4, Period 17
6 Players Aggregated
9
H
10
I
J
Comparison To All Information Probability
Kullback-Leibler = 0.550
0.900
0.9
Omniscient
IA Mechanism
0.8
0.800
Probability
0.7
0.700
0.6
0.600
0.5
0.500
0.4
0.400
0.3
0.300
0.2
0.200
0.1
Series1
Series2
0.100
0.0
A
0.000
1
B
2
C
3
4
D
5
E
6
F
States
7
G
8
Experiment 4, Period 17
7 Players Aggregated
H
9
10
I
J
Comparison To All Information Probability
Kullback-Leibler = 0.120
0.900
0.9
Omniscient
IA Mechanism
0.8
0.800
Probability
0.7
0.700
0.6
0.600
0.5
0.500
0.4
0.400
0.3
0.300
0.2
0.200
0.1
Series1
Series2
0.100
0.0
A
0.000
1
B
2
C
3
4
D
5
E
6
F
States
7
G
8
Experiment 4, Period 17
8 Players Aggregated
H
9
10
I
J
Comparison To All Information Probability
Kullback-Leibler = 0.133
0.900
0.9
Omniscient
IA Mechanism
0.8
0.800
0.7
0.700
Probability
0.6
0.600
0.5
0.500
0.4
0.400
0.3
0.300
0.2
0.200
0.1
0.100
0.0
0.000
Series1
Series2
A
1
B
2
C
3
4
D
5
E
F
G
6 States
7
8
9
Experiment 4, Period 17
9 Players Aggregated
H
10
I
J
Comparison To All Information Probability
0.9
Probability
0.8
Omniscient
0.7
IA Mechanism
0.6
market
0.5
Best Individual
0.4
0.3
0.2
0.1
0.0
A
B
C
D
E
F
States
Experiment 4, Period 17
G
H
I
J
KL Measures for Private Info Experiments
No
Information
Market
Prediction
Best Player
Nonlinear Aggregation
Function
1.977 (0.312)
1.222 (0.650)
0.844 (0.599)
0.553 (1.057)
1.501 (0.618)
1.112 (0.594)
1.128 (0.389)
0.214 (0.195)
1.689 (0.576)
1.053 (1.083)
0.876 (0.646)
0.414 (0.404)
1.635 (0.570)
1.136 (0.193)
1.074 (0.462)
0.413 (0.260)
1.640 (0.598)
1.371 (0.661)
1.164 (0.944)
0.395 (0.407)
Experimental Economics Program
Group Size Performance
Experimental Economics Program
Did the Markets Pick out Experts?
Group
Exp 1
Exp 2
Exp 3
Exp 4
Exp 5
Random
1.36
0.93
1.18
1.12
1.15
Payoff
1.45
1.09
1.24
1.13
1.39
Value
0.72
0.91
0.94
1.13
1.22
Optimal
0.53
0.72
0.75
0.83
0.77
•KL measure of all query data
•Pick groups of 3
Experimental Economics Program
Did Previous Queries Pick out Experts?
Group
Exp 1
Exp 2
Exp 3
Exp 4
Exp 5
Random
1.15
0.92
1.18
1.07
1.21
Query
0.78
0.89
0.71
0.92
0.81
Optimal
0.60
0.59
0.69
0.72
0.72
•KL measure of second half of query data
•Pick groups of 3
Experimental Economics Program
Public Information
•
Information observed by more than one
•
Double counting problem
Experimental Economics Program
Information Aggregation with Public Information
Kullback-Leibler = 2.591
1.2
1.2
120.00%
Probability
Probability
Probability
11
100.00%
0.8
0.8
80.00%
Omnicient
Omnicient
Omnicient
Public
Public
Public
IAM
IAM No Info
0.6
0.6
60.00%
0.4
0.4
40.00%
0.2
20.00%
0.2
000.00%
AA
B
AB
C
BC
D D E
C D
E E FF F
States
States
States
G
GG
HHH
Public Info Experiment 3, Period 9
11 Players Aggregated
III
JJJ
Dealing with Public Information:
Add a Game to the Second Stage
Event 1
Stage 1: Information Market
Event 2
Event 3
Call Market to Solicit Risk Attitudes
Time
Event 4
Event 5
Stage 2: Probability Reporting & Aggregation
Event 6
Event 7
Event 8
Individual Report of Probability Distribution
Matching Game to Recover Public Information
Modified Nonlinear Aggregated Function
Experimental Economics Program
Assumptions
•
Individuals know their public information
•
Private & Public Info Independent
•
Structure of Public Info Arbitrary
Experimental Economics Program
Matching Game
Outcome
A
B
C
Reports of Probability Distribution
Player 1: q1
q11
q12
q13
Player 2: q2
q21
q22
q23
Player 3: q3
.
.
.
q31
q32
q33
.
.
.
.
.
.
Choose player (3) by
Max (match function)
.
.
.
Player 1’s Payoff: (match function)*(C1+C2*Log(q33))
Match function: f(q1,q2)=(1-0.5*sum(abs(q1i-q2i)))^2
Experimental Economics Program
Matching Game
•
Any match function f(q1,q2) with property
–
Max when q1=q2
•
Multiple Equilibria
•
Payoff increases as entropy decreases
•
Hopefully, individuals report public information
Experimental Economics Program
Aggregation Function with
Public Information Correction
Bayes Law with a) Behavioral Correction
b) Public Info Correction
1
2
N
 p1s   p 2s   p Ns 
  
 ...

q1s   q 2s   q Ns 

Ps | I  
1
2
N
 p1s   p 2s   p Ns 
  
 ...


s  q1s   q 2 s 
 q Ns 
Normalizing constant
for individual risks
 i=r(V i /i)c
Holding value/Risk
- measure relative risk of individuals
“market” risk
~sum of prices/winning payoff
Experimental Economics Program
Public Information Experiments
•
5 Experiments
•
Various Information Structures
–
All subject received 2 private draws & 2 public draws
–
All subject received 3 private draws & 1 public draws
–
All subject received 3 private draws & half of the subjects
receive 1 public draws
–
All subject received 3 private draws & 1 public draws. 2
groups of independent public information.
•
9 to 11 participants in each experiments
Experimental Economics Program
Correcting for Public Information
Kullback-Leibler = 0.291
1.2
Omnicient
1
sim aggr
IAM
0.8
IAM (true public)
0.6
0.4
0.2
0
A
B
C
D
E
F
G
H
Public Info Experiment 3, Period 9
11 Players Aggregated
I
J
KL Measures for Public Info Experiments
Public
Info
Correction
Perfect
Public Info
Correction
No Info
Market
Prediction
Best
Player
Nonlinear
Aggregation
Function
2 draws
for all
1.332
(0.595)
0.847
(0.312)
0.932
(0.566)
2.095
(1.196)
0.825
(0.549)
0.279
(0.254)
2 draws
for all
2 draws
for all
1.420
(0.424)
0.979
(0.573)
0.919
(0.481)
2.911
(2.776)
0.798
(0.532)
0.258
(0.212)
3
3 draws
for all
1 draws
for all
1.668
(0.554)
1.349
(0.348)
1.033
(0.612)
2.531
(1.920)
0.718
(0.817)
0.366
(0.455)
4
3 draws
for all
1 draws
for half
1.596
(0.603)
0.851
(0.324)
1.072
(0.604)
0.951
(1.049)
0.798
(0.580)
0.704
(0.691)
1.528
(0.600)
0.798
(0.451)
1.174
(0.652)
0.886
(0.763)
1.015
(0.751)
0.472
(0.397)
Private
Info
Public
Info
1
2 draws
for all
2
Expt
5
3 draws
for all
Two
groups
of
public
info
Experimental Economics Program
Summary
•
IAM with public info correction did better than best
person.
•
IAM with public info correction did better than markets in
4 out of 5 cases.
•
IAM corrected with true public info did significant better
than all other methods.
Experimental Economics Program
Experimental Economics Program
or
ab
ov
e
-$
11
62
.9
m
-$
11
47
.3
m
-$
11
33
.3
m
-$
11
20
.2
m
-$
11
07
.7
m
-$
11
00
.4
m
-$
10
93
.2
m
-$
10
86
.0
m
-$
10
78
.8
m
-$
10
71
.6
m
-$
10
64
.3
m
-$
10
51
.8
m
-$
10
38
.7
m
$1
16
2.
9m
$1
14
7.
3
$1
13
3.
3
$1
12
0.
2
$1
10
7.
7
$1
10
0.
4
9.
1m
-$
10
24
.7
m
-$
10
0
Actual Value
$1053m
$1
09
3.
2
$1
08
6.
0
$1
07
8.
8
$1
07
1.
6
$1
06
4.
3
$1
05
1.
8
$1
03
8.
7
$1
02
4.
7
$1
00
9.
1
$0
Implied Probabilities of Revenue Bins, September 2003
35%
Official
Projection
30%
25%
20%
15%
10%
5%
0%
Implied Probabilities of Operating Profit Bins, September 2003
70%
Official
Projection
Actual Value
$113m
60%
50%
40%
30%
20%
10%
0%
$0 $37.1m
$37.1 - $46.1m $46.1m $54.4m
$54.4 $62.0m
$62.0 $69.3m
$69.3 $73.6m
$73.6 $77.8m
$77.8 $82.0m
$82.0 $86.2m
$86.2 $90.4m
Experimental Economics Program
$90.4 - $94.7 - $102.0 - $109.6 - $117.9 - $126.9m
$94.7m $102.0m $109.6m $117.9m $126.9m or above
Supplementary
Experimental Economics Program
Previous Research
•
Academic Studies
–
Information Aggregation in Markets
• Plott, Sunder, Camerer, Forsythe, Lundholm, Weber,…
–
Pari-mutuel Betting Markets
• Plott, Wit & Yang
•
Real World Applications
–
Iowa Electronic Markets
–
Hollywood Stock Exchange
–
HP Information Markets
–
Newsfuture
–
Tradesport.com
–
…
Experimental Economics Program
Risk Attitudes
1.000
0.900
0.800
0.700
0.600
Risk Loving
0.500
Risk Neutral
0.400
0.300
Risk Averse
0.200
0.100
0.000
A
B
C
D
E
F
G
H
Experimental Economics Program
I
J
Dealing with Risks Attitudes:
Two-Stage Mechanism
Event 1
Stage 1: Information Market
Event 2
Event 3
Call Market to Solicit Risk Attitudes
Time
Event 4
Event 5
Event 6
Event 7
Stage 2: Probability Reporting & Aggregation
Individual Report of Probability Distribution
Nonlinear Aggregated Function
Event 8
Experimental Economics Program
Probability Reporting
Outcome
A
B
C
Reports of Probability Distribution
p1
p2
Pays C1+C2*Log(p3)
Experimental Economics Program
p3
Second Stage: Aggregation Function
Bayes Law with Behavioral Correction
1
1s
2
2s
N
Ns
N
Ns
p p ...p
Ps | I  
1  2
 p1s p2s ...p
s
Normalizing constant
for individual risks
 i=r(V i / i)c
Holding value/Risk
- measure relative risk of individuals
“market” risk
~sum of prices/winning payoff
Experimental Economics Program
Private Information Experiments
•
5 Experiments
•
Various Information Conditions
•
–
All subject received 3 draws
–
Half received 5 draws, half received 1 draw
–
Half received 3 draws, half received random number of draws
8 to 13 participants in each experiments
Experimental Economics Program
Next Step
•
Field Test (Fine and Huberman) …
Experimental Economics Program