25 minute
presentation!
An Experimental Test of
Information & Decision Markets
Robin Hanson, Takashi Ishikida and
John Ledyard
Caltech
2/4/2005
Information Markets
• Standard Information Markets seem to work.
– Small but complete set of securities
– Many informationally small and unbiased traders.
• Theory and evidence from experiments and applications
are all positive.
• But all assume, require, use, …..
– Straight-forward behavior
• Price taking, honest revelation, etc.
– Complete set of state dependent contracts
– Common knowledge of all priors, …
• Even then we see “failures to fully aggregate information”
– Incomplete Bayesian updating
– Incompletely revealing Rational Expectations Equilibrium
DIMACS
2
Decision vs Prediction
• A policy maker does not just want to know the probability
that “terrorist attacks in the US will increase in 2005.”
• They want to know the probability that “terrorist attacks in
the US will increase in 2005” if “US troops remain in Iraq
for 2005.”
• With N events (attacks, troop size, …) and S outcomes for
each (increase from 10-20%, decrease, …), a complete set
of state dependent contracts requires N^S - 1 contracts.
S = 2, N = 8 => 255 contracts
DIMACS
3
Remember PAM?
Goal:
Collect accurate
predictive
information on
political and
economic
stability in the
middle east.
DIMACS
4
Remember PAM?
Every nation*quarter:
-Political stability
-Military activity
-Economic growth
-US $ aid
-US military activity
& all combinations
& ………
8 nations, 5 indices,
4 quarters
DIMACS
5
Remember PAM?
Every nation*quarter:
-Political stability
-Military activity
-Economic growth
-US $ aid
-US military activity
& all combinations
& ………
8 nations, 5 indices,
4 quarters (N = 180)
•Even if we only use up-down questions,
completeness requires 2^180 =1.5*(10^54) contracts.
DIMACS
6
Using Conditional Contracts
• The good news
– There may more overall trading.
• Traders may know more about and be more willing to trade on
the relatively more precise event {terrorism up | troops up} as
opposed to the less precise {terrorism up}.
• The bad news
– There may be thinner trading per security.
• Too many markets to pay attention to.
– Thinner trading => bad price discovery and incomplete
arbitrage => prices do not aggregate information.
DIMACS
7
Decision Markets
• Markets for Decision Analysis will be thin.
– Large and possibly incomplete set of securities
– Few informationally large and biased traders
• Theory is unlikely to be a good predictor of
behavior.
• Current applications and experiments may
not be applicable to the thinner situation.
• How can we know what will actually work?
DIMACS
8
Experimental Test Beds
• Create an environment that”captures” as much of the
problem as possible (the econ wind tunnel)
– Three traders, three events with 2 outcome each (8 states)
– Common prior with asymmetric information
• 10 draws from one urn of 6 equally likely => (1,0,1), (1,1,0),…..
• Each trader sees only two entries of each draw: (1,0,x), (1,1,x),…
• Run different mechanisms and market designs
• Measure performance
– How close are final prices to the fully informed posterior?
DIMACS
9
Theory Benchmarks - 3 events
priors
1
0.9
posteriors
uniform
0.8
0.7
CDF
0.6
3 Variable CDFs
0.5
Prior Distribution
0.4
0.3
Individual Posterior
0.2
Uniform Disribution
0.1
0
0
0.05
0.1
0.15
0.2
0.25
KL Distance from Group Posterior
DIMACS
0.3
0.35
0.4
10
Individual Scoring Rule
priors
1
posteriors
0.9
uniform
0.8
0.7
CDF
0.6
3 Variable CDFs
0.5
Individual (72)
0.4
Prior Distribution
0.3
Individual Posterior
0.2
Uniform Disribution
0.1
0
0.00E+00
5.00E-02
1.00E-01
1.50E-01
2.00E-01
2.50E-01
DIMACS
KL Distance from Group Posterior
3.00E-01
3.50E-01
4.00E-01
11
Standard Markets
priors
1
0.9
posteriors
uniform
0.8
0.7
CDF
0.6
3 Variable CDFs
0.5
Independent DA (24)
0.4
Individual (72)
0.3
Prior Distribution
0.2
Individual Posterior
Uniform Disribution
0.1
0
0
0.05
0.1
0.15
0.2
0.25
KL Distance from Group Posterior
DIMACS
0.3
0.35
0.4
12
Design Matters
• Asking is not enough.
• “Let there be markets” is not enough.
• Conjecture: An IM will work better in thin
situations, if we use (to “thicken” trading)
– Conditional contracts and
– a Combinatoric (package bid) Call Market
• Includes “no arbitrage” pricing but is intermittent
• Does not directly address “monopolistic agents”
DIMACS
13
Combinatoric Call Market
1
0.9
uniform
posteriors
0.8
0.7
CDF
0.6
3 Variable CDFs
0.5
Independent DA (24)
priors
0.4
Combined Value (22)
Individual (72)
0.3
Prior Distribution
0.2
Individual Posterior
Uniform Disribution
0.1
0
0
0.05
0.1
0.15
0.2
0.25
KL Distance from Group Posterior
DIMACS
0.3
0.35
0.4
14
Design Matters
• We are not yet at complete aggregation.
• Conjecture: An IM will work even better in
thin situations, if we use (to “thicken”
trading)
– Conditional contracts and
– A Combinatoric Sequentially Shared (Market)
Scoring Rule
• Is continuous and directly addresses report
manipulation
• But it involves a subsidy to traders.
DIMACS
15
Shared Scoring Rule - w/CC
1
0.9
posteriors
0.8
uniform
0.7
CDF
0.6
3 Variable CDFs
0.5
Independent DA (24)
0.4
Combined Value (22)
priors
Market Maker (8)
0.3
Individual (72)
Prior Distribution
0.2
Individual Posterior
Uniform Disribution
0.1
0
0
0.05
0.1
0.15
0.2
0.25
KL Distance from Group Posterior
DIMACS
0.3
0.35
0.4
16
Tentative Conclusions
• Standard markets and surveys do not work
will in thin situations.
• Using conditional contracts and assuming
some self - selection, either combinatoric
call markets or combinatoric sequentially
shared scoring rules significantly improve
performance over standard markets.
DIMACS
17
Open Questions
• There are many others we did not test
– Pari-mutuel mechanisms
• Economides, Lange, and Longitude (some
combinatorics)
• Pennock - Dynamic Pari-mutel Market
• Plott - Auction then Pari-mutuel
DIMACS
18
Open Questions
• There are many others we did not test
– Pari-mutuel mechanisms
• Economides, Lange, and Longitude (some
combinatorics)
• Pennock - Dynamic Pari-mutel Market
• Plott - Auction then Pari-mutuel
– Others
• HP • ……….
DIMACS
19
Some Open Questions
• There are many other mechanisms we did not test.
– Pari-mutuel mechanisms
• Economides, Lange, and Longitude (some combinatorics)
• Pennock - Dynamic Pari-mutel Market
• Plott - Auction then Pari-mutuel
– Others
• HP • ……….
• There are many other environments we did not test in.
– Information monopolist
– External incentives to manipulate internally
– And for PAM -- do these results survive in an ultra-thin world?
DIMACS
20
A Force 12 Storm
• Create an environment that really stresstests the mechanisms
– Six traders, 8 events w/ two outcomes each
(256 states)
– Common prior with asymmetric information
• 10 draws from one urn of 8! equally likely:
– (1,0,1,0,1,1,0,0), (1,1,0,0,0,0,0,0),…..
• Each trader sees only 4 different entries:
– (1,0,x,0,x,1,x,x), (1,0,x,0,x,1,x,x), …
DIMACS
21
1
posteriors
0.9
0.8
priors
0.7
8 Variable CDFs
CDF
0.6
0.5
Independent DA (18)
Combined Value (12)
0.4
Market Maker (17)
0.3
Individual (144)
Prior Distribution
0.2
uniform
Individual Posterior
Uniform Distribution
0.1
0
0
0.2
0.4
0.6
0.8
1
1.2
DIMACS
KL Distance
from Group Posterior
1.4
1.6
1.8
2
22
Summary of Testing
• Thin: 3 traders, 3 events
• 7 independent prices from 3 people in 12 minutes
• Markets < Individual Scoring Rule < Call < SSSR
• SSSR ~ Call given that the group beats the prior
– With selection, SSSR and Call Market do best.
• Ultra-Thin: 6 traders, 8 events
• 255 independent prices from 6 people in 12 min.
• Markets ~ Individual Scoring Rule ~ Call < SSSR
– SSSR beats the priors at the top (60%)
– Nothing else even beats the priors
– SSSR is only one with any aggregation
DIMACS
23
Final Thoughts
• Information Markets are possible and desirable.
– Can improve our ability to identify and deal with uncertainty.
• Many policy applications will be in thin situations.
• Traditional market designs do not work in thin
situations.
– Information monopolies, adverse decisions, partial updating
• The SSSR (w/conditionals) definitely sharpens the
signal/noise ratio in thin and ultra-thin markets over
traditional markets.
• Can we do better? Undoubtedly.
DIMACS
24
DIMACS
25