The Experimental Economist
Chris Starmer
TSU Short course in Experimental and Behavioural
economics, 5-9 November 2012
Aims for this Session
• To explore
– What are experiments?
– What are they for?
– What can they tell us about
the economic world?
• Will draw on two
‘demonstration’ experiments
• But first…….
2
Folk wisdom
• Part of the folk law of C20th economics:
– Economics a non-experimental discipline
Sceptic #1
"unfortunately, we can seldom test
particular predictions in the social
sciences by experiments explicitly
designed to eliminate what are
judged to be the most important
disturbing influences."
M. Friedman (1953)"The Methodology of Positive
Economics".
Sceptic #2 - Richard Lipsey
“It is rarely, if ever,
possible to conduct
controlled experiments
with the economy.
Thus economics must
be a non-laboratory
science.”
(Introduction to Positive
Economics, 1979)
Enthusiast
“as currently practised,
economics is ideally suited for
experimental investigation”
John Hey (1991), Experiments in
Economics.
Recent Trends
• Experimentation practised world-wide e.g.
– USA: Caltech, Harvard, Princeton, UCLA......
– Europe: Germany, Netherlands, France ,Italy,
UK
• Textbooks (1990s)
• Journal of Economic Literature
– Classification (late 1980s)
• Econometrica Guidelines
– Palfrey and Porter, Econometrica, 1991.
• Experimental Economics (Journal - 1998)
What is an Experiment?
Let’s
Explore!
Demo Experiment 1
A (very) Simple Auction Market
• Run two auctions
• Each auction – I sell a
£1 Coin to the highest
bidder (£1=2.5GEL)
• Different auction rules
• I tell students….One
auction is for real
– You only find out
which one, after the
two auctions have
ended
10
Selling money – Auction 1
• Going to sell a £1
coin
• “English auction”
• The coin goes to the
highest bidder
What do you
think happens?
Do people bid?
How much do I
sell the coin
for?
11
Selling money - Auction #2
• As before - the coin
goes to the highest
bidder
– And they pay their
bid
What do you think
happens?
Do people bid?
• Additional Rule:
– the second-highest
bidder also has to
pay the amount of
his/her last bid– but
they get nothing in
return.
How much do I sell
the coin for?
12
Why have I shown you this?
13
Illustrates some typical
features of experiments
Feature 1
• Experimenter creates decisions for
people to make
• Decisions are usually REAL
– There is MONEY on the table
– Participants can gain (and sometimes
lose) as consequence of their decisions
Feature 2
• Experimenter often interested in what
happens when you change some
aspect of the decision
– In this case, I changed the auction rule
(who pays)
• The two auctions are ‘TREATMENTS’
with different ‘CONDITIONS’
Feature 3
• A good experiment has a well-defined
purpose
• Treatment comparison may be intended
to :
– Test a Theory
– Explore a Hunch
• Experimental design
– should normally be directed to a well formulated research question
Feature 4
• Demo experiments show it is not
necessarily hard to run an experiment
– Some are very simple to implement
– Some are of course more complicated
• It is harder to run GOOD experiments
– Carefully designed to shed light on welldefined and worthwhile research
questions
Experiment 2: Mind Reading!
Instructions
• Imagine - You are paired with one other person
• Answer the questions on the next slide with the
aim of trying to give the same answer as the
person you are paired with.
• You should do this on your own and don’t try to
communicate in any way.
 You score a point for every answer that
matches the one given by the person you are
paired with.
19
Here are the questions:
1. Name any flower:
2. Name any car manufacturer:
3. Write down any positive number:
4. Write down any colour:
5. Write down one of the words: ‘Heads’ or ‘Tails’
Write your answers down PRIVATELY in ENGLISH on a
piece of paper
20
Session 2 - Part II
So what is an experiment?
• An environment created and/or
manipulated by an experimenter
• For purpose of
– observing some variable (X) of interest
In econ experiment, usually some kind of
(economically relevant) human decisions
- while controlling possible influences on X
- Note controlling a variable might mean
a variety of things including:
- Holding it constant, varying it
systematically
22
Some Basic Design Principles
General aim
to explore relationships
between variables of
economic interest
Multiple influences
economic variables,
typically affected by
multiple influences in
“the wild”
y = f(x1, x2………xn)
• Data generated in the
wild
– might investigate
statistically
– (e.g. econometrics)
• Alternatively
– may investigate
relationships
experimentally
– i.e., try to ISOLATE
effects of particular23
variables
Control
Suppose we believe:
y = f(x1, x2,………, xn)
Wish to explore relation
between y and x1
So, construct an
environment in which...
1) Observe y
2) Vary the level of x1
3) ‘Controlling’ effects of other
variables
e.g. Holding constant:
x2……..xn
Hence ISOLATE effect of x1
on y
24
Some purposes of experiments
• Testing predictions of theory
• Looking for ‘surprising’ regularities
• Designing and Testing institutions…..
The Wind Tunnel Analogy
The Biggest Auction Ever
K. Binmore and P.
Klemperer, 2002.
"The Biggest Auction
Ever: The Sale of the
British 3G Telecom
Licences," Economic
Journal, 112, 2002.
• Advised UK Govt on design
of auctions
• Sale of 3G spectrum
licences
• Tricky theoretical problem
• Auctions ran in spring 2000
• Raised £22.5 Billion
– 2.5% GNP
Success and Failure
• Sometimes experimental results confirm
predictions of economic theory
– This itself can be a surprise if the
predictions seem surprising or rest on
strong assumptions
• See day 5 on Markets
• Sometimes results challenge economic
theory:
– see tomorrow on “preference reversal”
28
Programmes of Research
• Surprising experimental results typically
generate further research
– Experiments to test robustness of
experimental findings
• E.g. did subjects understand the instructions?
– New theories or conjectures to explain
surprising results
29
Back to mind reading
experiment
What question(s) was that
experiment designed to investigate?
30
Here are the questions:
1. Name any flower:
2. Name any car manufacturer:
3. Write down any positive number:
4. Write down any colour:
5. Write down one of the words: ‘Heads’ or ‘Tails’
Write your answers down PRIVATELY in ENGLISH on a
piece of paper
31
Col
Heads Tails
Heads
Prize
No Prize
No Prize
Prize
Row
Tails
Fully rational agents can’t ‘solve’ this game
ordinary people seem pretty good at it
T. Schelling - 86% of folks choose heads
32
Investigating Coordination
• Coordination games are common in
world
• Experiment, as described, investigates:
– Can people coordinate better than
chance/rationality predicts?
• Looking for a regularity in behaviour
• If there, current theory can’t explain it
– Perhaps more theory and/or experiment
can help us understand it better?
33
Mehta, Starmer and Sugden,
American Econ. Review (1994)
Design
 20 similar tasks
 Subjects randomised between two conditions.
 Coordinators (“motivated”)
 motivated to try to coordinate and paid according to
success
– Pickers (“unmotivated”)
• ‘rewarded’ so long as all questions answered – no
coordination motive
Col
Heads Tails
Heads
Prize
No Prize
No Prize
Prize
Row
Tails
Fully rational agents can’t ‘solve’ this game
Yet ordinary people seem pretty good at it
T. Schelling (1960) Strategy of Conflict, 86% of folks choose heads
What explains coordination
success of ordinary people
• MSS consider three possible hypotheses
– Primary Salience
– Secondary Salience
– Schelling Salience
• MSS Experiments designed to
discriminate between them
Primary Salience
• Some labels ‘stick out’ more than others
– they come to mind, more easily, for what
ever reason
– Call these the ‘Salient’ labels
• Primary Salience Hypothesis
– some people have a tendency to choose
alternatives with salient labels
– maybe a non-rational propensity
Secondary Salience
• In a two player game, I act according
to secondary salience if:
– I try to guess what has primary salience
for you
– I assume you choose that
– I optimise relative to that prediction
Schelling Salience
• Primary and secondary assume ‘seed’ of nonrational behaviour
• Schelling suggests a theory of rational
coordination
– view coordination as a ‘team’ activity
– try to identify a rule which, if followed by
everyone, would produce a good outcome for the
group
– if you can find such a rule, follow it!
Predictions
• Primary
– no difference between groups
• Secondary
– S’s motivated to coordinate - answers
more concentrated on options most
popular among unmotivated groups
• Schelling
– modal choice may vary between groups
Results - Task 1 Flower
Unmotivated
Rose
35.2
Daffodil
13.6
Daisy
10.2
Tulip
9.1
Motivated
Rose
67.2
Daisy
13.3
Daffodil
6.7
Coord ind: 0.184
#resp types: 26
Coord ind: 0.447
#resp types: 11
Task 2 - Car Manufacturer
Unmotivated
Ford
53.4
Rover
5.7
Motivated
Ford
81.1
Coord ind: 0.290
#resp types: 22
Coord ind: 0.659
#resp types: 11
Task 3 - Positive Number
Unmotivated
7
11.4
2
10.2
10 5.7
{1
4.5}
Motivated
1
40.0
7
14.4
10 13.3
2
11.1
Coord ind: 0.052
#resp types: 28
Coord ind: 0.206
#resp types: 17
Conclusions
• More than primary salience!
Some Observations
• Motivated by specific hypotheses
• Subjects allocated to groups
–
–
–
–
groups face decisions
different ‘conditions’ (decision environments)
incentives key to setup
hypotheses predict different behaviour in the two
groups
• Randomisation of subjects to groups
– not just differences between individuals
• Mixed success in answering research
questions
– good on primary vs other
– not so good on Secondary vs Schelling
• So, as is typical, more to do.........
Should I Believe Results of
Experiments?
46
Should I Believe Results of
Experiments?
• One standard query
• Most experiments observe simple and
abstract decisions of students in unusual
decision environments making decisions
for small sums of money – do such data
tell us much about how the economy
works?
• A question of external validity
47
Possible responses on external
validity
1. Theory testing:
Theories of economic phenomena are
much simpler than the phenomena
themselves
If I want to test a theory in an
experiment, it can be an advantage to
make the experiment simple like the
theory
48
Theory, Experiment and World
LAB
EXPERIMENTS
Theory
Economic
Phenomena
“in the wild”
Increasing complexity
Possible responses on external
validity
2. If you are worried that an experiment
leaves out something important to the
real world phenomena……….
Make the experiment more like the target
phenomenon
50
Theory, Experiment and World
LAB
EXPERIMENTS
Theory
FIELD
Economic
EXPERIMENTS Phenomena
“in the wild”
Increasing complexity
Questions?
52