Laboratory Experiments in
Economics: Coming of Age
Shyam Sunder, Yale University
Barcelona LeeX Experimental Economics
Summer School in Macroeconomics
Universitat Pompeu Fabra
Barcelona, June 12, 2012
A Discipline Grows Up
• Since Chamberlin reported the results of his classroom
experiment in 1948, the acceptability, recognition, role,
and methods of this sub-discipline have evolved
• Unlike 1970s and 80s, when editors of economics journals
routinely rejected experimental papers as a deviant
curiosity, a recent issue of AER has more papers using
experimental method than any other
• It is clear that the experimental method has grown beyond
its “childhood” phase, is no longer “outside the tent”
• Being inside the tent brings responsibilities of “adulthood”
for the sub-discipline?
4/10/2015
Design of Experiments
2
Responsibilities
• Identifying core concerns of the discipline on substantive, not just
methodological grounds
• Contribution to core concerns of a discipline
• Constructive interchange with sister methodologies of the discipline
• Balance between advancement of method and substantive knowledge
of real phenomena
• Five special concerns
– Time
– Institutions
– Properties of institutions
– Is the experimenter a part of the game; subject expectations
– Lab results as the final word, and main source of research
questions
4/10/2015
Design of Experiments
3
Core concerns of the discipline:
substantive, not just methodological
• Disciplines get sterile when methods take the front seat,
obscuring its classic or newly-identified substantive questions
• While methodological development is necessary part of a
healthy discipline, the dominant concern must still be with a
better understanding of our world we live in
• What proportion of the effort of the discipline goes into research
about questions about the world (external references), and
questions about research itself (internal references)?
• A simple test: try explaining your research question (and results)
to your parents, and assess if they appreciate your contributions
to human civilization
4/10/2015
Design of Experiments
4
Contribution to core concerns of a
discipline
• Where do we look for questions to address?
– On the street, news, and observation of the world
– Questions arising in the classroom (by students as well as in our
own minds) that we cannot answer to our satisfaction
– Unresolved (perhaps abandoned) puzzles of the discipline
– Incremental variations on recent publications
– Proving your advisor or academic god-parent right
• Identifying core concerns of economics that could not be
addressed without experiments
• What are the core concerns of economics and sister disciplines
such as psychology? Do we need to distinguish among them?
Does distinction mean un/willingness to learn from others?
4/10/2015
Design of Experiments
5
Constructive interchange with
sister methodologies
• Contribution of experimental method will
also depend on how well we are able to take
advantage of constructive interchange with
sister methodologies of economics
• For example, economic theory and
mathematical modeling
• What can be a constructive relationship
between theory and experiments?
4/10/2015
Design of Experiments
6
Assumptions
• Purpose of building models is to gain a better
understanding of some real phenomena of interest
• Real phenomena are complex (perhaps infinitely detailed),
rarely possible to understand/characterize them completely
• Theory identifies one or a few critical variables to gain a
satisfactory (not perfect) understanding of the phenomenon
of interest
• Theories are neither wrong nor right; some are more
helpful than others in gaining insight
• Compare theories on basis of their help in understanding of
the real phenomena
4/10/2015
Design of Experiments
7
Infinite Detail: Fractals
http://www.google.com/search?q
=fractals&hl=en&client=safari&
rls=en&prmd=imvns&tbm=isch
&tbo=u&source=univ&sa=X&ei
=jNDWT8LMIIzE8QPJlsCUAw
&ved=0CG0QsAQ&biw=1199
&bih=600
4/10/2015
Design of Experiments
8
Nature of Theory
• Essence of theory is its simplicity
• Simplification by abstraction from details of real
phenomena
• Assumptions perform the function of discarding
the mass of detail
• Key assumptions and assumptions of convenience
• Lack of correspondence between assumptions of
convenience and reality is the essence of a theory,
and not a defect of theory (no assumptions, no
theory)
4/10/2015
Design of Experiments
9
Empirical Test of Theory
• Theory is to real phenomena what a drawing or
stick figure is to human body, or map to earth
surface
• Correspondence is crude, but captures some
essential feature(s)
• Model identifies some tautologies which are
necessarily true when assumptions hold (unless
there are errors in derivation)
• What does it mean to empirically test a theory?
4/10/2015
Design of Experiments
10
Single Theory Experiments
• Only one interesting theory is available for the
phenomenon of interest
• “Test” is an assessment of robustness of the theory to
deviations from assumptions of convenience
• If data are gathered from an environment that corresponds
exactly to the assumptions of the theory, we should expect
no deviations (if we do observe deviations, either the
theory has error or the correspondence is missing)
• Empirical test is a costly method of discovering errors of
derivation
4/10/2015
Design of Experiments
11
Creating Theory in Lab
• Exact correspondence to theory in the field or lab
is not easy
• Even if we could, little could be learned from it
except about presence of errors of derivation or
correspondence
• Error in derivation or lab environment or data
collection
• Little useful scientific inference is possible from
such mutual lack of correspondence
4/10/2015
Design of Experiments
12
Scientific Value of A Singletheory Empirical Test
• Assessment of how robustly the predictions of the theory
correspond to data as the environment of data becomes less
similar to the convenience assumptions of the theory
• See Figure 1.
– A is not robust,
– B is highly robust, and
– C lies in between.
• Under all three cases, the model is literally true (when all
its assumptions hold). However, as the environment
deviates from the strict assumptions, A’s predictive power
declines sharply
4/10/2015
Design of Experiments
13
F ig u re 1 .
S in g le T h eo ry E xp erim en t
P e rc e n t C o rre s p o n d e n c e b e tw e e n th e D a ta a n d T h e o ry
P re d ic tio n
100%
B
C
A
0
0
D istance b etw een the M o d el and the D ata E nviro nm ents
4/10/2015
Design of Experiments
14
How Do We Identify Key the
Assumptions?
• Distinguishing between model and theory
• Model is a (“stick figure”) logical structure; a theory uses
the model to suggest some statements about the real
phenomena of interest
• Think of the real phenomena that motivates the model and
the theory
• Ask which assumptions of the model are intended to limit
the real environments sought to be understood
• Number of states, preferences, probability distributions
tend to be assumptions of convenience
4/10/2015
Design of Experiments
15
Design of Robustness
Experiments
• Conduct a series of experiments, all holding the key
assumptions, and progressively relaxing the convenience
assumptions (e.g., the number of states)
• Conduct a series of experiments progressively increasing
the number of alternative choices available to subjects
(increasing the number of possible outcomes)
• If model predictions are supported by data when more
alternatives are available, result is more robust;
e.g.,Vernon Smith (1962) paper
4/10/2015
Design of Experiments
16
Fig. 3: Single Theory Experiment
Smith (1962)
4/10/2015
Design of Experiments
17
F ig u re 2 : M u lti-T h e o ry E x p e rim e n ts
P re d ic tio n s
R e s u lts
4/10/2015
W
T heo ry 1
T heo ry 3
T heo ry 2
X
Y
Design of Experiments
Z
18
Fig. 4: Multi-Theory Experiment,
Plott and Sunder (1982)
4/10/2015
Design of Experiments
19
1. Time
• Most economic models include time dimension (usually
denoted by symbol t
• Few models specify what t represents in real terms—
seconds, hours, days, years, or generations
• Presumably, such theories are so general that they holds
for all interpretations of the time interval in real units
• Lab experiments could be a way of finding the appropriate
interpretations of time in specific theories, in case they
exist, and thus make a significant contribution of economic
theory
4/10/2015
Design of Experiments
20
2. Institutions
• Experimental methods have highlighted the importance of
economic institutions, their properties, and their evolution
over time
• However, study of institutions in lab presents a special
challenge
• Most individual decisions involve choice of a point on a
function, but institutions being functions themselves,
examination of their evolution calls for choices from a set
of functions
• Choice on a function and of a function call for very
different cognitive skills, experience, and time, and are
difficult to study in the few hours of a typical session
4/10/2015
Design of Experiments
21
3. Properties of Institutions
• Experiments have been employed to identify the
properties of institutions
• Real life institutions have great deal of detail, and
thus can be simplified for laboratory use in
thousands of ways
• When we try to use experiments to identify
institutional properties, how do we choose which
implementation of the institution in the lab is
appropriate?
4/10/2015
Design of Experiments
22
4. Experimenter as a part of the
game
• What are the boundaries of the game we
hypothesize the subjects to be playing?
• What do we know about the expectations subjects
bring to the lab? What, if any, control can we
exercise on their expectations
• Is experimenter inside that boundary or outside?
• How do we keep ourselves outside the boundary?
• Is it enough to tell them so?
4/10/2015
Design of Experiments
23
5. Lab results as the final word?
• When can we stop with the lab results,
convinced that we have a good
understanding of the phenomenon of
interest?
• When do we need to follow up the lab
results with data from the field?
4/10/2015
Design of Experiments
24
Fundamental Principle of Research
Designs (after Einstein)
• Research design should be as simple as
possible, but no simpler.
4/10/2015
Design of Experiments
25
Research Question
• What question do you wish to answer with your research?
• A question is one sentence with a question mark at the end
(?).
• It should be a question whose answer you would like to
know, but do not know
• After asking your friends, if you are the only one who does
not know, think again, unless you have reasons to disagree
with them
• What might the possible answers to the question?
• How could one distinguish what is a better answer?
4/10/2015
Design of Experiments
26
Macro-Examples
•
•
•
•
•
•
Robert Lucas and Edward Prescott
What is the question?
Why Experiment?
What is essential?
What is not essential?
Robustness check
4/10/2015
Design of Experiments
27