The Basics of Designing and Running an Economics Experiment
Economics 328
Spring 2005
The Purposes of Running an Experiment



Test a Theory
Gather Empirical Regularities to Inform a Theory
Test Institutions
What Makes a Good Experiment?

Should an experiment replicate reality?


Should an experiment replicate a formal model?


No! We have field studies for that.
No! We have theory for that.
A good experiment tries to capture the most relevant
features of reality in a simple, carefully controlled
environment. Good experiments are usually
designed to test specific hypotheses, sometimes
derived from the implications of some economic
theory and other times based on previous
observations in either experiments or field data.
What is a Controlled Experiment?

As much as possible, the experiment controls all
elements of the environment in which the experiment
takes place.



The experimenter determines the rules – what choices are
available to subjects, when decisions are made, and what
the consequences of these decisions will be.
The experimenter controls subjects’ payoffs as a function of
the actions they take.
The experimenter controls the information that is available
to subjects.
What is a Controlled Experiment?

Between treatments, the experimenter only changes variables which
are directly relevant to the hypothesis being tested, otherwise holding
the environment fixed.



Control vs. Treatment Sessions
Avoid Confounds (e.g. Don’t change more than one thing at a time.)
Variables that cannot be directly controlled are typically controlled via
randomization.
Example: Many experiments are designed to test subjects’ attitudes
towards fairness are affected by some treatment variable. To the extent
that subjects enter the lab with differing attitudes about fairness, a true
controlled experiment can’t be run. However, by randomly assigning
subjects to treatments, we can eliminate subjects differing attitudes as a
cause of differences between treatments. This relies on the law of large
numbers, implying that a large sample may be necessary.
Designing an Experiment


Identify an interesting question or questions. These should be issues that are
better addressed through a controlled experiment than through gathering field
data.
Determine hypotheses you would like to test.
Example: Increased payoffs lead to fewer mistakes in solving logic problems.
Example: Face-to-face interaction will lead to greater concern with fairness.

Design a simple environment that allows you to test the hypotheses you are
interested in. Remember, the more complicated the environment is, the more
likely you are to lose control. An experiment is not meant to replicate reality.
The reason we use controlled experiments is because reality is too complicated.
Brandts and Cooper:
The Weak Link Game

The employee payoff
function is given by the
following equation:

π  200  5E i  B  min
i
e



j1,2,3,4
Worker i's Payoff Table, B = 6
 E 
j
Having all five players
choose the same effort level
is a Nash equilibrium.
The critical variable here is
B, the bonus rate. Higher
values of B give greater
benefits to successful
coordination.
Trying to move to a higher
effort level is a risky
strategy.
Effort
by
Worker i
0
10
20
30
40
0
200
150
100
50
0
Minimum Effort by Other Workers
10
20
30
200
200
200
210
210
210
160
220
220
110
170
230
60
120
180
40
200
210
220
230
240
Worker i's Payoff Table, B = 14
Effort
by
Worker i
0
10
20
30
40
0
200
150
100
50
0
Minimum Effort by Other Workers
10
20
30
200
200
200
290
290
290
240
380
380
190
330
470
140
280
420
40
200
290
380
470
560
Brandts and Cooper:
Research Questions




Can an increase in the bonus rate enable a firm to overcome
coordination failure?
Does the magnitude of the bonus rate increase matter or is the
simple fact of an increase effective as such?
If an increase in the bonus rate brings about improved coordination,
can the bonus rate increase be revoked without affecting the
improved outcome?
Does the length of time a firm has been underperforming affect the
impact of the increase?
Designing an Experiment



Test your hypotheses by varying a small number of variables
within the experiment. If at all possible, you should not vary
more than one variable at a time.
Within vs. Between Subject Designs
Some important factors to remember . . .



Subjects get bored.
Subjects get confused.
Subjects learn as they gain experience within the experiment.
You should anticipate changes in their behavior (order effects).
Design for Brandts and Cooper
List of Treatments
Cell 1 Cell 2 Cell 3 Cell 4 Cell 5 Cell 6
Bonus Rate
Rounds 1 – 10
Bonus Rate
Rounds 11 – 20
Bonus Rate
Rounds 21-30
6
6
6
6
6
6
14
10
8
14
14
6
14
10
8
10
6
14
Results for Brandts and Cooper
Comparison of Treatments, Second Block
40
Average Minimum Effort
30
20
10
0
10
11
12
13
14
15
16
17
18
Period
Treatment 6/6/14
Treatment 6/8/8
Treatment 6/10/10
Treatment 6/14/X
19
20
Results of Classroom Experiment
Effect of Bonus Rate Increase
40
Average Minimum Effort
30
20
10
0
5
6
7
8
Round
B = 8 or B = 10
B = 14
9
10
Results of Classroom Experiment
Effect of Bonus Rate Increase
40
Average Effort
30
20
10
0
5
6
7
8
Round
B = 8 or B = 10
B = 14
9
10
Designing an Experiment


Avoid the use of deceit. This is a very tempting trap, but one that is
highly frowned upon by the profession. In particular, what is going to
happen the next time you try to run an experiment?
Anonymity: Most experiments guarantee subjects anonymity. In other
words, subjects are guaranteed that no other subjects (or indeed,
nobody other than the researchers) will be able to ever identify their
actions or payoffs. For some experiments, abandoning anonymity is an
important part of the design. If so, subjects should know what
information about them is be revealed publicly. Subjects should have
the option of withdrawing if they do not want information about them
revealed publicly.
Subjects

What population should you use as subjects?


How do you get subjects?


Undergraduates are the easiest to get, but subjects with relevant experience are often
more interesting to study.
Advertisements can be placed on posters, electronic bulletin boards, or in newspapers.
Spamming, while effective, is generally not a good way to make friends.
What should be in the advertisement?


It should include a very brief description, a summary of payoffs (average payoff is
sufficient), the time necessary to complete the experiment, and contact information.
(E-mail addresses work well. A website where people can sign up for the experiment
works even better.)
The advertisement should stress the monetary payoffs. Remember, you want
controlled experiments. This means that you want subjects who care about the
monetary payoffs being offered.
Play Games!!!
Earn Cash!!!
Participate in an Experiment in
Economic Decision Making!!!
Average Earnings = $20 - 25!!!
Takes 1½ Hours or Less!!!
Wednesday, September 4th, 4:30
Thursday, September 5th, 4:30
Monday, September 9th, 4:30
Tuesday, September 10th, 9:30
More Times Available
Sign up at :
www.weatherhead.cwru.edu/exper
Play Games!!!
Earn $$$
Subjects



Avoid unintentional selection of the subject population. For example, if you post
all of you ads in the weight room in Veale, your subject population will be
skewed towards men. You should also avoid unintentional clustering (e.g. don’t
sign up ten people from the same fraternity for a single session).
Have times and locations for sessions selected before posting any ads. Be
certain to schedule potential subjects as soon as they respond to your ad. Be
certain to get contact information from your subjects. Confirm that all subjects
will attend the day before the session. Even with confirmation, you should
anticipate that only about 80% of subjects will show up (CWRU students tend to
be pretty good about showing up).
Keep careful track of all contacts with potential subjects. You should never use
a subject twice for the same set of experiments. Most experimental groups
keep a black list – subjects who don’t show up or disrupt sessions are
undesirable.
Instructions




Most experiments include some form of consent form. For those of you who
plan on publishing your research, this brings up the issue of the dread IRB . . .
The overarching goal of instructions is to make certain that the subjects
understand the rules of the experiment. If subjects do not understand the
rules, you have immediately lost control.
Instructions should be as clear and complete as possible. Any critical points
should be repeated at least once. If in doubt, make the instructions too
detailed. Nobody likes sitting through lengthy instructions, but remember your
goal – the subjects need to understand the rules of the experiment.
Context: Many experiments like to frame their instructions in a generic context
that does not have any real world connotations. In my opinion, this often
generates confusion for experimental subjects, but that is a controversial point.
All experimenters agree that you should avoid any loaded terms.

Example: In a Prisoner’s Dilemma Game, you wouldn’t want to label the two
strategies as “Cooperate” and “Cheat” even though this is how game theorists often
refer to them.
Instructions

Demand Effects: A common flaw of experimental instructions is that they
suggest how the subjects ought to behave in a way that compromises the
experimental design. This should be avoided scrupulously. Sometimes demand
induced effects can be quite subtle.



Example: “The employees of a firm are always better off if all of them
spend more hours on Activity A.”
Example: All of the examples in your instructions have exactly the same
choices being made.
Payoff Quizzes: Payoff quizzes should be used to ensure that subjects
understand the relationship between their choices, the choices of others, and
their payoffs. Payoff quizzes also serve as a device for reinforcing important
points from the instructions. Take care to avoid demand effects in writing your
payoff quiz.
Instructions

Answering Questions: You should have subjects raise their
hands and tell an experimenter any questions privately. (Use
someone other than the person reading instructions.) This
limits the chance of losing control by having one subject sharing
his/her strategy with everyone in the session. Anything that
you think is substantive, you should repeat aloud to the entire
class. A common question is "Should I do blah, blah, blah . . ."
or "What should I do . . ." Do not answer these questions
directly, as this leads to demand effects. I always answer
something like, "We don't know what you should do. The whole
purpose of running the experiments is to see what you will do.
You should do whatever seems best to you." Nobody has
slugged me yet, although I imagine they have been tempted.
Actually Running the Experiment



Computers vs. Paper & Pencil Experiments
Run pilots in advance! Getting experimental subjects is hard work. You don't
want to waste a room full of subjects because there is a problem with your
procedures. Think everything out in advance and do some practice.
Setting Up the Room



Always set the room up in advance. Time is precious, so you don't want to waste time
moving desks or handing out papers.
Separate people who arrive at the same time. You generally don't want subjects
talking during the experiment, and people who arrive together are often friends.
Be certain that subjects can easily see you and any materials you might post in giving
out instructions. Make it as hard as possible for them to see the materials being filled
out by other subjects. Many established labs will have either dividers or sunken
terminals. For our purposes, get a big room and spread people out.
Actually Running the Experiment



Keep careful records of subjects' arrivals to the lab. I usually keep a checklist. I
usually will not allow any subject who is more than five minutes late participate
in the session (10 if I'm desperate). I sometimes have done "bully recruiting,"
but this is generally best avoided.
You should generally have one person who runs the experiment (a "conductor)
and a group of people who help out ("assistants"). The conductor is the front
man. He/she reads the instructions, gives any general instructions, and
oversees the experiment. The conductor need not have any particular authority
in reality – the goal is to have someone with a loud voice who will seem credible
to the subjects. The assistants do all the other myriad tasks involved in running
an experiment.
Both the conductor and the assistant should be careful to avoid any demand
effects. Do not lurk over the subjects while they are making decisions, do not
rush them if they are thinking about a choice, and don't talk among yourselves
unnecessarily during the session.
Actually Running the Experiment

You will want to have all forms made up in advance. In general, you
will need the following sorts of forms. Make up a checklist before the
experiment to make certain you don't forget anything.






Consent Forms (for publishable research)
Instructions
Payoff Tables: Make certain these are easy to read. Have these separate
from the instructions so subjects can easily refer to them.
Record Sheets: You generally want a sheet that allows subjects to record
all the information they have received, all the choices they have made, and
all the payoffs they have received. This both allows them to know the
history of their past choices and to confirm that you are paying them what
they are owed.
Play Sheets: Sheets where subjects record their actions.
Receipts
Actually Running the Experiment


Paying Subjects: You should pay subjects in cash as privately as
possible immediately after the end of the session. As much as
possible, you should calculate the payoffs yourself rather than
having the subjects do it. Be certain to get receipts!!! These
need the subject's name, SS#, the date, the amount paid, and a
signature.
Collecting Data: In designing your experiment, you should try
to make it as easy as possible to figure out afterwards what
happened. In particular, design the action sheets carefully and
store them neatly as the experiment progresses. Buying lots of
folders and labeling them in advance is a lot cheaper than
spending days deciphering a mess. Do not rely on subjects'
record sheets – these are often inaccurate.
Some Issues of Experimental Techniques



As much as possible, random number generation should be public and credible.
For pencil and paper experiments, I try to use devices people will understand –
rolling dice and flipping coins. Occasionally pre-generating the random numbers
is desirable, although I try to avoid this.
Many experiments involve rematching subjects. If they are not supposed to be
matched with the same person, you need to think out the matchings in advance
to make certain that they won't be rematched. This is easy for 1 x 1 matchings,
harder for group matchings.
Anonymity: For many experiments, we don't want subjects to know whom they
are playing. For most experiments, using ID numbers rather than names will be
good enough. For experiments with rematching, you may want to conceal the
ID numbers. Some experimenters use more extreme techniques. For most
cases, I think these are overkill.
Some Issues of Experimental Techniques



Risk Aversion: To limit the effects of risk aversion for experiments
where it is a nuisance variable, many experiments use a "lottery
technique." The value of this technique is dubious and it confuses the
living daylights out of subjects. You should be sensitive to the possible
effects of risk aversion on your results.
Feedback: Experimental results are quite sensitive to what feedback
subjects are given. Any sort of feedback is acceptable, just think about
its effects in advance and be certain that you aren't accidentally leaking
additional information. Often times the feedback is an important
component of the experimental design.
Asymmetric Information: Often times you want some information to
only go to some subjects. Be certain that this doesn't slip over into
deception. The other common problem is accidentally revealing
information to people who aren't supposed to have it.