7/8/14
1
Lab Experiments in the Field
Capetown
PODER
OUTLINE FOR TODAY:
Leftover from yesterday
 Multi-site experiments: Solidarity
7/8/14

Capetown
2
NUTS AND BOLTS
7/8/14
Capetown
3
SECOND STEPS:
(AFTER THE QUESTION/THEORY)
Instrumentation
7/8/14

Construct Validity – how will I test what I want to
test?
 Paper/Pencil or Computer?
 Timeline of experiment
 Instructions

Capetown

Sampling/Randomization
What subject pool?
 How will Treatment be randomized?


Analysis Plan
What are the units of analysis
 Power tests

4
SUBJECT SELECTION, I


Convenient, inexpensive and relatively homogeneous

May behave differently from target population, young,
educated, and talk to each other (diffusion)
Capetown

students
7/8/14
 Convenience Samples:
 Students advantages:
Student disadvantages:
Classroom:
Representative sample of students
 Environment might affect behavior:


Lab:
May select certain students
 Neutral environment


Data: Eckel and Grossman ExEc:
Students give more to charity in the classroom than in
the lab
 Why?

5
SUBJECT SELECTION, II
Groups:
7/8/14
 Specialized
Elderly
 Professionals
 Medical cases
 Poor
 Residents of hurricane-vulnerable areas
 Public officials
 Population Samples


Pluses: External validity, Heterogeneity
Minuses: Costly, risk of decreased control,
heterogeneity
Capetown

6
SUBJECT SELECTION III
Subject selection should suit the question you are
asking
 Theory testing:

Independent of subject characteristics?
Policy (measurement or institutional design):


Capetown

7/8/14

Target group subjects
Examples:
WEIRD people (Henrich, et al. 2010)
 People from other cultures (Barr and Serra 2010)

7
SECOND STEPS:
(AFTER THE QUESTION/THEORY)
Instrumentation
7/8/14

Construct Validity – how will I test what I want to
test?
 Paper/Pencil or Computer?
 Timeline of experiment
 Instructions

Capetown

Sampling/Randomization
What subject pool?
 How will Treatment be randomized?


Analysis Plan
What are the units of analysis
 Power tests

8
NUTS AND BOLTS, I
Lab log.
 IRB and Ethics
 Pilot experiments.
 Lab set-up
 Subject registration
 Experimenter(s)
 Monitor(s)
 Randomizing Devices
 Instructions
 Subject confidence (non-deception)
7/8/14

Capetown
9
LAB BOOK (LUPIA & VARIAN 2010)








Capetown

1. State your objectives.
2. State a theory.
3. Explain how focal hypotheses are derived from the theory if the
correspondence between a focal hypothesis and a theory is not 1:1.
4. Explain the criteria by which data for evaluating the focal hypotheses were
selected or created.
5. Record all steps that convert human energy and dollars into datapoints.
6. State the empirical model to be used for leveraging the data in the service of
evaluating the focal hypothesis. (a) All procedures for interpreting data
require an explicit defense. (b) When doing more than simply offering raw
comparisons of observed differences between treatment and control groups,
offer an explicit defense of why a given structural relationship between
observed outcomes and experimental variables and/or set of control variables is
included.
7. Report the findings of the initial observation.
8. If the findings cause a change to the theory, data, or model, explain why the
changes were necessary or sufficient to generate a more reliable inference.
9. Do this for every subsequent observation so that lab members and other
scholars can trace the path from hypothesis to data collection to analytic
method to every published empirical claim.
ELNs: OneNote in Microsoft or Growlybird Notes for the Mac
(http://www.growlybird.com/GrowlyBird/Notes.html)
7/8/14

10
NUTS AND BOLTS, I
7/8/14
Capetown
Lab log.
 IRB and Ethics
 Pilot experiments.
 Lab set-up
 Subject registration
 Experimenter(s)
 Monitor(s)
 Randomizing Devices
 Instructions
 Subject confidence (non-deception)

11
ETHICS
IRB keeps us honest (some countries don’t have)
7/8/14

Focus on potential harm to subjects
 Consent, debriefing limit harm, but may impact sample
 Balance between potential benefit and risk

Capetown

Field experiments:





No consent process! Unwitting subject, high potential cost
Findley et al 2014. – no consent, no debriefing
Correspondence studies on discrimination (more later)
Intervention studies: elections, political institutions
Facebook study on emotional contagion: no consent,
potential risk, very low potential benefit
12
NUTS AND BOLTS, I
7/8/14
Capetown
Lab log.
 IRB and Ethics
 Pilot experiments.
 Lab set-up
 Subject registration
 Experimenter(s)
 Monitor(s)
 Randomizing Devices
 Instructions
 Subject confidence (non-deception)

13
NUTS AND BOLTS, II
7/8/14
Capetown
Subject questions
 “Learning periods”
 Experiment
 Recording data
 Termination of experiment
 Debriefing
 Subject payment
 Bankruptcy
 Backup plan

14
WRITEUP AND REPORTING
7/8/14
Capetown
Biases in published data
 Registration and CONSORT

15
BIASES IN PUBLISHED DATA
7/8/14
Capetown
Selective reporting + publication bias => many
published studies have p=.05.
 Data mining and selective presentation of results
have been a concern in economics for a long time
 These concerns are not limited to Economics:

Medical trials, Ioannidis (2005, “Why most published
research findings are false”)
 Psychology, Simmons et al. 2011, “False - positive
psychology: Undisclosed flexibility in data collection and
analysis allows presenting anything as significant”)
 Political science, Humphreys et al. (2012, “Fishing”)


Finds affect millions of people. How to fix?
16
EXAMPLE: (GERBER AND MALHOTRA, AJPS)
7/8/14
Capetown
17
ECONOMICS, “STAR WARS” (BRODEUR ET
AL 2013)
7/8/14
Capetown
18
CONSORT/REGISTRATION (HUMPHREYS
ET AL, 2013)

http://www.consortstatement.org/
Capetown
CONSORT
Statement: improve
the reporting of a
randomized controlled
trial (RCT), enabling
readers to understand
a trial's design,
conduct, analysis and
interpretation, and to
assess the validity of
its results.
7/8/14

19
REGISTRATION
Benefit:

Capetown
Limits selective reporting/fishing
 Rounds out “body of evidence”
 Forces researcher to think through design, statistical
analysis

7/8/14

Potential costs
Limits exploratory research
 Serendipitous findings may be hard to publish


But: frees it from the burden of (false) presentation
as formal hypothesis testing.
20
GENERAL REMARK: LAB V. FIELD
Capetown
has greater internal validity
 Lab is cheap, field is costly
 Lab mistakes can be fixed; often not
so in field
 Students v. population
7/8/14
 Lab
Population has higher variance,
harder to detect effects
 Selection bias is not limited to lab
 Greater monitoring costs to ensure
population sample

21
IF TIME, HERE ARE UDT SLIDES
7/8/14
Capetown
22
THE ULTIMATUM GAME

Task:



Two players must divide a fixed amount-$100
Game is played once only
Players
Proposer- chooses a split
 Respondent-accepts or rejects


Payoffs:
If accepted, money split as planned
 If rejected, both players get zero


Game theory:
Start with responder. Payoff-maximizer accepts
anything >0
 Therefore proposer offers smallest possible amount

$5 ULTIMATUM GAME
(ECKEL AND GROSSMAN, RUN IN 1992, PUB. 2000)
ULTIMATUM
GAME RESULTS
Unequal offers are rejected
 Payoff-maximizing offer is modal offer is 60/40 split
 Most common split is 60/40
 Looks as if proposers are “rational”
 What about responders?


(Why do they reject?)
BEHAVIOR
MODEL:

IS INCONSISTENT WITH STANDARD
Why do people offer >0?
Value other’s payoff (Altruism)
 Afraid to be rejected (lose it all)



Risk averse?
Why do people reject?

Fairness: think the distribution is not fair overall

Note we see people rejecting really high offers, too.
Selfish fairness: care about own relative payoff
 Low offer means cost of punishment is low

THE DICTATOR “GAME”

Task:


two players must divide a fixed amount-$100
Players
Proposer- chooses a split
 Respondent-passively accepts

Dictator game allows player 1 to make an
altruistic allocation
 We use this game to measure altruism

REAL PEOPLE PLAY DICTATOR GAME :
HISTOGRAM OF DECISIONS
Giving in the Dictator Game
0.7
Percent of Subjects
0.6
0.5
0.4
0.3
0.2
0.1
0
10
9
8
7
6
5
Amount Kept
4
3
2
1
DICTATOR
RESULTS/COMPARISON
2/3 of subjects are selfish
 Many give away something – altruistic
 Measure of altruism!

Varies across individuals
 Reliable
 Correlated with behavior like volunteering


Different conditions lead to different outcomes
“Double blind” (see prev. slide)
 Identity
 Charity

MEN
AND WOMEN
–
DICTATOR
Men
donate
$0.82
Women
donate
$1.60
Source: Eckel and Grossman, Economic Journal, 1998
TRUST GAME
Player A and Player B both begin with $100.
(important!)
 Player A decides how much, if any, to send to
Player B.


Any amount sent is tripled on the way to B.
Player B decides how much, if any to send back
to Player A.
 Game theory: B returns 0, so A sends 0
 This game is used to measure trust and
reciprocity.

TRUST GAME RESULTS:
7/8/14
Capetown
People send positive amounts
 Trust (just) pays on average
 In the field, higher levels of trust and reciprocity
 Measure of individualized trust
 (More on this later).

32
TESTING THEORY V. MEASUREMENT

These games violate predictions of standard
game theory (assuming payoff-maximizing
agents)


Led to a huge amount of research (experimental and
theoretical)
These games are useful for measuring
preferences and social norms
LAB EXPERIMENTS IN THE
FIELD
34
Lab or field? False dichotomy
 Lab experiments complement field exps.

Lab experiments in the field (extra-lab) for
measurement, etc.
 Lab experiments back home to settle methodological
issues that arise in the field
 Pretest and refine experimental designs before going
into the field


Issues:

External validity (perpetually)
7/8/14
Capetown
NEXT: SOLIDARITY
35