Designing and Conducting Experiments
Rick K. Wilson
Department of Political Science
Rice University
Catherine C. Eckel
Economics
Texas A&M University
Course Objective:
This course develops the role of experimental methods in the social sciences.
Experiments are one of the tools available for research in the social sciences, one that
plays an increasingly prominent role in economics and political science.
There are three goals for the course. First is to illustrate the connection between theory
and experimental design in both directions: the use of experimental designs to test
theories directly in the lab, and the impact of experimental results on the development
of theory. Second, we will discuss the elements of experimental design, and what makes
a good experimental study: the simplest design is often the best. Finally, we want to
provide a sense of the kinds of problems – in design, implementation, and data analysis
– facing various types of experiments (from lab experiments, to lab-in-the-field
experiments, to field tests and randomized control trials).
To learn about experiments, there is no substitute for hands-on experience, so
participants will work individually or in groups of up to size four to design an
experiment that will be presented in the workshop. On the last day of class students will
present their experimental designs, with feedback from the instructors and other
students, and time permitting some groups may be able to conduct a pilot of their
design.
COURSE OUTLINE: READINGS
Note: *Papers that should be closely read beforehand.
Monday – January 11th
Introduction to Experimental Design
OBJECTIVES: Experimental Design in the social sciences.
*Friedman, Daniel, and Shyam Sunder. Experimental Methods : A Primer for Economists.
Experimental Design – p. 2
London: Cambridge University Press, 1994. Chapters 1-3
*McDermott, R. (2013). "The Ten Commandments of Experiments." PS: Political Science
and Politics 46(3): 605-10.
*Croson, Rachel, and Simon Gachter. "The Science of Experimental Economics." Journal of
Economic Behavior and Organization 73, no. 1 (2010): 122-31.
Interesting new directions:
Berinsky, Adam J., Gregory A. Huber, and Gabriel S. Lenz. "Evaluating Online Labor Markets
for Experimental Research: Amazon.Com's Mechanical Turk." Political Analysis 20
(2012): 351-68.
Brandts, Jordi, and Gary Charness. "The Strategy Versus the Direct-Response Method: A
First Survey of Experimental Comparisons." Experimental Economics 14 (2011):
375-98.
Charness, Gary, Uri Gneezy, and Michael A. Kuhn. "Experimental Methods: Between-Subject
and within-Subject Design." Journal of Economic Behavior and Organization 81, no. 1
(2012): 1-8.
Harbaugh, William T., Ulrich Mayr, and Daniel R. Burghart. "Neural Responses to Taxation
and Voluntary Giving Reveal Motives for Charitable Donations." Science 316, no. 15
June 2007 (2007): 1622-25.
Canonical Games.
OBJECTIVES: Classic experimental games
* Eckel, Catherine C. "Economic Games for Social Scientists." In Laboratory Experiments in
the Social Sciences, edited by Murray Webster and Jane Sell, 335-55. London:
Academic Press, 2014.
For some of the original articles, see:
Isaac, R. M. and J. M. Walker. 1988. "Group Size Effects in Public Goods Provision: The
Voluntary Contributions Mechanism." Quarterly Journal of Economics 103 (1): 17999.
Berg, Joyce E., John W. Dickhaut, and Kevin McCabe. "Trust, Reciprocity, and Social History."
Games and Economic Behavior 10, no. 1 (1995): 122-42.
Experimental Design – p. 3
Eckel, C. C., and P. J. Grossman. "Altruism in Anonymous Dictator Games." Games and
Economic Behavior 16, no. 2 (1996): 181-91.
Fehr, Ernst, and Simon Gächter. "Cooperation and Punishment in Public Goods
Experiments." American Economic Review 90, no. 4 (2000): 980-94.
Cameron, Lisa A. "Raising the Stakes in the Ultimatum Game: Experimental Evidence from
Indonesia." Economic Inquiry 37, no. 1 (1999): 47-59.
Ostrom, Elinor, James Walker, and Roy Gardner. "Covenants with and without a Sword:
Self-Governance Is Possible." American Political Science Review 86, no. 2 (1992):
404-17.
Hands on – brainstorming and group production
OBJECTIVES: Laboratory section. Students will work to begin planning an experiment.
The emphasis will be on replication.
Tuesday – January 12th
Institutions and Markets.
OBJECTIVES: Using experiments to understand Political and Market Institutions
*Gürerk, Özgür, Bernd Irlenbusch, and Bettina Rockenbach. "The Competitive Advantage of
Sanctioning Institutions." Science 312, no. 5770 (2006): 108.
*Hamman, John R., Roberto A. Weber, and Jonathan Woon. "An Experimental Investigation
of Electoral Delegation and the Provision of Public Goods." American Journal Of
Political Science 55, no. 4 (2011): 737-51.
*Kessler, Judd B., and Alvin E. Roth. "Organ Allocation Policy and the Decision to Donte."
American Economic Review 102, no. 5 (2012): 2018-47.
Other interesting work:
Eckel, C.C., and S. Füllbrunn. (2015). "Thar She Blows? Gender, Competition, and Bubbles in
Experimental Asset Markets." American Economic Review 105(2): 906-20.
Experimental Design – p. 4
Labor
OBJECTIVES: Using experiments to estimate pricing and behavioral responses
*Charness, Gary, and Peter Kuhn. "Lab labor: What can labor economists learn from the
lab?." Handbook of labor economics 4 (2011): 229-330.
*Gneezy, Uri, and Aldo Rustichini. "A Fine Is a Price." Journal of Legal Studies XXIX (2000):
1-17.
Other interesting articles
Bertrand, M., and S. Mullainathan. (2004). "Are Emily and Greg More Employable Than
Lakisha and Jamal: A Field Experiment on Labor Market Discrimination." American
Economic Review 94: 991-1013.
Experimental Design –group projects
OBJECTIVES: This time will be used to continue planning for experiments.
Wednesday – January 13th
Heterogeneity.
OBJECTIVES: Dimensions of individual heterogeneity including altruism, risk, time and
competitiveness. We will focus particularly on gender..
*Niederle, Muriel, and Lise Vesterlund. "Do Women Shy Away from Competition? Do Men
Compete Too Much?" Quarterly Journal of Economics 122, no. 3 (2007): 1067-101.
*Kanthak, Kristin, and Jonathan Woon. "Women Don't Run? Election Aversion and
Candidate Entry." American Journal Of Political Science (forthcoming): 1-18.
Other Interesting Articles:
Mani, Anandi, Sendhil Mullainathan, Eldar Shafir, and Jiaying Zhao. "Poverty Impedes
Cognitive Function." Science 341, no. 30 August 2013 (2013): 976-80.
Experimental Design – p. 5
Augenblick, Ned, Muriel Niederle, and Charles Sprenger. "Working over Time: Dynamic
Inconsistency in Real Effort Tasks." edited by National Bureau of Economic
Research. Cambridge, MA, 2013.
Charness, Gary, Uri Gneezy, and Alex Imas. "Experimental Methods: Eliciting Risk
Preferences." Journal Of Economic Behavior & Organization 87 (2013): 43-51.
Crosetto, Paolo, and Antonio Filippin. "A Theoretical and Experimental Appraisal of Five
Risk Elicitation Methods." In Jena Economic Research Paper No. 2013-009:
http://dx.doi.org/10.2139/ssrn.2220631, 2013.
Filippin, Antonio, and Paolo Crosetto. "A Reconsideration of Gender Differences in Risk
Attitudes." In Grenoble Applied Economics Laboratory Working Papers 2014-01:
http://dx.doi.org/10.2139/ssrn.2402139, 2014.
Shah, Anuj K., Sendhil Mullainathan, and Eldar Shafir. "Some Consequences of Having Too
Little." Science 338, no. 2 November 2012 (2012): 682-85.
Dealing with Heterogeneity
OBJECTIVES: Statistical problems with small samples and modeling heteorgeneity.
*Moffatt, Peter G. (2015) Experimetrics. Chapters I and II.
Other interesting articles:
Imai, Kosuke, and Aaron Strauss. "Estimation of Heterogeneous Treatment Effects from
Randomized Experiments, with Application to the Optimal Planning of the Get-outthe_Vote Campaign." Political Analysis 19, no. 1 (2011): 1-19.
Dave, Chetan, Catherine C. Eckel, Cathleen A. Johnson, and Christian Rojas. "Eliciting Risk
Preferences: When Is Simple Better?" Journal of Risk & Uncertainty 41, no. 3 (2010):
219-43.
Gaines, Brian J., and James H. Kuklinski. "Experimental Estimation of Heterogeneous
Treatment Effects Related to Self-Selection." American Journal Of Political Science 55,
no. 3 (2011): 724-36.
Aronow, Peter M., and Cyrus Samii. "Does Regression Produce Representative Estimates of
Causal Effects?" American Journal Of Political Science (forthcoming): 1-18.
Experimental Design – p. 6
Experimental Design – group projects
OBJECTIVES: This time will be used to continue planning for projects.
Thursday – January 14th
Lab-in-the-field and Field experiments
OBJECTIVES: Experiments run in the field vs. field experiments.
*Duflo, Esther. "Field Experiments in Development Economics." edited by Massachusetts
Institute of Technology. Cambridge, MA, 2005.
*Enos, Ryan D. "Causal Effect of Intergroup Contact on Exclusionary Attitudes." Proceedings
of the National Academy of Sciences 111, no. 10 (2014): 3699-704.
*Attanasio, Orazio, Abigail Barr, Juan Camilo Cardenas, Garance Genicot, and Costas Meghir.
"Risk Pooling, Risk Preferences, and Social Networks." American Economic Journal:
Applied Economics 4, no. 2 (2012): 134-67.
Other interesting papers.
Kamdar, Amee, Steven D. Levitt, John A. List, Brian Mullaney, and Chad Syverson. "Once and
Done: Leveraging Behavioral Economics to Increase Charitable Contributions." In
SPI Working Paper Series, edited by Science of Philanthropy Initiative, 2015.
Michelitch, Kristin. "Does Electoral Competition Exacerbate Interethnic or Interpartisan
Economic Discrimination? Evidence From a Field Experiment in Market Price
Bargaining." American Political Science Review 109, no. 1 (2015): 43-61.
Broockman, David E. (2013). “Black Politicians Are More Intrinsically Motivated to
Advance Blacks’ Interests: A Field Experiment Manipulating Political Incentives”
American Journal of Political Science. 57:3
Findley, Michael, Daniel Nielson, and J. C. Sharman. "Causes of Non-Compliance with
International Law: A Field Experiment on Anonymous Incorporation." American
Journal of Political Science (forthcoming) (2014).
Charness, Gary, Uri Gneezy, and Michael A. Kuhn. "Experimental Methods: ExtraLaboratory Experiments-Extending the Research of Experimental Economics."
Journal Of Economic Behavior & Organization 91 (2013): 93-100.
Experimental Design – p. 7
IRBs, Pre-Registration, Writing Up, Reporting Results, Replication
OBJECTIVES: Discussion of what happens on the way to carrying out an experiment.
Strategies for ensuring you have a paper to write up. Publicizing your work.
*Humphreys, M., R. Sanchez de la Sera, and P. ven der Windt. (2013). "Fishing,
Commitment, and Communication: A Proposal for Comprehensive Nonbinding
Research Registration." Political Analysis 21(1): 1-20.
Other interesting articles:
Webster, Murray. 2014. "Funding Experiments, Writing Proposals." In Laboratory
Experiments in the Social Sciences, 2nd Edition, edited by Murray Webster and Jane
Sell, 473-502. San Diego, CA: Elsevier.
Eckel, Catherine C., Haley Harwell, and Jose Gabriel Castillo. "Four Classic Public Goods
Experiments: A Replication Study." Department of Economics: Texas A&M, 2015.
Experimental Design – individual and group projects
Friday – January 15th
Class Experiments
OBJECTIVES: Class experiments will be run. Individuals are expected to run their
experiments on classmates. Classmates will provide feedback.
Class Experiments.
OBJECTIVES: Class experiments will be run. Groups are expected to run their
experiments on classmates. Classmates will provide feedback.