Quantitative Methods for Policy Research and Evaluation
PUBL 6316 & POLS 6383
M 5.30pm – 8.30pm
AH 322
Dr. Jeronimo Cortina
424 PGH
jcortina@central.uh.edu
Office Hours: M 3:00pm- 5:00pm or by appointment
I. Course Description
What is the difference between ―good‖ and ―bad‖ policies? How can we distinguish between
them? Why is this important? The purpose of this course is to provide students with some basic
(and not so basic) scientific tools to develop a set of skills in order to conduct and evaluate public
policies.
Ultimately, the goal of this course is twofold. First, to provide students a practical statistical tool
to perform some more advanced statistical methods useful in answering policy questions when
using observational or experimental data. Second, this course will allow students to critically
review published research, whether for policy or academic purposes that claim to answer causal
questions. The main focus of this course is to focus on the challenge of answering causal
questions, that is, those that take the form ―Did X cause Y?‖ using data that do not necessarily
conform to a well implemented randomized study. The course will be based on examples from
real public policy issues in order to illustrate key ideas and methods. First, we will explore how
best to design a study to answer causal questions given the logistical, budgetary and ethical
constraints inherit in policy evaluation. We then discuss several approaches to drawing causal
inferences from observational studies including propensity score matching, instrumental
variables, differences in differences, fixed effects models and regression discontinuity designs.
In order to make the class interesting and dynamic all students are expected to be ACTIVE
participants and are to be PREPARED to critically ENGAGE and DISCUSS assigned readings
as scheduled.
II. Grading Procedures:
There will be 7-8 homeworks, which will total 70% of the final grade and one final project or
final exam (30%) that will involve both data analysis and a thoughtful description of both the
analysis and the findings. One homework will involve a class presentation. Depending on the
size of the class, some assignments may be done in groups.
1
Final letter grades will be calculated using the following percentages:
95-100% = A
90-94 = A-
87-89 = B+
84-86 = B
80-83 = B-
77-79 = C+
74-76 = C
70-73 = C-
67-69 = D+
64-66 = D
60-63 = D59 or less = F
III. Course Outline
The following outline describes the topics that will be covered and its readings. Those readings
marked with an * are recommended, and thus not required. Some of the reading assignments
might change in case that there is a more appropriate reading (there will be a class announcement
to indicate if a reading has been change with enough time in advance). Readings are available in
Jstor or their respective Journal (through MD Anderson Library) or will be at Blackboard.
Session 1: Review Session and Introduction to Causal Inference
*Cortina, Jeronimo. 2009. "To Treat or not to Treat: Causal Inference in the Social Sciences." In
A Quantitative Tour of the Social Sciences, ed. A. Gelman and J. Cortina. New York:
Cambridge University Press.
*Hill, Jennifer., Reiter, Jerome., and Zanutto, Elaine. (2004) ―A comparison of experimental and
observational data analyses‖ Applied Bayesian Modeling and Causal Inference from an
Incomplete-Data Perspective. Edited by Andrew Gelman and Xiao-Li Meng. West
Sussex, England: Wiley.
*Leamer, Edward (1983) "Let's take the con out of econometrics", American Economic Review,
73(1): 31-43
*Hill, J. (2004) ―Evaluating School Choice Programs‖ forthcoming in Statistics: A Guide to the
Unknown edited by Hal Stern et al.
*Holland, Paul W. (1986), ``Statistics and causal inference (with discussion)'', Journal of the
American Statistical Association, 81: 945-970
*Sobel, M. ―Causal Inference in the Social and Behavioral Sciences‖ in Handbook of Statistical
Modeling for the Social and Behavioral Sciences. pp. 1-38
2
Session 2: Randomized Experiments and the Rubin Causal Model
Katz, L.F., Kling, J.R., and Liebman, J.B. (2001) ―Moving to Opportunity in Boston: Early
Results of a Randomized Mobility Experiment‖ The Quarterly Journal of Economics
116: 607-654.
Winship, Christopher and Michael Sobel (2004) ―Causal Inference in Sociological Studies‖ in
Handbook of Data Analysis edited by Melissa Hardy and Alan Bryman, London: Sage
Publications, 481-504 – the version you’ll see is number 1-63 and you’ll be responsible
through p. 25 for this week
*Rosenbaum, P. (2002) Observational Studies, 2nd ed., New York: Springer, Chapter 2
*Rubin, D. (1990) ―Formal modes of statistical inference for causal effects‖ Journal of Statistical
Planning and Inference 25: 279-292. (available through e-journals)
*Rubin, DB (1974) "Estimating Causal Effects of Treatments in Randomized and NonRandomized Studies" Journal of Educational Psychology, 66: 688-701
*Sobel, Michael E. (1996), ―An introduction to causal inference‖, Sociological Methods and
Research, Vol. 24, Iss. 3; p. 353-379
Session 3: Observational Studies
Rosenbaum, P. (2002) Observational Studies, 2nd ed., New York: Springer, Chapter 1
Winship, Christopher and Michael Sobel (2004) ―Causal Inference in Sociological Studies‖ in
Handbook of Data Analysis edited by Melissa Hardy and Alan Bryman, London: Sage
Publications, 481-504 – the version you’ll see is number 1-63 and you’ll be responsible
for 36-38 for this week
Donohue, J. J., and S.D. Levitt (2001) ―The impact of legalized abortion on crime‖ The
Quarterly Journal of Economics, 116(2): 379-420
*Rubin, D. (1977) ―Assignment to Treatment Groups on the Basis of a Covariate‖ Journal of
Educational Statistics, 2: 1-26
*LaLonde, R. (1986) Evaluating the Econometric Evaluations of Training Programs, American
Economic Review, 76: 604-620
*Rosenbaum, P. (2002) ―Covariance adjustment in randomized experiments and observational
studies‖, Statistical Science, 17(3): 286-327
*William G. Cochran and Donald B. Rubin, (1973) "Controlling Bias in Observational Studies:
A Review", Sankhya, 35: 417-446 (LL)
3
Session 4: Propensity Score - Theory
Rosenbaum, PR and D B. Rubin (1985) "Constructing a control group using multivariate
matched sampling methods that incorporate the propensity score", The American
Statistician, 39: 33-38
Kam, Cindy D., and Carl L. Palmer. 2008. "Reconsidering the Effects of Education on Political
Participation." The Journal of Politics 70 (3):612-31.
*Rosenbaum, Paul R. and Rubin, Donald B. (1984) "Reducing Bias in Observational Studies
Using Subclassification on the Propensity Score" Journal of the American Statistical
Association, 79: 516—524
Session 5: Propensity Score– Practice
D’Agostino, R (1998) "Propensity score methods for bias reduction in the comparison of a
treatment to a non-randomized control group" Statistics in Medicine, 17: 2265-2281.
Hill J, Waldfogel J, Brooks-Gunn J (2002) ―Differential effects of high-quality child care‖
Journal of Policy Analysis and Management, 21 (4): 601-627
*Foster, M. ―Propensity Score Matching: An Illustrative Analysis of Dose Response‖
forthcoming in Medical Care
Session 6: Propensity Score Wrap-Up
Dehejia, Rajeev H. and Wahba, Sadek (1999) "Causal Effects in Nonexperimental Studies:
Reevaluating the Evaluation of Training Programs", Journal of the American Statistical
Association, 94: 1053—1062
Session 7: Instrumental Variables Models – Introduction and Theory
Angrist, J D. (1990) "Lifetime Earnings and the Vietnam Era Draft Lottery: Evidence from
Social Security Administrative Records," American Economic Review, 80: 313-336
Angrist, J D., Imbens, G W. and D B. Rubin, (1996) " Identification of Causal Effects Using
Instrumental Variables," Journal of the American Statistical Association, 91: 444-472
*Wooldridge, J. M. (2003) Introductory Econometrics Chapter 15 (LL)
4
Session 8: Instrumental Variables Models – Practice (Stata)
Angrist JD, Evans WN (1998), "Children and their parents' labor supply: Evidence from
exogenous variation in family size", American Economic Review 88(3): 450-77
*E Michael Foster. (2000) ―Is more better than less? An analysis of children’s mental health
services‖ Health Services Research. Chicago: Vol. 35, Iss. 5; p. 1135
*Levitt, Steven D. 1996. ―The Effect of Prison Population Size on Crime Rates: Evidence from
Prison Overcrowding Litigation.‖ Quarterly Journal of Economics, 111(2): 319-51.
Session 9: Difference in Differences
Chapter 18 on Panel Data Model in Ashenfelter book Statistics and Econometrics (published by
Wiley, 2003), pp. 262-273
Angrist, J. D., and Krueger, A. (1999), ―Empirical Strategies in Labor Economics,‖ in Orley
Ashenfelter and David Card (eds), Handbook of Labor Economics, Vol. 3A, Amsterdam: HorthHolland, A version is available online at http://www.irs.princeton.edu/pubs/pdfs/401.pdf
ONLY PP 19-23
Bogart & Cromwell. ―How much is a neighborhood worth?‖ J. Urban Economics 47
*Card, D. and A. Krueger (1994) ―Minimum Wages and Employment: A Case Study of the Fastfood Industry in New Jersey and Pennsylvania,‖ American Economic Review,
84(4): 772-784.
*Meyer, B. (1995) ―Natural and Quasi-Experiments in Economics,‖ Journal of Business and
Economic Statistics, 13(2): 151-161
Session 10: Fixed Effects
Aaronson, Daniel. (1998) "Using Sibling Data to Estimate the Impact of Neighborhoods on
Children's Educational Outcomes" The Journal of Human Resources, 33(4): 915-946
*Korenman and Neumark (1991) "Does Marriage Really Make Men More Productive?" Journal
of Human Resources, 26(2): 282-307
*Chay, K. and Greenstone, M. (2003) ―The Impact of Air Pollution on Infant
Mortality: Evidence from Geographic Variation in Pollution Shocks Induced By A
Recession‖ Quarterly Journal of Economics 118(3): 1121-1167.
5
Session 11: Regression Discontinuity
Brian A. Jacob, Lars Lefgren (2004) ―Remedial Education and Student Achievement: A
Regression-Discontinuity Analysis‖ Review of Economics and Statistics 86(1)
Session 12: Structural Equations Modeling (SEM), Path Analysis
Holland, P. (1988) ―Causal Inference, Path Analysis, and Recursive Structural Equation
Models‖ Sociological Methodology, 18: 449-493
Session 13: Wrap-Up
Additional reading that could be helpful at some point:
*Over, M. and D. Jolliffe and A. Foster (1995) "Huber correction for two-stage least squares
estimates," Stata Technical Bulletin, 29: 24-25 (Reprinted in Stata Technical Bulletin
Reprints, vol.5, p.140-142)
*Bound, J, Jaeger, D.A., Baker, R.M. (1995) "Problems with instrumental variables estimation
when the correlation between the instruments and the endogenous explanatory variable is
weak" Journal of the American Statistical Association, 90: 443-450.
IV. Academic Integrity
Cheating and plagiarism will not be tolerated and will result in a grade penalty or failure of the
course. Each student in this course is expected to abide by the University of Houston’s policies
against cheating and plagiarism. The University’s statement on academic honesty is available
from the student handbook, which can be found at
http://www.uh.edu/dos/pdf/2009-2010StudentHandbook.pdf
You are encouraged to work, study together and to discuss information and concepts covered in
lectures and the sections with other students. You can give "consulting" help to or receive
"consulting" help from such students. However, this permissible cooperation should never
involve one student having possession of a copy of all or part of work done by someone else, in
the form of an e-mail, an e-mail attachment file, a diskette, or a hard copy.
Should copying occur, both the student who copied work from another student and the student
who gave material to be copied will both automatically receive a zero for the assignment (this
applies to homework, exams, quizzes, etc.). Penalty for violation of this Code can also be
extended to include failure of the course and University disciplinary action.
During examinations, you must do your own work. Talking or discussion is not permitted during
the examinations, nor may you compare papers, copy from others, or collaborate in any way.
Any collaborative behavior during the examinations will result in failure of the exam, and may
lead to failure of the course and University disciplinary action.
6
V. Accommodations for students with disabilities
The University of Houston is committed to providing reasonable accommodations for eligible
students with disabilities, including students who have learning disabilities, health impairments,
psychiatric disabilities, and/or other disabilities. If you believe you have a disability which
requires accommodation, please contact the Center for Students with Disabilities (CSD) at 713743-5400 voice or 713-749-1527 (TTY)
VI. Cell Phones, Beepers & Laptops
Since they cause interruptions and distractions, cell phones and beepers should be turned off
during class time. Please do not use any Instant Messaging software if you bring your laptop to
take notes. In particular, no cell phones, beepers and laptops may be accessible during exams.
7