PPOL 670-01: Program Evaluation
Georgetown Public Policy Institute
Fall 2003
Mondays, 6:00 – 7:40 p.m.
Walsh 390
Professors:
Carolyn Hill, GPPI 111
cjh34@georgetown.edu
Tel: (202) 687-7017
Office hours: Tuesdays 2-3 pm,
Dan Levy
dml25@georgetown.edu
Tel: (202) 484-3090
Office hours:
and by appointment
In person- by appointment
Virtual- Fridays 12-1pm
(Details in Technology handout)
Course Description
This course introduces students to the approaches used by social scientists to evaluate the implementation
and impacts of public policies. Topics covered include reasons for and uses of program evaluations; the
different kinds of information gained through implementation analysis and the integration of qualitative and
quantitative research; statistical power and effect size; and cost-benefit analysis. The bulk of the course
focuses on the techniques, advantages, and drawbacks of experimental and quasi-experimental designs.
Texts and Materials
Required:  Rossi, Peter H., Howard E. Freeman, and Mark W. Lipsey. 1999. Evaluation: A Systematic
Approach, 6th edition (Thousand Oaks, CA: Sage Publications).
 Course Packet (available from Kerry Pace).
On reserve at Lauinger Library:
 Berk, Richard A. and Peter H. Rossi. 1990. Thinking About Program Evaluation
(Thousand Oaks, CA: Sage Publications).
 Greenberg, David, and Mark Shroder. 1997. The Digest of Social Experiments, 2nd ed.
(Washington, DC: Urban Institute Press).
 Shadish, Cook, and Campbell, 2002. Experimental and Quasi-Experimental Designs for
Generalized Causal Inference (Houghton Mifflin).
Additional materials (more information will be provided about these):
 Evaluations from:
-- 21st Century Community Learning Centers Program: This program, funded by the
U.S. Department of Education, seeks to provide after-school experiences to enrich
children’s educational, social, and developmental skills.
-- National Evaluation of Welfare-to-Work Strategies (NEWWS): Funded by the U.S.
Department of Health & Human Services and the U.S. Department of Education, this
evaluation examined 11 mandatory welfare-to-work programs that were created or
adapted to fit the provisions of the Job Opportunity and Basic Skills (JOBS) program
created by the Family Support Act of 1988.
-- PROGRESA (Programa de Educación, Salud y Alimentación): A conditional cash
transfer program targeted to poor families in Mexico. Households get cash assistance if
they send their children to school and visit health care providers.
1
Course Requirements and Grading
Class participation and engagement
Evaluation reviews, written assignments (4)
Take-home final exam
15%
60%
25%
Notes on the Reading List
The reading list for this class is extensive, and some of the papers are quite technical. We do not expect
you to understand every line in every paper that you read. However, you are equipped to understand much
of what is in these papers, and you should attempt to do so. We think it’s reasonable to expect that you
will spend an average of 10 hours a week outside of reading and preparing for this course (after all, we
only get 100 minutes together each week!).
We will pass out additional readings throughout the semester, as needed.
The “optional” readings on the reading list are just that: truly optional. Many are classics of the field,
which you may want to have a reference for at some point in the future.
Class Participation and Engagement (15%)
We strongly believe that student participation can substantially enrich the learning experience for both the
students and the instructors. In this spirit, class participation is strongly encouraged. We will strive to lead
stimulating discussions and will ask questions to highlight concepts and assess class comprehension. You
are encouraged to ask questions and to share with the class any relevant insights you may have from your
work experience or from previous exposure to these topics.
Effective class participation requires that you read the assigned readings before coming to class. We will
(randomly) team each student with one other student in the class. Together, your pair will be responsible
for posting your Most Burning Question to the Blackboard discussion board no later than noon on
Mondays that class is held. This process will help you grapple with the things you don’t understand, and
will help us target the ideas that you are struggling with (and move quickly over the things you do
understand).
Part of your class participation grade will be based on the submission of your questions and the “quality”
of your questions. For example, instead of submitting questions such as “I don’t get selection bias,” say
some more and try to express what exactly it is about selection bias that you don’t understand, e.g.:
“We’re having trouble with the concept of selection bias. We’ve read the definition over and over, and
see the examples in the assigned readings, but we’re having trouble thinking of any other possible
examples of selection bias, and don’t feel confident that we would be able to identify it in a research
article. Is it that people in programs are just different than other people? Does it matter how different?
Does it apply only to things we can see or measure? Or does it apply to things we can’t see or measure?
(if the latter, then aren’t we always concerned about selection bias? Seems like we would be).”
Evaluation reviews, written assignments (60% total: 15% each)
These will consist of analyses of articles, short problem sets, and short papers. We encourage you to
discuss or work through these in groups, but the write-ups must be your own work.
Take-home final exam (25%)
We will ask you to critique an article or evaluation using the concepts you have learned throughout the
course.
2
Letter Grades: These will be based on the weighted average of all your coursework. We do not have a
predetermined distribution of final grades in mind, but in the interest of setting clear expectations, we
thought it would be good to provide you with a typology of what we would consider an “A” student:
When we think of an A student, here is what we have in mind: This is a student who, before and after
each class, actively reads the book material and handouts corresponding to that class. This is a student
who actively and constructively participates in class, and takes an inquisitive attitude towards the
material, constantly questioning why things are the way they are. This is a student who devotes full effort
to all course activities: homework assignments, classroom exercises, debates, etc. And of course, this is a
student who scores very highly on homework assignments and exams.
COURSE OUTLINE
Please read the assigned readings for each week before coming to class (including the first week!)
September 1:
NO CLASS (Labor Day)
September 8:
Introduction: What is Program Evaluation? (Dan and Carolyn)
 Rossi et al., Chapters 1 – 3
 Michalopoulos 2003, pp. 1-11
 Haveman 1987
 Smith 2000, pp. 1-11
Optional:
 Rossi et al., Chapters 4 – 5
 Bradley & Shaefer 1998, chapter 8
 Holland 1986
September 15:
Implementation & Process Analysis
Guest Lecturer: Michelle Derr, Mathematica Policy Research (invited)
 Rossi et al., Chapter 6
 Charlesworth & Born 2003
 Palumbo and Calista 1990
September 22:
A Framework for Assessing Program Impacts (Carolyn)
 Rossi et al., Chapter 7
 Shadish, Cook, & Campbell 2002, chapters 1-3
Optional
 Shadish, Cook, & Campbell 2002, pp. 462-483
3
September 29:
Assessing Program Impacts using Experiments (Dan)
 Rossi et al., Chapter 8
 The Economist 2002
 Smith 2000, pp. 11-33
 Cook 2002
 Ludwig, Duncan, & Hirschfield 2001
 Orr 1999, chapter 3
 Boruch & Foley 2000
Optional:
 Moffitt 2003
 Shadish, Cook, & Campbell 2002, chapters 8-9
Assignment #1 Due
October 6:
Assessing Program Impacts using Quasi-Experimental Methods (Carolyn)
(Comparison Groups, Difference-in-Differences)
 Rossi et al., Chapter 9
 Meyer 1995
 Angrist & Krueger 1999
 Bell et al.1995, Chapter 2
 Ravallion 2002
Optional
 Shadish, Cook, and Campbell 2002, chapter 5
October 13:
NO CLASS (Columbus Day)
October 20:
Assessing Program Impacts using Quasi-Experimental Methods (Dan)
(IV, Regression Discontinuity)
 Rossi et al., Chapter 9
 Smith 2000, pp. 11-33
 Levitt 1997
 Jacob & Lefgren 2002
Optional
 Angrist, Imbens, and Rubin 1996
 Angrist 2003
 Chay & Greenstone 1999
 Shadish, Cook, & Campbell, chapter 7
Assignment #2 Due
4
October 27:
Assessing Program Impacts using Quasi-Experimental Methods (Carolyn)
(Matching, Interrupted Time Series, and Back to Comparison Groups)
 Dehejia & Wahba 1999
 Bloom 2003
 Hollister & Hill 1995
Optional
 Rosenbaum & Rubin 1983
 Shadish, Cook, & Campbell, chapter 6
November 3:
Assessing Program Impacts: Experiments vs. Quasi-Experiments (Dan)
 Burtless 1995
 Heckman and Smith 1995
 Gueron 2002
Optional :
 Lalonde 1986
 Fraker & Maynard 1987
 Friedlander & Robins 1995
 Heckman, Ichimura, & Todd 1997
 Heckman, Ichimura, & Todd 1998
 Heckman, Ichimura, Smith, & Todd 1998
 Smith & Todd forthcoming
 Bloom et al, 2003
 Glazerman et al, 2002
 Moffitt 2002
 Shadish, Cook, & Campbell 2002, pp. 484-497
 Wilde & Hollister 2002
November 10:
Statistical Power, Effect Size, and Design Sensitivity (Dan)
 Rossi et al., pp. 244-247
 Lipsey 1998
 Orr 1999, chapter 4
Optional
 Lipsey 1990, pp. 11-96
 Rosenthal 2000
Assignment #3 Due
November 17:
Cost-Benefit and Cost Effectiveness Analysis (Carolyn)
 Rossi et al., Chapter 11
 Boardman et al 1996
 Hahn & Wallsten 2003
 Weimer & Vining 1992 chapter 9
Optional
 Stokey and Zeckhauser 1978
5
November 24:
Cost-Benefit and Cost Effectiveness Analysis (Carolyn)
[continued from above]
December 1:
Putting it all together: Integrating Evaluation Methods (Dan)
 McLaughlin 1985
 Sherwood & Doolittle 2002
Optional
 Cronbach et al. 1990, chapter 4
Assignment #4 Due
December 8:
Using the Results of Program Evaluation (Dan and Carolyn)
[NOTE: This is during the University’s study days period; however, the Registrar tells
Monday-only classes to meet on this day]
 Rossi et al., Chapter 12
 Cook and Shadish 1986
 Coleman 1982
 Orr 1999, chapter 7
 Weiss 1972
 Weiss 1978
Optional:
 Shadish, Cook, and Leviton 1991
 Weiss 2000
6
PPOL 670 READING PACKET
Fall 2003
Professors Carolyn Hill & Dan Levy
Angrist, Joshua D., and Alan B. Krueger. 1999. “Empirical Strategies in Labor Economics.” In Handbook
of Labor Economics, Volume 3, edited by O. Ashenfelter and D. Card. (Amsterdam: Elsevier
Science), pp. 1277-1366.
Bell, Stephen H., Larry L. Orr, John D. Blomquist, and Glen G. Cain. 1995. “The Case for ApplicantBased Comparison Groups.” Chapter 2 in Program Applicants as a Comparison Group in
Evaluating Training Programs (Kalamazoo, MI: W.E. Upjohn Institute for Employment Research).
Bloom, Howard S. 2003. “Using ‘Short’ Interrupted Time-Series Analysis to Measure the Impacts of
Whole-School Reforms.” Evaluation Review 27(1): 3-49.
Boardman, Anthony E. et al. 1996. “Estimating Impacts from Demonstrations.” Chapter 9 in Cost-Benefit
Analysis: Concepts and Practice (Upper Saddle River, NJ: Prentice Hall), pp. 250-291.
Boruch, Robert F., and Ellen Foley. 2000. “The Honestly Experimental Society: Sites and Other Entities
as the Units of Allocation and Analysis in Randomized Trials.” In Leonard Bickman, ed. Validity
and Social Experimentation: Donald Campbell’s Legacy (Thousand Oaks, CA: Sage Publications),
pp. 193-238.
Burtless, Gary. 1995. “The Case for Randomized Field Trials in Economic and Policy Research.” Journal
of Economic Perspectives 9(2): 63-84.
Charlesworth, Leanne, and Catherine Born. 2003. “Approaches to Data Collection for Implementation
Analysis.” In Mary Clare Lennon and Thomas Corbett eds. Policy Into Action: Implementation
Research and Welfare Reform (Washington, DC: Urban Institute), pp. 239-279.
Coleman, James S. 1982. “Policy, Research, and Political Theory.” In William H. Kruskal, ed. The Social
Sciences: Their Nature and Uses (Chicago: University of Chicago Press), p. 95-99. Presented at the
375th Convocation Address, University of Chicago, December 18, 1979. Originally published in the
University of Chicago Record 14(2) (April 4, 1980): 78-80.
Cook, Thomas D. 2002. “Randomized Experiments in Educational Policy Research: A Critical
Examination of the Reasons the Education Evaluation Community has Offered for not Doing
Them.” Educational Evaluation and Policy Analysis 24(3): 175-199.
Cook, Thomas D., and William R. Shadish, Jr. 1986. “Program Evaluation: The Worldly Science.”
Annual Review of Psychology 37: 193-232.
Dehejia, Rajeev H., and Sadek Wahba. 1999. “Causal Effects in Nonexperimental Studies: Reevaluating
the Evaluation of Training Programs.” Journal of the American Statistical Association 94(448):
1053-1062.
The Economist, March 2nd 2002. “Try it and see.”
Gueron, Judith M. 2002. “The Politics of Random Assignment: Implementing Studies and Affecting
Policy.” In Frederick Mosteller and Robert Boruch (eds.) Evidence Matters: Randomized Trials in
Education Research (Washington, DC: Brookings Institution Press).
7
Hahn, Robert, and Scott Wallsten. 2003. “Whose Life is Worth More? (And Why is it Horrible to Ask?”
Washington Post, Sunday, June 1. p. B3.
Haveman, Robert H. 1987. “Policy Analysis and Evaluation Research.” Chapter 8 in Poverty Policy and
Poverty Research (Madison: University of Wisconsin Press).
Heckman, James J., and Jeffrey A. Smith. 1995. “Assessing the Case for Social Experiments.” Journal of
Economic Perspectives 9(2): 85-110.
Hollister, Robinson G., and Jennifer Hill. 1995. “Problems in the Evaluation of Community-Wide
Initiatives.” In James B. Connell, Anne C. Kubisch, Lisbeth B. Schorr, and Carol H. Weiss (eds.)
New Approaches to Evaluating Community Initiatives (Washington, DC: Aspen Institute).
Jacob, Brian A., and Lars Lefgren. Forthcoming. “The Impact of Teacher Training on Student
Achievement: Quasi-Experimental Evidence from School Reform Efforts in Chicago.” Journal of
Human Resources.
Levitt, Steven D. 1997. “Using Electoral Cycles in Police Hiring to Estimate the Effect of Police on
Crime.” American Economic Review 87(3): 270-290.
Lipsey, Mark W. 1998. “Design Sensitivity: Statistical Power for Applied Experimental Research.”
Chapter 2 in Leonard Bickman and Debra J. Rog, eds. Handbook of Applied Social Research
Methods (Thousand Oaks, CA: Sage Publications), pp. 39-68.
Ludwig, Jens, Greg J. Duncan, and Paul Hirschfield. 2001. “Urban Poverty and Juvenile Crime: Evidence
from a Randomized Housing-Mobility Experiment.” Quarterly Journal of Economics vol(issue):
655-679.
McLaughlin, Milbrey Wallin. 1985. “Implementation Realities and Evaluation Design.” In R. Lance
Shotland and Melvin M. Mark, eds. Social Science and Social Policy (Thousand Oaks, CA: Sage
Publications), pp. 96-120.
Meyer, Bruce D. 1995. “Natural and Quasi-Experiments in Economics.” Journal of Business and
Economic Statistics 13(2): 151-161.
Michalopoulos, Charles. 2003. “Precedents and Prospects.” In Howard S. Bloom, ed. Learning More
from Social Experiments: Emerging Analytic Approaches (under review).
Orr, Larry L. 1999. “Alternative Random Assignment Models,” “Sample Design,” and “Social
Experimentation and the Policy Process.” Chapters 3, 4, and 7 in Social Experiments: Evaluating
Public Programs with Experimental Methods (Thousand Oaks, CA: Sage Publications), pp. 69-101,
103-137, 233-258.
Palumbo, Dennis J., and Donald J. Calista. 1990. “Opening up the Black Box: Implementation and the
Policy Process.” Chapter 1 in Dennis J. Palumbo and Donald J. Calista, eds. Implementation and the
Policy Process: Opening up the Black Box (New York: Greenwood Press), pp. 1-17.
Ravallion, Martin. 2001. “The Mystery of the Vanishing Benefits: An Introduction to Impact Evaluation.”
The World Bank Economic Review 15(1): 115-140.
Shadish, William R., Thomas D. Cook, and Donald T. Campbell. 2002. “Experiments and Generalized
Causal Inference,” Statistical Conclusion Validity and Internal Validity,” and “Construct Validity
and External Validity.” Chapters 1, 2, and 3 in Experimental and Quasi-Experimental Designs for
Generalized Causal Inference (Boston: Houghton Mifflin).
8
Sherwood, Kay E., and Fred Doolittle. 2003. “What Lies behind the Impacts? Implementation Research
in the Context of Net Impact Studies.” In Mary Clare Lennon and Thomas Corbett, eds. Policy into
Action: Implementation Research and Welfare Reform (Washington DC: Urban Institute Press), pp.
193-231.
Smith, Jeffrey. 2000. “Evaluating Active Labor Market Policies: Lessons from North America,” in
MittAB-Schwerpunktheft 2000: Evaluation aktiver Arbeitsmarktpolitik, Nuremberg: IAB: pp. 345356.
Weimer, David L., and Aidan R. Vining. 1992. “Benefit-Cost Analysis.” In Policy Analysis: Concepts
and Practice, 2nd ed. (Englewood Cliffs, NJ: Prentice Hall), pp. 259-311.
Weiss, Carol H. 1978. “Improving the Linkage Between Social Research and Public Policy.” In Laurence
E. Lynn, Jr. (ed.) Knowledge and Policy: The Uncertain Connection (Washington, DC: National
Academy of Sciences).
Weiss, Carol H. 1972. “Utilization of Evaluation Results.” In Evaluation Research: Methods for
Assessing Program Effectiveness (Englewood Cliffs, NJ: Prentice-Hall, Inc.), pp. 110-128.
9