1J
METHODS OF EVALUATION RESEARCH
Dr. Kevin Smith, Department of Political Science, University of Nebraska, Lincoln, USA
7 July to 18 July 2014 (two week course/35 hrs)
This course is a general introduction to the basic logic and methods of evaluation research and policy
analysis. It is organized around the assumption that the systematic analysis of public programs and
policy is motivated by the desire to answer one of two main questions: (1) What should we do? In other
words, establishing what program or policy should or should not be implemented and why. (2) What have
we done? In other words, establishing what a given program or policy has or has not actually achieved.
The first week will introduce a systematic framework for the scope and methods of evaluation research,
then apply this framework to ex ante policy analysis, i.e. looking at how to answer the first question by
systematically evaluating different policy options and deciding which is “best.” The primary focus this
week will be on projection techniques and the rationale, concepts and methodology of cost analysis,
which is a common approach to ex ante policy analysis. Much of the second week will concentrate on ex
post policy analysis, i.e. looking at how to answer the second question by systematically evaluating the
actual impact or outcomes of a policy or program. The focus this week will be on the logic and methods of
impact analysis (quantitative program evaluation).
Evaluation research is a multi-disciplinary field that employs a broad variety of methods and
models. This course will not make you an expert policy analyst; it should provide you with a firm
understanding of the skills necessary to undertake a thorough program or policy evaluation and the
knowledge to be an informed consumer of a broad range of evaluation and policy analysis research.
Students should come out of this course with a basic set of analytical skills that allows them to do the
following:
* Systematically frame a problem
*Identify causal claims
*Design and execute a basic matrix analysis (a Goeller scorecard and a goals/alternatives
matrix).
*Understand how to do a simple trend analysis: single factor projection, Delphi technique,
statistical analysis of trend data.
* Understand and execute a basis cost benefit/effectiveness analysis (includes discounting,
calculating present values, valuation techniques, measures of allocative efficiency, sensitivity
analsysis).
* Understand how to design and execute a systematic impact analysis (i.e. do a basic quantitative
program evaluation).
* Understand the methodological limitations of mainstream impact analysis techniques and how
they can be addressed (e.g. “second best” solutions, Leamer bounds, SWAT analysis).
*Most importantly, students should leave the course ready to “hit the ground running,” i.e. feel
reasonably confident they can construct a systematic plan of attack to address a question of
“what should we do?” or “what have we done?”
We are going to be covering quite a bit of ground, the goal is to provide the basic logic and
structure of policy analysis and introduce a series of commonly employed quantitative (and one or two
qualitative) policy analysis methods. I assume a basic knowledge of OLS regression and a basic
familiarity with a standard statistical package like STATA SPSS, though lack of the latter should not be a
deterrent to taking the course—the basic process to execute the analyses will be demonstrated in class.
Other than that, I am making no other assumptions about statistical or mathematical skills (except,
perhaps, that you don’t have them!)..
Required Readings
Smith, Kevin B. and Christopher W. Larimer. 2013. The Public Policy Theory Primer. 2nd ed. Boulder, CO:
Westview Press.
A reading packet will be made available through the summer school.
Recommended Readings
A good accompanying text for the class is:
Dunn, William N. 2008. Public Policy Analysis: An Introduction. 4th ed. Upper Saddle River, NJ: PrenticeHall.
Background Readings
Any basic stats or econometrics text that goes through simple multiple regression will do (if you are turned
off by hard core econometrics textbooks I highly recommend Applying Regression & Correlation by
Jeremy Miles and Mark Shevlin—it’s extremely user friendly, and if you can get through the first five
chapters of this book you’ll have all the basic stats skills you need for this course). A familiarity with a
decent undergraduate policy analysis textbook wouldn’t hurt, but is not necessary. Good, easily
accessible examples would be:
Bickers, Kenneth and John T. Williams. 2001. Public Policy Analysis. New York: Houghton Mifflin.
This will set students up well for the rationalist approach to policy analysis that is the primary focus of this
course.
Clemons, Randall and Mark K. McBeth. 2001. Public Policy Praxis. Upper Saddle River, NJ: Prentice
Hall.
Provides and alternative approach to the rationalist approach to public policy analysis.
Another good book that introduces the basic framework and concepts important to this class, but does it
in an applied rather than academic fashion, is this:
Bardach, Eugene. 2005. A Practical Guide for Policy Analysis. Washington, D.C.: CQ Press
Provisional Timetable and readings
Day 1:
Context and Structure of Policy Analysis.
An introduction to the scope, logic, and methods of policy and evaluation research.
Required Readings:
Smith and Larimer, Chapter 1.
Suggested Readings:
DeLeon, Peter. 2006. “The Historical Roots of the Field.” In The Oxford Handbook of
Public Policy. M. Moran, M. Rein and R. Goodin, eds. New York: Oxford University Press.
Lab Assignment: Regression review.
Day 2:
Type III (Yes three) Errors: Making Systematic Sense of Policy Problems.
--Type III errors
--characteristics of policy problems
--classes of policy problems
--structuring policy problems
--developing and analyzing policy arguments
.
Required Readings:
Dunn, Chapters 3
Suggested readings: TBA
Lab Assignment: Problem identification
Day 3:
Peering ‘round Corners: Predicting Policy Outcomes, or The Basic Requirement for Ex
Ante Analysis
--Extrapolative forecasting
--Theoretical (logical) forecasting
--Judgmental forecasting
Required Readings: Dunn, Chapter 4..
Suggested Readings:
Gupta, Dipak. 2001. Analyzing Public Policy: Concepts, Tools and Techniques.
Washington, D.C.: CQ Press. Chapters 8-13.
Lab Assignment: Forecasting exercise
Day 4:
What Should we Do?: Introduction to the logic and methods of Ex Ante Analysis Methods
--Matrix analysis
--efficiency as a basis for social welfare
--social welfare and social surplus
--monetizing policy inputs as opportunity costs
--discounting
Required Readings: Smith and Larimer, Chapter 5.
Suggested Readings: Dunn Chapter 5
Lab Assignment: A basic ex ante analysis
Day 5:
Valuing Policy Outcomes (Benefits)
--secondary plug-ins
--intermediary goods
--hedonic pricing
Required Readings: .
Reese, Mike. 2001. “Hedonic Quality Adjustment Methods for College Textbooks.”
U.S. Bureau of Labor Statistic.
Liegly, Paul R. 2003. “Hedonic Quality Adjustment Methods for Clothes Dryers in the
U.S.” U.S. Depratment of Labor.
Suggested Readings:
Henry M. Levin and Patrick J. McEwan. 2001. Cost-Effectiveness Analysis. 2nd Ed.
Thousand Oaks, CA: Sage Publications. Chapter 7.
Lab Assignment: Using regression to monetize benefits.
Day 6:
Measures of Allocative Efficiency/Sensitivity Analysis; the Pros and Cons of Cost
Analytic Approaches to Ex Ante Analysis
–
–
–
–
–
–
–
–
–
–
Measures of allocative efficiency
Basic time issues
Terminal values
Expected net benefits
Partial sensitivity analysis
Switching/break even values
Monte Carlo sensitivity analysis
Extreme case sensitivity analysis
Minimax regret decision criteria
Expected net benefits
Required Readings
Hahn, Tetlock and Burnett. 2000. “Should You be Allowed to Use Your Cell
Phone While Driving?” Regulation
Lab Assignment: Sensitivity Analysis Exercise
.
Day 7:
Introduction to Impact (Ex Post) analysis.
–
–
–
–
–
Logic of impact analysis
Key challenges of impact analysis
Program theory and outcome lines
Importance of counterfactual to meeting these challenges
Basic research designs and advantages/disadvantages
Required Readings: Smith and Larimer.Chapter 6.
Suggested Readings:
Langbein, Laura, with Claire Felbinger. 2006. Public Program Evaluation: A Statistical
Guide. Armonk, NY: M.E. Sharpe. Chapters 1 and 2.
Lab Assignment: Basic regression impact analysis (no explicit research design)
Day 8:
Research Design and The Regression Framework for Operationalizing Impact
Analysis Research Designs
Required Readings: Chapter 5. Mohr, Lawrence. 1995. Impact Analysis for Program
Evaluation. 2nd Ed. Thousand Oaks, CA: Sage Publications.
Suggested Readings:
Mohr, Lawrence. 1995. Impact Analysis for Program Evaluation. 2nd Ed. Thousand
Oaks, CA: Sage Publications.
Lab Assignment: Estimating counterfactuals from regression/experimental and/or
quasi-experimental designs using regression
Day 9
Correlational Impact Analysis/Sensitivity Analysis in Ex Post Studies.
Required Readings: Smith, Kevin B. and Scott Granberg-Rademacker. “Money
Only Matters If You Want It To? Exposing the Normative Implications of Empirical
Research.” Political Research Quarterly.
Meier, Kennth and Lael Keiser. 1996. “Public Administration as a Science of the
Artificial.” Public Administration Review.
Bretschneider, Stuat, Frederick Marc-Aurele, Jiannan Wu. 2005. “Best Practices
Research: A Methodological Guide for the Perplexed.” Journal of Public
Administration Research and Theory.
Suggested Readings: Smith and Larimer Chapter 7/10,
Lab Assignment: Correlational impact analysis/sensitivity analysis exercise
Day 10
Putting It Altogether: Doing Policy Analysis in the Real World
Required Readings:: “Designing Impact Evaluations: Jamaica’a PATH Program.”
Case study.
Labe Assignment: None