PAI786: Urban Policy
Class 2:
Evaluating Social Programs
Urban Policy: Evaluating Social Programs
• Class Outline
▫ Positive vs. normative analysis
▫ The role of program evaluation
▫ Basic principles of program evaluation
▫ Program evaluation and decision making
Urban Policy: Evaluating Social Programs
• Positive vs. Normative Analysis
▫ Positive analysis: How do people behave?
 How are prices and quantities determined in a
particular market?
 What is the impact of a particular government
program and people’s behavior?
 If the necessary data are available, positive
statements can be tested .
Urban Policy: Evaluating Social Programs
• Positive vs. Normative Analysis
▫ Normative analysis: What is a good outcome?
 What are the appropriate objectives for government
intervention in a given market?
 Which objectives are the most important?
 Normative statements cannot be tested, but they
certainly can be debated!
Urban Policy: Evaluating Social Programs
• The Role of Program Evaluation
▫ Government programs change behavior.
▫ You cannot determine whether the outcome of a
government program meets your own objectives
without determining how it changes behavior.
▫ Program evaluation is necessary to identify the
programs that best meet your own objectives!
Urban Policy: Evaluating Social Programs
• What is the basic problem facing someone
wanting to evaluate any public program?
▫ What you want is to know how one place differs with
and without the program.
▫ What you observe is either (a) what the world is like
after and before the program or (b) what one place is
like with the program and another is without it.
▫ Thus, you cannot be sure that the effects you
observe are not due to non-program differences
over time or across places.
Urban Policy: Evaluating Social Programs
• What is the basic problem facing someone
wanting to evaluate any public program?
▫ Another way to put this is that the great challenge of any
program evaluation is to identify the “counterfactual”,
that is, to identify what would have happened if the
program had not been implemented.
▫ The counterfactual cannot be observed directly, so all
evaluation methods are attempts to estimate the
counterfactual in an unbiased way.
Urban Policy: Evaluating Social Programs
• Example (from Hollister)
▫ Consider a program that provides training and
counseling to improve participants’ employability.
▫ Suppose a high share of previously unemployed
participants become employed after leaving the
program.
▫ Does this evidence indicate that the program is
effective?
Urban Policy: Evaluating Social Programs
• Example (continued)
▫ Answer: No!
▫ People tend to move from unemployment to
employment over time, and programs tend to select
people who are unemployed.
 So the increase in employment may reflect the natural
process of moving to employment, not program impact.
 This is called regression to the mean.
▫ Local labor market conditions might have improved at
the time the program was implemented.
 So the increase in employment might reflect factors other
than the impact of the program.
 This is called omitted variable bias.
Urban Policy: Evaluating Social Programs
• The two ways to estimate program
impacts are
▫ random assignment
▫ statistical control.
▫ Random assignment ensures that differences
across time and place are not correlated with
program participation.
▫ Statistical controls can account for observable
differences across place or time and for certain
kinds of unobservable factors.
Urban Policy: Evaluating Social Programs
• Random assignment is the preferred
method in most cases.
▫ It provides results that are intuitively
compelling and scientifically sound.
▫ If you want to know a program’s impacts,
become an advocate for evaluation using
random assignment!
Urban Policy: Evaluating Social Programs
• Random assignment can be applied at
many different scales.
▫ Some evaluations randomly assign treatment to
individuals.
▫ Others randomly assign treatment to
organizations (firms, schools, etc.)
▫ Still others randomly assign treatment to
communities.
Urban Policy: Evaluating Social Programs
• Random assignment has been used to
study, among other things:
▫
▫
▫
▫
▫
▫
▫
▫
▫
▫
Welfare-to-work programs
Unemployment insurance
Job training
Income maintenance
Housing assistance
Electricity pricing
Education
Early childhood development
Criminal justice policy
Child health and nutrition
Urban Policy: Evaluating Social Programs
• Random assignment is not always
feasible.
• A huge literature indicates that the best
statistical studies:
▫ must have extensive data to ensure that
differences aren’t due to unobservable factors.
▫ must have comparable experimental and
control groups based on observable factors.
Urban Policy: Evaluating Social Programs
• Comparable Control Groups
• Recent advances in evaluation methodology have
discovered that it is very important to have
comparable control groups, because the impact of a
program may depend on the traits of the recipients.
• This leads to matching techniques, which focus
treatment-control comparisons on groups that are
comparable on all observable traits.
• Matching cannot solve the often-encountered
problem that treatment and control groups may
differ on unobserved traits—a problem that arises in
many statistical studies.
Urban Policy: Evaluating Social Programs
• For the case of community economic
development programs, Hollister
discusses several evaluations that do not
use random assignment.
▫ This discussion gives you a sense of what to look
for in statistical studies.
▫ You may want to return to it when we discuss
community economic development!
Urban Policy: Evaluating Social Programs
• Formal evaluation of programs or
management reforms are often not
available.
• Thus, it is appropriate for you (when you
become public officials!) to use your own
judgment:
▫ to select programs and reforms that appear
to have worked in other places
▫ to design new programs and reforms
Urban Policy: Evaluating Social Programs
• Evaluations of intermediate results can
also be helpful.
• Here is the figure in Hollister:
Resources/
Inputs
Activities
1
2
Your Planned Work
Outputs
3
Outcomes
4
Impact
5
Your Intended Results
Urban Policy: Evaluating Social Programs
• But evaluation should always be in the
back of your mind.
▫ Search for evaluations of the programs or
reforms you are interested in.
▫ Make an honest judgment about the quality of
existing evaluations.
▫ Informally apply basic evaluation
principles to programs and reforms you are
considering.
▫ Implement formal evaluations whenever
possible!
Urban Policy: Evaluating Social Programs
• Informal Evaluations
▫ Informal evaluations can be very helpful.
▫ Learn about the market in which the program will
operate—that is about the economic and social factors
that influence the behavior of market participants.
▫ Think about how various government programs
change the incentives of people in this market.
▫ Use your understanding from other cases to make an
educated guess about the impact of these changes in
incentives on behavior—and hence on your objectives.
Urban Policy: Evaluating Social Programs
• The Punchline
▫ You may undermine your own objectives if you
don’t take program evaluation seriously
▫ Look for (and advocate!) high-quality program
evaluation studies.
▫ When these studies are not available, make your
best judgment about the relevant positive analysis
using the best evidence and analysis you can find.