PPA 691 – Seminar in Public
Policy Analysis
Lecture 9 – Monitoring Policy
Outcomes
Introduction
 The consequences of policy actions are
never fully known in advance and, for this
reason, it is essential to monitor policy
actions after they have occurred.
 In fact, policy recommendations may be
viewed as hypotheses about the relationship
between policy actions and policy
outcomes: if action A is taken at time t1,
outcome O will result at time t2.
Monitoring in Policy Analysis
 Monitoring is the policy-analytic procedure used
to produce information about the causes and
consequences of public policies.
 Monitoring, since it permits analysts to describe
relationships between policy-program operations
and their outcomes, is the primary source of
knowledge about policy implementation.
 Monitoring is primarily concerned about with
establishing factual premises about public policy.
Monitoring in Policy Analysis
 Monitoring performs at least four major
functions:
– Compliance: monitoring helps determine whether
the actions of program administrators, staff, and
other stakeholders are in compliance with standards
and procedures imposed by legislatures, regulatory
agencies, and professional bodies.
– Auditing: monitoring helps determine whether
resources and services intended for certain target
groups and beneficiaries have actually reached
them.
Monitoring in Policy Analysis
 Monitoring performs at least four major
functions:
– Accounting: monitoring produces information
that is helpful in accounting for social and
economic changes that follow the
implementation of broad sets of public policies
and programs over time.
– Explanation: monitoring also yields
information that helps to explain why the
outcomes of public polices and programs differ.
Monitoring in Policy Analysis
 Sources of information.
– Information must be:
• Relevant.
– Macronegative versus micropositive.
• Reliable.
– Observations precise and dependable.
• Valid.
– Information about policy outcomes should measure what
we think it is measuring.
Monitoring in Policy Analysis
 Sources of information.
– Information on policy outcomes.
Historical Statistics of the United States
Current Opinion
Statistical Abstract of the United States
United States Census of Population by States
County and City Data Book
Congressional District Data Book
Census Use Study
National Opinion Research Center General Social
Survey
Social and Economic Characteristics of Students
Survey Research Center National Election Studies
Educational Attainment in the United States
National Clearinghouse for Mental Health
Information
Current Population Reports
National Clearinghouse for Drug Abuse Information
The Social and Economic Status of the Black
Population in the United States
National Criminal Justice Reference Service
Female Family Heads
Child Abuse and Neglect Clearinghouse Project
Monthly Labor Review
National Clearinghouse on Revenue Sharing
Handbook of Labor Statistics
Social Indicators
Congressional Quarterly
State Economic and Social Indicators
Law Digest
Monitoring in Policy Analysis
 Sources of information.
– When information is not available from existing
sources, monitoring may be carried out by
some combination of questionnaires,
interviewing, field observation, and the use of
agency records.
Monitoring in Policy Analysis
 Types of policy outcomes.
– Consequences:
• Policy outputs – the goods, services, or resources received by
target groups and beneficiaries.
• Policy impacts – actual changes in behavior and attitudes that
result from policy outputs.
– Populations:
• Target groups – individuals, communities, or organizations on
whom policies and programs are designed to have an effect.
• Beneficiaries – groups for whom the effects of policies are
beneficial or valuable.
Monitoring in Policy Analysis
 Types of policy actions. Policy actions have
two major purposes.
– Regulation – actions designed to ensure
compliance with certain standards or
procedures.
– Allocation – actions that require inputs of
money, time, personnel, and equipment.
– Regulative and allocative actions may have
consequences that are distributive or
redistributive.
Monitoring in Policy Analysis
 Types of policy actions.
– Policy inputs – the resources – time, money,
personnel, equipment, and supplies – used to
produce outputs and impacts.
– Policy processes – the administrative,
organizational, and political activities and
attitudes that shape the transformation of policy
inputs into policy outputs and impacts.
Monitoring in Policy Analysis
POLICY ACTIONS
POLICY OUTCOMES
ISSUE AREA
Inputs
Processes
Outputs
Impacts
Criminal Justice.
Dollar
expenditures for
salaries,
equipment,
maintenance.
Illegal arrests as
percentages of
total arrests.
Criminals arrested
per 100,000
known crimes.
Criminals
convicted per
100,000 known
crimes.
Municipal
Services.
Dollar
expenditures for
sanitation workers
and equipment.
Morale among
workers.
Total residences
served.
Cleanliness of
streets.
Social welfare.
Number of social
workers.
Rapport with
welfare recipients.
Welfare cases per
social worker.
Standard of living
of dependent
children.
Monitoring in Policy Analysis
 Definitions and indicators.
– Variables versus constants.
– Definitions.
• Constitutive definitions – gives meaning to words used to
describe variables by using synonymous words.
– Provide no concrete rules or guidelines for actually
monitoring changes.
• Operational definitions – gives meaning to a variable by
specifying the operations necessary to experience and measure
it.
• Indicators – directly observable characteristics.
Approaches to Monitoring
APPROACH
TYPES OF
CONTROL
TYPE OF
INFORMATION
REQUIRED
Social systems accounting
Quantitative
Available and/or new
information
Social experimentation
Direct manipulations and
quantitative
New information
Social auditing
Quantitative and/or
qualitative
New information
Research and practice
synthesis
Quantitative and/or
qualitative
Available information
Approaches to Monitoring
MANIPULABLE ACTIONS
POLICY
INPUTS
POLICY
PROCESSES
CONTROLLED OUTCOMES
POLICY
OUTPUTS
POLICY
IMPACTS
In1
P1
O1
Im1
In2
.
.
.
Ini
P2
.
.
.
Pj
O2
.
.
.
Om
Im2
.
.
.
Imn
PC1
E1
SES1
PC2
.
.
.
PCp
E2
.
.
.
Eq
SES2
.
.
.
SESr
PRECONDITIONS
UNFORESEEN
EVENTS
UNMANIPULABLE CAUSES
SIDE-EFFECTS
AND
SPILLOVERS
UNCONTROLLED EFFECTS
Social Systems Accounting
 An approach and set of methods that permit
analysts to monitor changes in objective and
subjective social conditions over time.
Social Systems Accounting
 The major analytic element of social
systems accounting is the social indicator.
– Statistics that measure social conditions and
changes therein over time for various segments
of a population. By social conditions, we mean
both the external (social and physical) and the
internal (subjective and perceptional) contexts
of human existence in a given society.
Social systems accounting
 Representative social indicators.
AREA
INDICATOR
Health and illness
Persons in state mental hospitals
Public safety
Persons afraid to walk alone at night
Education
High school graduates aged 25 and older
Employment
Labor force participation by women
Income
Percent of population below the poverty line
Housing
Households living in substandard units
Leisure and recreation
Average annual paid vacation in manufacturing
Population
Actual and projected population
Government and politics
Quality of public administration
Social values and attitudes
Overall life satisfaction and alienation
Social mobility
Change from father’s occupation
Physical environment
Air pollution index
Science and technology
Scientific discoveries
Social Systems Accounting
 Must assume that the reasons for indicator change
are related to policy actions. This is not always
the case.
 Advantages.
– Identify areas of insufficient information.
– When indicators provide reliable information, possible
to modify policies.
Social Systems Accounting
 Limitations.
– Choice of indicators influenced by values of analysts.
– Social indicators frequently do not cover manipulable
policy actions.
– Most social indicators are based available data about
objective (rather than subjective) social conditions.
– Social indicators provide little information about how
inputs are transformed into outcomes.
Social Systems Accounting
Social Systems Accounting
Social Systems Accounting
Model Summarya
Breaks in disaster patterns
1950 - 1971
1972 - 1988
1989 - 2006
a
Predictors: (Constant), Number of years
*p<.05; **p<.01; ***p<.005
R
R Square Sig.
0.322
0.103
0.705
0.497 ***
0.525
0.275 *
Coefficientsa
Breaks in
disas ter patterns
1950 - 1971
Model
1
1972 - 1988
1
1989 - 2006
1
(Cons tant)
Number of years
(Cons tant)
Number of years
(Cons tant)
Number of years
Uns tandardized
Coefficients
B
Std. Error
13.516
2.561
.340
.243
71.419
11.060
-1.551
.403
-6.745
22.133
1.218
.494
a. Dependent Variable: Total Dis as ter Declarations
Standardized
Coefficients
Beta
.322
-.705
.525
t
5.277
1.400
6.458
-3.849
-.305
2.465
Sig.
.000
.179
.000
.002
.764
.025
Social Systems Accounting
Social Experimentation
 Use of social indicators often leads to
random innovation rather than systematic
change.
 Social experimentation is the process of
systematically manipulating policy actions
in way that permits more or less precise
answers to questions about the sources of
change in policy outcomes.
Social Experimentation
 Characteristics and procedures.
– Direct control of experimental treatment.
– Comparison (control) groups.
– Random assignment.
 Goal: internal validity – the capacity of
experiments and quasi-experiments to make
valid causal inferences about the effects of
actions on outcomes.
Social Experimentation
 Threats to internal validity.
– History.
– Maturation.
– Instability.
– Instrumentation and testing.
– Mortality.
– Selection.
– Regression artifacts.
Social Experimentation
 Social experimentation is weakest in the
area of external validity or generalizability.
 Neglects policy processes.
Seattle-Denver Income Maintenance
Experiment
 A social experiment is a field test of one or
more social programs — or, to use the
phraseology of the natural sciences, a test of
one or more "treatments."
 A social experiment is a field test in the
sense that families or individuals are
actually enrolled in a pilot social program
offering some type of special benefit or
service.
Seattle-Denver Income Maintenance
Experiment
 It is "experimental" in the sense that
families or individuals are enrolled in each
of the tested programs on the basis of a
random assignment process, for example,
the flip of a coin.
Seattle-Denver Income Maintenance
Experiment
 To draw conclusions about the effects of the
treatment, it is necessary to collect
information about the people who are
enrolled in each experimental program and
about those who receive no special
treatment (called the control group), and
then to compare them on the basis of the
collected information.
Seattle-Denver Income Maintenance
Experiment
 The primary focus of SIME/DIME — the
effect of income maintenance on work
effort or labor supply — was a good subject
for experimentation .
 An increase in unearned income or a
decrease in the effective wage rate were
hypothesized to lead to decreases in labor
supply.
Seattle-Denver Income Maintenance
Experiment
 In the case of SIME/DIME, the experiment
was in fact designed to test the effect of two
different kinds of social programs on
participant work effort.
 The two policies were a variety of cash
transfer or negative income tax programs
and several combinations of job counseling
and education or training subsidy programs.
Seattle-Denver Income Maintenance
Experiment
 Any observed effects of the experiment was
interpreted as the differential effects of the
experimental treatment compared to
existing government programs.
 Thus any observed experimental-control
differences in outcomes must be interpreted
as estimates of the effect of replacing the
early 1970s status quo with the
experimental programs.
Seattle-Denver Income Maintenance
Experiment
Table 4.
Labor Supply Response of Husbands:
A. Overall NIT response
(Percentage difference in annual hours worked)
Year
Sample Group
1
2
3
4
5
3-year sample
-1.6
-7.3
-7.3
-0.5
-0.2
5-year sample
-5.9 -12.2 -13.2 -13.6 -12.3
Total Sample
-3.1
-9
-9.3 —
—
Seattle-Denver Income Maintenance
Experiment
Table 5.
Labor Supply Response of Wives:
A. Overall NIT response
(Percentage difference in annual hours worked)
Year
Sample Group
1
2
3
4
5
-4
-16.5 -15.2 -2
+13.4
3-year sample
-15.1 -26.5 -21.6 -27.1 -24
5-year sample
-8.1 -20.1 -17.4 —
—
Total Sample
Seattle-Denver Income Maintenance
Experiment
Table 6.
Labor Supply Response of Female Heads:
A. Overall NIT response
(Percentage difference in annual hours worked)
Year
Sample Group
1
2
3
4
5
-5.5 -14.1 -21.6 -8.9
-7.7
3-year sample
-7.9 -15
-21.2 -28.3 -31.8
5-year sample
-6.3 -14.3 -21.4 —
—
Total Sample
Social Auditing
 One of the limitations of social systems
accounting and social experimentation is
that both approaches neglect or
oversimplify policy processes.
Social Auditing
 Social auditing explicitly monitors relations
among inputs, processes, outputs, and
impacts in an effort to trace policy inputs
from disbursement to final recipient.
 Social auditing helps to determine whether
ineffective outcomes are the result of
inadequate inputs or processes that divert
resources or services from intended target
groups or beneficiaries.
Social Auditing
 Processes monitored of two main types:
– Resource diversion (from target or beneficiary
groups).
– Resource transformation (changes in meaning
of policy actions from administrator to
recipient).
Social Audit Steps
 The three phases of a social audit
 Phase 1: Design and data collection
– Clarify the strategic focus
– Design instruments, pilot test
– Collect information from households and key
informants in a panel of representative
communities
Social Audit Steps
 Phase 2: Evidence-based dialogue and
analysis
– Link household data with information from
public services
– Analyze findings in a way that points to action
– Take findings back to the communities for their
views about how to improve the situation
– Bring community members into discussion of
evidence with service providers/planners.
Social Audit Steps
 Phase 3: Socialization of evidence for
public accountability
– Work-shopping
– Communication strategy
– Evidence-based training of planners and service
providers
– Media training
– Partnerships with civil society
Research and practice synthesis.
 An approach to monitoring that involves the
systematic compilation, comparison, and
assessment of the results of past efforts to
implement public policies.
 Two primary sources of available information.
– Case studies of policy formulation and implementation.
– Research reports that address relations among policy
actions and outcomes.
Research and practice synthesis.
 Two methods.
– Case survey method – a set of procedures used
to identify and analyze factors that account for
variations in the adoption and implementation
of policies.
• Requires case coding scheme, a set of categories
that capture key aspects of policy inputs, processes,
outputs, and impacts.
Research and practice synthesis
 Two methods.
– Research survey method – a set of procedures
used to compare and appraise results of past
research on policy actions and outcomes.
• Yields empirical generalizations about sources of
variation in policy outcomes, summary assessments
of the confidence researchers have in these
generalizations, and policy alternatives or action
guidelines that are implied in these generalizations.
• Requires the construction of a format for extracting
information about research reports.
Research and practice synthesis
 Advantages.
– Comparatively efficient way to compile and
appraise an increasingly large body of cases
and research reports on policy implementation.
– The case survey method is one among several
ways to uncover different dimensions of policy
processes that affect policy outcomes.
– The case survey method is a good way to
examine both objective and subjective
conditions.
Research and practice synthesis
 Limitations.
– All related to reliability and validity of
information.
Research and Practice Synthesis:
Example
 David H. Greenberg; Charles
Michalopoulos; Philip K. Robins.
 Do Experimental and Nonexperimental
Evaluations Give Different Answers
about the Effectiveness of GovernmentFunded Training Programs?
Research and Practice Synthesis:
Example
 Abstract
 This paper uses meta-analysis to investigate
whether random assignment (or experimental)
evaluations of voluntary government-funded
training programs for the disadvantaged have
produced different conclusions than
nonexperimental evaluations.
 Information includes several hundred estimates
from 31 evaluations of 15 programs that operated
between 1964 and 1998.
Research and Practice Synthesis:
Example
 The results suggest that experimental and
nonexperimental evaluations yield similar
conclusions about the effectiveness of
training programs, but that estimates of
average effects for youth and possibly men
might have been larger in experimental
studies.
Research and Practice Synthesis:
Example
 The results also suggest that variation
among nonexprimental estimates of
program effects is similar to variation
among experimental estimates for men and
youth, but not for women (for whom it
seems to be larger), although small sample
sizes make the estimated differences
somewhat imprecise for all three groups.
Research and Practice Synthesis:
Example
 The policy implications of the findings are
discussed.
 © 2006 by the Association for Public Policy
Analysis and Management
Techniques for Monitoring
TABULAR
DISPLAYS
INDEX
NUMBERS
INTERRUPTED
TIMESERIES
ANALYSIS
CONTROLSERIES
ANALYSIS
REGRESSION DISCONTINUITY
ANALYSIS
APPROACH
GRAPHIC
DISPLAYS
Social systems
accounting
X
X
X
X
X
O
Social auditing
x
x
x
x
x
o
Social
experimentation
x
x
x
x
x
x
Research and
practice
synthesis
x
x
o
o
o
o