POSC 431 – Public Policy
Analysis
Introduction to Public Policy
Analysis
Introduction
 Policy analysis is a social and political activity.
– The subject matter concerns the lives and well-being of
large numbers of our fellow citizens.
– The process and results of policy analysis usually
involve other professionals and interested parties.
• Done in teams and office-wide settings.
• Immediate consumer is a client, who may be a hierarchical
superior.
• Ultimate audience includes diverse subgroups of politically
attuned supporters and opponents.
Introduction
 Policy analysis: More art than science.
– Draws on intuition as much as method.
 Eightfold path.
– Steps.
•
•
•
•
•
•
•
•
Define the problem.
Assemble some evidence.
Construct the alternatives.
Select the criteria.
Project the outcomes.
Confront the tradeoffs.
Decide!
Tell your story.
Introduction
 Eightfold path (contd.)
– Steps not necessarily taken in precisely this
order, and not all may be significant for each
problem. But serves as a starting point.
 Iteration is continual.
– The problem-solving process – being the
product of trial and error – is iterative, so that
you usually must repeat each of these steps,
sometimes more than once.
Introduction
 Iteration is continual (contd.)
– The spirit in which you take any of these steps,
especially the earliest phases of your project, should be
highly tentative.
 Some of the guidelines are practical, but most are
conceptual.
– Most concepts are obvious, but some are technical and
some are common terms used in special ways
– All concepts become intelligible through experience
and practice.
Introduction
 The concepts come embedded in concrete
particulars.
– In real life, policy problems appear as a
confusing welter of details.
– Concepts are formulated in the abstract.
– Analyst must learn to see the analytic concepts
in the concrete manifestations.
Introduction
 Your final product.
– Coherent narrative style.
– Steps.
• Describe problem.
• Lay out alternatives.
• Each course of action has projected outcomes,
supported by evidence.
• If no alternative dominates, discuss nature and
magnitude of trade-offs.
• State recommendation.
The Eightfold Path
 Define the Problem.
– Think of deficits and excesses.
– The definition should be evaluative.
– Quantify if possible.
– Conditions that cause problems are also problems.
– Missing an opportunity is a problem.
– Common pitfalls in problem definition.
• Defining the solution in the problem.
• Be skeptical about the causal claims implicit in diagnostic
problem definitions.
The Eightfold Path
 Assemble Some Evidence.
– Think before you collect.
• The value of evidence.
• Self-control.
–
–
–
–
–
Do a literature review.
Survey “best practice.”
Use analogies.
Start early.
Touching base, gaining credibility, brokering
consensus.
– Freeing the captive mind.
The Eightfold Path
 Construct the alternatives.
– Start comprehensive, end up focused.
– Model the system in which the problem is
located.
– Reduce and simply the list of alternatives.
– Design problems.
– A linguistic pitfall.
The Eightfold Path
 Select the criteria.
– Apply evaluative criteria to judging outcomes,
not alternatives.
– Criteria selection builds on problem definitions
– and continues.
– Evaluative criteria commonly used in policy
analysis
• Efficiency.
• Equality, equity, fairness, justice.
• Freedom, community, and other ideas.
The Eightfold Path
 Select the criteria (contd.)
– Weight conflicting evaluative criteria.
• The political process takes care of it.
• The analyst imposes a solution.
– Practical criteria.
• Legality.
• Political acceptability.
• Robustness and improvability.
– Criteria in optimization models.
• Linear programming.
• Improving linguistic clarity.
The Eightfold Path
 Project the Outcomes.
– Projection = Model + Evidence.
– Attach magnitude estimates.
– Break-even estimates.
– The optimism problem.
•
•
•
•
Scenario writing.
The other guy’s shoes heuristic.
Undesirable side effects.
The ethical costs of optimism.
The Eightfold Path
 Project the outcomes (contd.)
– The outcomes matrix.
– Linguistic pitfalls.
 Confront the trade-offs.
– Commensurability
• Break-even analysis revisited.
– Without projecting outcomes, there is nothing
to trade-off.
– Simplify the comparison process.
The Eightfold Path
 Decide!
– The twenty-dollar-bill test.
 Tell your story.
– The New York taxi driver test.
– You, your client, and your audiences
– What medium to use?
– Your story should have a logical narrative flow.
– Some common pitfalls.
•
•
•
•
•
Following the eightfold path.
Compulsive qualifying.
Showing your work.
Listing without explaining.
Style.
The Eightfold Path
 Tell your story (contd.)
– Report format.
• Table format.
• References and sources.
– Memo format.
– The sound bite and the press release.
Problem-Centered Policy
Analysis
POSC 431 – Public Policy
Analysis
Introduction to the Public Policy
Process
Policy Analysis in the PolicyMaking Process
 Policy analysis is the activity of creating
knowledge of and in the policy-making process.
 In creating knowledge of policy-making processes
policy analysts investigate the causes,
consequences, and performance of public policies
and programs.
 Such knowledge remains incomplete, however,
unless it is made available to policymakers and the
public they are obligated to serve.
The Process of Policy Inquiry
 This course provides a methodology for policy
analysis.
– Methodology is a system of standards, rules and
procedures for creating, critically assessing, and
communicating policy-relevant knowledge.
 Problem solving is a key element of the
methodology of policy analysis
 Policy analysis is also a methodology for
formulating problems as a part of a search for
solutions.
The Methodology of Policy
Analysis
 The methodology of policy analysis draws
from and integrates elements of multiple
disciplines
–
–
–
–
–
Political science
Sociology
Psychology
Economics
Philosophy
The Methodology of Policy
Analysis (contd.)
 Policy analysis is partly descriptive, seeking
knowledge about causes and consequences
of public policies.
 But, policy analysis is also normative,
creating and critiquing knowledge claims
about the value of public policies for past,
present,and future generations.
The Methodology of Policy
Analysis (contd.)
 Policy-relevant knowledge involves choosing
among competing values such as health, wealth,
security, peace, justice, equality, and freedom.
– To choose among competing values is not a technical
judgment, but a moral judgment. Thus, policy analysis
is a form of applied ethics.
 Policy analysis also seeks to create knowledge that
improves the efficiency of choices among
alternative policies.
The Methodology of Policy
Analysis (contd.)
 The methodology of policy analysis aims at
creating, critically assessing, and
communicating policy-relevant knowledge.
The Methodology of Policy
Analysis (contd.)
 Knowledge in this context refers to
plausibly true beliefs, as distinguished from
beliefs that are certainly true, or even true
with a particular statistical probability
– Statistical evidence performs only a
supplementary or reinforcing role.
The Methodology of Policy
Analysis (contd.)
 In short, policy analysis has developed a
core of moderately coherent underlying
theories, a variety of methods that enjoy
reasonably broad assent among
practitioners, a tradition of criticism
directed at political, ideological, and ethical
issues raised by policy analysis, and
systematic as well as anecdotal evidence of
having improved the capacities of clients to
solve problems.
The Methodology of Policy
Analysis (contd.)
 A key feature of research and analysis on
social problems over the past 40 or more
years is the growing recognition of
complexity.
– The methodological core of policy analysis can
be broadly characterized as a form of critical
multiplism.
The Methodology of Policy
Analysis (contd.)
 For critical multiplism, inductive
plausibility, not certainty, is the defining
characteristic of knowledge and a major
standard of success in policy inquiry.
– The analyst does not enumerate supporting or
confirming cases but identifies, evaluates, and
eliminates competing explanations.
The Methodology of Policy
Analysis (contd.)
 The other major standard is policy
relevance.
– Knowledge that assists in formulating and
solving problems, as these problems are
experienced by policymakers and citizens on
whom the policies have an impact, including
the disenfranchised.
The Methodology of Policy
Analysis (contd.)
 General guidelines for policy inquiry under critical
multiplism.
– Multiple operationalism.
• The use of multiple measures of policy constructs.
– Multimethod research.
• The use of multiple methods to observe policy processes and
outcomes.
– Multiple analytic synthesis.
• The synthesis and critical assessment of available analyses of
similar policies and programs.
– Multivariate analysis.
• The inclusion of multiple variables in policy models.
The Methodology of Policy
Analysis (contd.)
 General guidelines for policy inquiry under critical
multiplism (contd.)
– Multiple stakeholder analysis.
• The investigation of interpretive frameworks and perspectives
of multiple policy stakeholders.
– Multiple perspective analysis.
• The incorporation into policy analysis of multiple perspectives
– ethical, political, organizational, economic, social, cultural,
psychological, and technical.
– Multimedia communications.
• The use by policy analysts of multiple communications media
is essential for ensuring that knowledge is policy relevant.
The Methodology of Policy
Analysis (contd.)
 Given typical time and financial constraints,
it is usually impossible to observe all of
these guidelines.
 In addition, depending on the stage of the
policy process being analyzed, several
guidelines may not be relevant.
 Multiple methods makes analysts less likely
to commit errors resulting from limited
perspectives.
Policy-Relevant Information
 The methodology of policy analysis designed to
answer five questions:
– What is the nature of the problem?
• Policy problems.
– What present and past policies have been established to
address the problem, and what are their outcomes?
• Policy outcomes.
– How valuable are these outcomes in solving the
problem?
• Policy performance.
Policy-Relevant Information
 The methodology of policy analysis
designed to answer five questions:
– What policy alternatives are available to
address the problem, and what are their likely
future outcomes?
• Policy futures.
– What alternatives should be acted on to solve
the problems?
• Policy actions.
Five types of policy relevant
information.
POLICY
PERFORMANCE
POLICY
OUTCOMES
POLICY
PROBLEMS
POLICY
ACTIONS
POLICY
FUTURES
Policy-Analytic Procedures
 Methodology of policy analysis incorporates
standards, rules, and procedures.
– The standards and rules govern the selection of
procedures and the interpretation of results.
 The five procedures common to all human
problem solving:
–
–
–
–
–
Definition
Prediction
Prescription
Description
Evaluation
Policy-Analytic Procedures
 Five procedures as applied to policy analysis:
– Problem structuring (definition)
• Yields information about the conditions giving rise to a policy
problem.
– Forecasting (prediction)
• Supplies information about future consequences of acting on
policy alternatives, including doing nothing.
– Recommendation (prescription)
• Provides information about the relative value or worth of these
future consequences in solving or alleviating the problem.
Policy-Analytic Procedures
– Monitoring (description)
• Yields information about the present or past
consequences of acting on policy alternatives.
– Evaluation (evaluation)
• Provides information about the value or worth of
these consequences in solving or alleviating the
problem.
Policy-Analytic Procedures
 The five procedures serve as the means of
organizing particular methods and techniques of
analysis.
– Methods are general procedures for producing and
transforming policy relevant information.
– Each of the methods is supported by several techniques.
 The complete framework for problem-centered
policy analysis combines policy-relevant
information transformed by policy analytic
procedures.
Five policy-analytic procedures
Forecasting
Evaluation
Problem
Structuring
Problem
Structuring
Problem
Structuring
Problem
Structuring
Monitoring
Recommendation
Problem-centered policy analysis
POLICY
PERFORMANCE
Forecasting
Evaluation
Problem
Structuring
POLICY
OUTCOMES
POLICY
PROBLEMS
Problem
Structuring
Problem
Structuring
POLICY
FUTURES
Problem
Structuring
Monitoring
Recommendation
POLICY
ACTIONS
Phases of the policy-making
process
PHASE
CHARACTERISTICS
ILLUSTRATION
AGENDA SETTING
Elected and appointed officials place
problems on the public agenda. Many
problems are not acted on at all, while
others are addressed only after long
delays.
A state legislator and her cosponsor
prepare a bill that goes to the Health and
Welfare Committee for study and
approval. The bill stays in committee and
is not voted on.
POLICY
FORMULATION
Officials formulate alternative policies to
deal with a problem. Alternative policies
assume the form of executive orders, court
decisions, and legislative acts.
A state court considers the use of
standardized achievement tests such as the
SAT on grounds that the tests are biased
against women and minorities.
POLICY ADOPTION
A policy alternative is adopted with the
support of a legislative majority,
consensus among agency directors, or a
court decision.
In Roe v. Wade Supreme Court justices
reach a majority decision that women have
the right to terminate pregnancies through
abortion.
POLICY
IMPLEMENTATION
An adopted policy is carried out by
administrative units which mobilize
financial and human resources to comply
with the policy.
The city treasurer hires additional staff to
ensure compliance with a new law which
imposes taxes on hospitals that no longer
have tax-exempt status.
POLICY
ASSESSMENT
Auditing and accounting units in
government determine whether executive
agencies, legislatures, and courts are in
compliance with statutory requirements of
a policy and achieving its objectives.
The General Accounting Office monitors
social welfare programs such as
Temporary Assistance to Needy Families
(TANF) to determine the scope of welfare
fraud.
Policy analytic procedures and
phases of policy-making
Problem
Structuring
Forecasting
Recommendation
AGENDA SETTING
POLICY
FORMULATION
POLICY ADOPTION
Monitoring
POLICY
IMPLEMENTATION
Evaluation
POLICY
ASSESSMENT
Problem Structuring
 Problem structuring can supply policy-relevant
knowledge that challenges the assumptions
underlying the definition of problems reaching the
policy-making process through agenda-setting.
 Problem structuring can assist in discovering
hidden assumptions, diagnosing causes, mapping
possible objectives, synthesizing conflicting
views, and designing new policy options.
Forecasting
 Forecasting can provide policy-relevant knowledge
about future states of affairs which are likely to occur
as a consequence of adopting alternatives, including
doing nothing, that are under consideration at the
phase of policy formulation.
 Forecasting can examine plausible, potential, and
normatively valued futures, estimate the consequences
of existing and proposed policies, specify probable
future constraints on the achievement of objectives,
and estimate the political feasibility (support and
opposition) of different options.
Recommendation
 Recommendation yields policy-relevant
knowledge about the benefits and costs of
alternatives the future consequences of which have
been estimated through forecasting, thus aiding
policymakers in the policy adoption phase.
 Recommendation helps estimate levels of risk and
uncertainty, identify externalities and spillovers,
specify criteria for making choices, and assign
administrative responsibility for implementing
policies.
Monitoring
 Monitoring provides policy-relevant knowledge
about the consequences of previously adopted
policies, thus assisting policy-makers in the policy
implementation phase.
 Monitoring helps assess the degree of compliance,
discover unintended consequences of policies and
programs, identify implementational obstacles and
constraints, and locate sources of responsibility for
departures from policies.
Evaluation
 Evaluation yields policy-relevant knowledge about
discrepancies between expected and actual policy
performance, thus assisting policymakers in the
policy assessment phase of the policy-making
process. Evaluation not only results in
conclusions about the extent to which problems
have been alleviated; it also may contribute to the
clarification and critique of values driving a
policy, aid in the adjustment or reformulation of
policies, and establish a basis for restructuring
problems.
POSC 431 Public Policy
Analysis
The Functions of Policy Argument
Introduction
 Policy argumentation is central to policy
analysis and the policy-making process.
 The analysis and evaluation of policy
argumentation are central to the process of
critical thinking.
The Structure of Policy Arguments
 A policy argument is the product of
argumentation, which is the process.
 In real-life policy settings, arguments are complex
and prone to misunderstanding.
 To minimize misunderstanding, we use the
structural model of argument developed by
Stephen Toulmin, which is designed to investigate
structures and processes of practical reasoning.
The Structure of Policy Arguments
 The conclusions of practical arguments are
always uncertain, as are the reasons and
evidence that lead to these conclusions.
The Structure of Policy Arguments
 Types of knowledge claims.
– A knowledge claim is the conclusion of a
policy argument.
– Three types of knowledge claims:
• Designative: questions of fact. What are the
observed outcomes of a policy and why did they
occur?
• Evaluative: questions of value. Was the policy
worthwhile?
• Advocative: questions of right action. Which policy
should be adopted?
The Structure of Policy Arguments
 Types of knowledge claims (contd.).
– Policy arguments contain six elements:
• Information (I), Claim (C), Warrant (W), Backing (B), Rebuttal (R),
Qualifier (Q).
– The first four of these elements are present in every policy
argument.
– The claim C is the conclusion or output of an argument,
which is supported by policy-relevant information I, which
is the beginning or input of the argument.
– The warrant W is the justification, or reason, for concluding
C from I.
– The qualifier Q indicates that C has a given truth or
plausibility.
The Structure of Policy Arguments
 Example:
– The Senator supports the privatization of the
federal highway system, which have significant
gains in efficiency and a reduction in taxes.
Considering that the privatization of public
services has been successful in other areas, this
is definitely a “no brainer”. Besides this is the
same conclusion as a panel of experts on
privatization.
The Structure of Policy Arguments
 The underdetermination of conclusions by
information.
– Policy-relevant information does not fully determine
the conclusions of policy arguments. “Information does
not speak for itself.”
– Identical information can and often does lead to
different conclusions, which we call policy claims to
emphasize the fallible and indeterminate character of
arguments.
The Structure of Policy Arguments
 Example: Policy-relevant information from the Coleman
Report “Black students attending primarily black schools
had lower achievement test scores than black students
attending primarily white schools.”
– Designative claim and qualifier: “Since schools in large
urban areas are primarily black, the hopes of blacks for
higher education achievement [simply] cannot be realized.”
– Evaluative claim and qualifier: “The Coleman Report is
[clearly] a racist document based on ethnically biased
achievement tests.”
– Advocative claim and qualifier: “[There is no question] that
a national policy of compulsory school busing ought to be
adopted to achieve integrated schools.”
The Structure of Policy Arguments
 Warrants and rebuttals.
– Although each of the claims about the Coleman
report begins with the same information, very
different conclusions are drawn.
– Differences are due not to the information, but
to the role of the warrants in justifying
(plausibility or implausibility) the claims on the
basis of the information supplied.
Modes of Policy Argumentation
 Modes of policy argumentation are the characteristic routes
followed by information as it is transformed into policy
claims.
 The several different modes of argument involve reasoning
from authority, method, generalization, classification,
intuition, cause, sign, motivation, analogy, parallel case,
and ethics.
 Each of the eleven modes of argument has a different type
of warrant, and multiple modes can be found in any policy
argument.
 The warrants are the reasons offered by the proponent or
opponent of a policy to justify a claim, or inference, based
on the information supplied.
Modes of Policy Argumentation
 Authority
– Reasoning from authority is based on warrants
having to do with the achieved or ascribed
statuses of producers of policy-relevant
information. For example, experts, insiders,
scientists, specialists, gurus, power brokers.
Footnotes and references are disguised
authoritative arguments.
Argumentation from Authority
Modes of Policy Argumentation
 Method
– Reasoning from method is based on warrants
about the approved status of methods or
techniques used to produce information. The
focus is on the achieved or ascribed status or
"power" of procedures, rather than persons.
Examples include approved statistical,
econometric, qualitative, ethnographic, and
hermeneutic methods and techniques.
Argumentation from Method
Modes of Policy Argumentation
 Generalization
– Reasoning from generalization is based on similarities
between samples and populations from which samples
are selected. Although samples can be random,
generalizations can also be based on qualitative
comparisons. The assumption is that what is true of
members of a sample will also be true of members of
the population not included in the sample. For example,
random samples of n 30 are taken to be representative
of the (unobserved and often unobservable) population
of elements from which the sample is drawn.
Argumentation from Generalization
Modes of Policy Argumentation
 Classification
– Reasoning from classification has to do with
membership in a defined class. The reasoning is
that what is true of the class of persons or
events described in the warrant is also true of
individuals or groups which are members of the
class described in the information. An example
is the untenable ideological argument that
because a country has a socialist economy it
must be undemocratic, because all socialist
systems are undemocratic.
Argumentation from Classification
Modes of Policy Argumentation
 Cause
– Reasoning from cause is about the activity of
generative powers ("causes") and their consequences
("effects"). For example, a claim may be made based on
general propositions, or laws, of economics that state
invariant relations between cause and effect. Other
causal claims are based on observing conditions that
must be satisfied to infer that a policy has a specified
effect. Most argumentation in the social and natural
sciences is based on reasoning from cause.
Argumentation from Cause
Argumentation from Cause
Modes of Policy Argumentation
 Sign
– Reasoning from sign is based on signs, or indicators,
and their referents. The presence of a sign indicates the
presence of an event or condition, because the sign and
what it refers to occur together. Examples are indicators
of institutional performance such as "organizational
report cards" and "benchmarks," or indicators of
economic performance such as "leading economic
indicators." Signs are not causes, because causality
must satisfy additional requirements not expected of
signs.
Argumentation from Sign
Modes of Policy Argumentation
 Motivation
– Reasoning from motivation is based on the
motivating power of goals, values, and intentions in shaping individual and collective
behavior. For example, a claim that citizens
will support the strict enforcement of pollution
standards might be based on reasoning that,
since citizens are motivated by the desire to
achieve the goal of clean air and water, they
will act to offer their support.
Argumentation from Motivation
Modes of Policy Argumentation
 Intuition
– Reasoning from intuition is based on the
conscious or preconscious cognitive, emotional,
or spiritual states of producers of policyrelevant information. For example, the
awareness that an advisor has some special
insight, feeling, or "tacit knowledge" may serve
as a reason to accept his judgment.
Argumentation from Intuition
Modes of Policy Argumentation
 Analogy-Metaphor
– Reasoning from analogies and metaphors is based on
similarities between relations found in a given case and
relations characteristic of a metaphor, analogy, or
allegory. For example, the claim that government
should "quarantine" a country by interdicting illegal
drugs—with the illegal drugs seen as an "infectious
disease"—is based on reasoning that, since quarantine
has been effective in cases of infectious diseases,
interdiction will be effective in the case of illegal drugs.
Argumentation from Analogy
Modes of Policy Argument
 Parallel Case
– Reasoning from parallel case is based on
similarities among two or more cases of policy
making. For example, a reason that a local
government should strictly enforce pollution
standards is that a parallel policy was
successfully implemented in a similar local
government elsewhere.
Argumentation from Parallel Case
Modes of Policy Argument
 Ethics
– Reasoning from ethics is based on the rightness or
wrongness, goodness or badness, of policies or their
consequences. For example, policy claims are
frequently based on moral principles stating the
conditions of a "just" or "good" society, or on ethical
norms prohibiting lying in public life. Moral principles
and ethical norms go beyond the values and norms of
particular individuals or groups. In public policy, many
arguments about economic benefits and costs involve
unstated or implicit moral and ethical reasoning.
Ethical Argumentation
Systems of Argumentation
 Completeness. Elements of an argument should
comprise a genuine whole that encompasses all
appropriate considerations. For example, the
plausibility of arguments about the effects of a
policy depends on whether such arguments
encompass a full range of plausible rival
explanations similar in form and content to classes
of rival hypotheses (threats to validity) developed
in the tradition of quasi-experimentation.
Systems of Argumentation
 Consonance. Elements of an argument
should be internally consistent and
compatible. For example, ethical arguments
concerning the justice or fairness of a policy
are plausible to the degree that they
incorporate a system of internally and
externally consistent ethical hypotheses.
Systems of Argumentation
 Cohesiveness. Elements of an argument
should be operationally connected. For
example, the plausibility of an ethical
argument depends on whether responses to
several levels of descriptive and valuative
questions – levels ranging from verification
and validation to vindication – are
operationally linked.
Systems of Argumentation
 Functional regularity. Elements of an argument
should conform to an expected pattern. For
example, statistical arguments that offer estimates
of parameters of unobserved (and often
unobservable) populations are plausible to the
degree that patterns in the sample and the
population from which it is drawn are functionally
regular or uniform, not irregular, based on sample
data and background knowledge.
Class Example: Excerpt from Bush
Speech on Iraq
 It is clear that we need to change our strategy in Iraq. So
my national security team, military commanders, and
diplomats conducted a comprehensive review. We
consulted Members of Congress from both parties, allies
abroad, and distinguished outside experts. We benefited
from the thoughtful recommendations of the Iraq Study
Group a bipartisan panel led by former Secretary of State
James Baker and former Congressman Lee Hamilton. In
our discussions, we all agreed that there is no magic
formula for success in Iraq. And one message came
through loud and clear: Failure in Iraq would be a disaster
for the United States.
Class Example: Excerpt from Bush
Speech on Iraq
 The consequences of failure are clear: Radical Islamic
extremists would grow in strength and gain new recruits.
They would be in a better position to topple moderate
governments, create chaos in the region, and use oil
revenues to fund their ambitions. Iran would be
emboldened in its pursuit of nuclear weapons. Our enemies
would have a safe haven from which to plan and launch
attacks on the American people. On September the 11th,
2001, we saw what a refuge for extremists on the other
side of the world could bring to the streets of our own
cities. For the safety of our people, America must succeed
in Iraq.
Class Example: Excerpt from Bush
Speech on Iraq
 Many are concerned that the Iraqis are becoming too
dependent on the United States and therefore, our policy
should focus on protecting Iraq's borders and hunting
down al Qaeda. Their solution is to scale back America's
efforts in Baghdad or announce the phased withdrawal of
our combat forces. We carefully considered these
proposals. And we concluded that to step back now would
force a collapse of the Iraqi government, tear that country
apart, and result in mass killings on an unimaginable scale.
Such a scenario would result in our troops being forced to
stay in Iraq even longer, and confront an enemy that is
even more lethal.
POSC 431- Public Policy
Analysis
Problem Structuring
Nature of Policy Problems
 Problem structuring, which is a continuously
recurring phase of policy inquiry in which analysts
search among competing problem formulations of
different stakeholders, is no doubt the most
important activity performed by policy analysts.
 It is so important because policy analysts seem to
fail more often because they solve the wrong
problem than because they get the wrong solution
to the right problem.
Nature of Policy Problems
 Beyond problem solving.
– Policy analysis is a dynamic, multilevel process
in which methods of problem structuring take
priority over methods of problem solving (see
figure).
Priority of Problem Structuring in
Policy Analysis
Problem
Sensing
PROBLEM
SITUATION
Problem
Structuring
Problem
Dissolving
POLICY
PROBLEM
NO
Problem
Unsolving
RIGHT
PROBLEM?
YES
Problem
Solving
POLICY
SOLUTION
YES
POLICY
SOLUTION?
Problem
Resolving
Nature of Policy Problems
 Beyond problem solving.
– Distinctions among problem-related processes.
• Problem sensing versus problem structuring.
• Problem structuring versus problem solving.
• Problem resolving versus problem unsolving and problem
dissolving.
 Characteristics of problems.
– Interdependence of policy problems (policy messes).
• Analytic versus holistic approaches.
– Subjectivity of policy problems.
– Artificiality of policy problems.
– Dynamics of policy problems.
Nature of Policy Problems
 Characteristics of problems (contd.)
– In short, systems of problems (messes) cannot be
decomposed into independent subsets without running
the risk of producing the right solution to the wrong
problem.
– The whole is greater (qualitatively different) than the
sum of its parts.
– A recognition of the interdependence, subjectivity,
artificiality, and dynamics of policy problems alerts us
to the possible unanticipated consequences that may
follow from policies based on the right solution to the
wrong problem.
Nature of Policy Problems
 Problems versus issues.
– If policy problems are really systems of
problems, then policy issues are equally
complex.
– Policy issues reflect not only conflict over
courses of actions, but over definitions of the
problem.
Nature of Policy Problems
Major Issues
Secondary Issues
Functional Issues
Minor Issues
Nature of Policy Problems
 Problems versus issues.
– Major issues.
• Those encountered at highest levels of government
within and between federal, state, and local
jurisdictions.
• Involve questions of agency mission.
– Secondary issues.
• Located at the level of agency programs at the
federal, state, and local levels.
• The setting of program priorities and the definition
of target groups and beneficiaries.
Nature of Policy Problems
 Problems versus issues.
– Functional issues.
• Located at both the program and project levels.
• Involve questions of budget, finance, and
procurement.
– Minor issues.
• Located at the level of specific projects.
• Involve questions of personnel, staffing, employee
benefits, vacation times, working hours, and
standard operating procedures.
Nature of Policy Problems
 Problems versus issues.
– Strategic policies are policies where the
consequences are relatively irreversible.
– Operational polices are policies where the
consequences are relatively reversible.
Nature of Policy Problems
 Three classes of policy problems.
– Well-structured.
• Prototype: completely computerized decision
problems.
– Moderately structured.
• Prototype: prisoner’s dilemma.
– Ill-structured.
• Prototype: most important problems.
Nature of Policy Problems
STRUCTURE OF PROBLEM
Moderately
Structured
ELEMENT
Well structured
Ill Structured
Decision maker(s)
One or few
One or few
Many
Alternatives
Limited
Limited
Unlimited
Utilities (values)
Consensus
Consensus
Conflict
Outcomes
Certainty or risk
Uncertainty
Unknown
Probabilities
Calculable
Incalculable
Incalculable
Problem Structuring in Policy
Analysis
 The requirements for solving ill-structured
problems demand that the analyst take an
active role in defining the problem.
Problem Structuring in Policy
Analysis
 Creativity in problem structuring.
– The product of the analysis is sufficiently novel that
most people could not or would not have arrived at the
same solution;
– The process of analysis is sufficiently unconventional
that it involves the modification or rejection of previous
accepted ideas;
– The process of analysis requires sufficiently high
motivation and persistence that analysis takes place
with high intensity or over long periods of time;
Problem Structuring in Policy
Analysis
 Creativity in problem structuring (contd.).
– The product of analysis is regarded as valuable
by analysts, policymakers, and other
stakeholders, since it provides an appropriate
solution to the problem; And.
– The problem initially posed is so ambiguous,
value, and ill defined that part of the task is to
formulate the problem itself.
Problem Structuring in Policy
Analysis
 Phases of problem structuring.
– Problem search.
– Problem definition.
– Problem specification.
– Problem sensing.
Problem Structuring in Policy
Analysis
METAPROBLEM
Problem
Definition
Problem
Search
PROBLEM
SITUATION
SUBSTANTIVE
PROBLEM
Problem
Sensing
Problem
Specification
FORMAL PROBLEM
Problem Structuring in Policy
Analysis
 Errors of the third type (EIII).
– How well do the substantive and formal problems
correspond to the original problem situation?
• If most problem situations are messes, then models should
reflect complexity.
– Types of errors.
• Type I – rejecting the null hypothesis when it is true.
• Type II – accepting the null hypothesis when it is false.
• Type III – solving the wrong problem.
Types of Policy Models
 Policy models are simplified representations
of selected aspects of a problem situation
constructed for specific purposes.
– By definition, they are artificial constructs.
 Descriptive models.
– The purpose of descriptive models is to explain
and/or predict the causes and consequences of
policy choices.
• Used to monitor the outcomes of policy actions and
to forecast performance.
Types of Policy Models
 Normative models.
– Explain and predict, but also to provide rules
and recommendations for optimizing some
utility or value.
– Example: compound interest.
S n  (1  r ) S 0
n
Forms of Policy Models
 Verbal models.
– Expressed in everyday language.
– The equivalent of substantive problems.
– Limitation: the reasons for recommendations and
predictions may be hidden.
 Symbolic models.
– Use mathematical symbols to describe relationships
among key variables believed to characterize a
problem. The premises must be made explicit.
Y  a  bX
Adjusted income 2000
Forms of Policy Models
Table 1. Mean education and mean income
Birmingham, Alabama neighborhoods
50000
40000
30000
20000
10000
0
Rsq = 0.6369
0.0
.1
.2
.3
.4
.5
.6
.7
.8
Proportion with high school or higher
.9
1.0
Forms of Policy Models
 Procedural models.
– Represent dynamic relationships among
variables believed to characterize a policy
problem.
– Example:
• Decision tree.
Methods of Problem Structuring
 Boundary analysis (used to estimate boundaries of
metaproblem).
– Saturation sampling.
– Elicitation of problem representations.
– Boundary estimation.
 Classificational analysis.
– Uses logical division and logical classification.
– Criteria.
•
•
•
•
•
Substantive relevance.
Exhaustiveness.
Disjointness.
Consistency.
Hierarchical distinctiveness.
Methods of Problem Structuring
 Hierarchy analysis.
– Possible clauses, plausible causes, and
actionable causes.
– Uses same rules as classificational analysis.
 Synectics.
– Personal analogies.
– Direct analogies.
– Symbolic analogies.
– Fantasy analogies.
Methods of Problem Structuring
 Brainstorming.
– Groups should be composed of knowledgeable subjects.
– Idea generation and idea evaluation should be kept
separate.
– Atmosphere should be open and permissive.
– Idea-evaluating should only begin after idea-generating
has ceased.
– At the end of idea-evaluating, ideas should be
prioritized and incorporated into a proposal that
contains a conceptualization of the problem and its
potential solutions.
Methods of Problem Structuring
 Multiple perspective analysis.
– Technical perspective.
– Organizational perspective.
– Personal perspective.
 Assumptional analysis.
– Stakeholder identification.
– Assumption surfacing.
– Assumption challenging.
– Assumption pooling.
– Assumption synthesis.
Methods of Problem Structuring
 Argumentation mapping.
– Assessing probability and plausibility of policy
argument warrants.
PPA 503 – The Public
Policy-Making Process
Lecture 2a - The Context of Public
Policy: Values and Environment
Historical Development of Public
Policy
 The history of American politics and policy
is characterized by considerable change.
 The policy history of the United States is
characterized by “policy restraint”.
 The history is divided into four eras:
–
–
–
–
A period of divided power,
An era of state activism,
An era of national activism, and
An era of national standards.
Historical Development of Public
Policy
 Divided power (1787-1870).
– Weakness of national government under Articles of
Confederation, Shays’ Rebellion.
– The Constitution placed limits on scope of federal
government, but was still much stronger than the
government under the Articles. Also contained
structural impediments to radical policy.
• Division of powers and separation of powers.
• Policy conflicts tend to turn on jurisdictional questions as
much as policy questions. (Which level of government as
important as what to do).
Historical Development of Public
Policy
 Divided Power (contd.)
– Powers granted to Congress fell into two categories: management
of national responsibilities (defense) and commercial
responsibilities (interstate commerce).
– Powers broadened by “necessary and proper” clause”.
– In general, however, from the perspective of the average citizen,
the federal government was not the most important official
participant in policy making.
– Reinforced by structure, Founder’s understanding, rural nature of
the population, sparseness and homogeneity, libertarian political
philosophy, absence of industrialization.
– Embodied by conflict between Hamilton and Jefferson. History
supported Hamilton.
Historical Development of Public
Policy
 State activism (1870 to 1933).
– Industrialization and attendant increases in wealth,
poverty, crime, disease.
– Increased attempts at state regulation, but difficult for
individual states. Interstate commerce suggested
federal intervention, but federal government reluctant.
• Did pass Sherman Anti-Trust Act in 1890. Clayton Act 1914,
Pure Food and Drug Act. Many trust breakups. (Now remerging).
• Federal government supported laissez-faire economics in its
court and legislative decisions (Lochner vs. New York 1905).
– Integration of former slaves into society 1865 to 1880s.
Declined thereafter (Plessy vs. Ferguson 1896). Turned
control of issue over to states.
Historical Development of Public
Policy
 National activism (1933-1961).
– Demands triggered by Great Depression.
– Lochner rule influenced Supreme Court actions
on New Deal until 1937.
– New Deal created modern system of national
activism. Further enhanced by federal
government activity during World War II.
– National Defense system in 1950s.
Historical Development of Public
Policy
 National standards (1961 to 1981).
– Great Society.
– Scientific study of public policy impelled by
federal government’s efforts to set standards for
states and localities.
– Retrenchment caused by Vietnam and
Watergate restrictions on President.
• Budget and Impoundment Control Act.
• War Powers Act.
Historical Development of Public
Policy
 The end of big government? (1981-present).
– The election of Ronald Reagan.
– The Reagan Revolution changed the tenor of
American politics and signaled a realignment.
– These eras suggest that the American
constitution without substantial change is
flexible enough to bend, but not break in the
face of major policy upheavals.
Stability in American Politics and
Policy Making
Table 2.1. Elements of American Stability
Type
What this means
Examples in action
Ideological stability
Americans tend not to stray from
a set of ideological precepts
based largely on our national
experience.
No labor party because of some
suspicion of class warfare.
Political stability
Politics in the U.S. tends to be
fairly stable for extended periods.
Our constitutional structure has
changed little. 27 amendments.
Policy stability
Policies tend to change very little
over time.
The gradual evolution of Social
Security over sixty years.
Stability in power
Changes in power do not cause
major policy, political, or social
upheavals.
The transition from one President
or Congress to another is
generally very smooth
Source: Derived from James A. Anderson, Public Policymaking, 4th ed.
Stability in American Politics and
Policy Making
 Ideological stability means that Americans and their
representatives have not been quick to shift their basic political
beliefs. Since the founding and even before, Americans have
believed in personal liberty and equality (except for those outside
the mainstream), a generally limited government, “popular
sovereignty”, the rule of law, and respect for market economics,
private property, and free enterprise.
– Although attitudes have changed on a multitude of issues including
slavery, racial discrimination, the rights of women and children, and
voting rights, the core values have remained the same.
– The acceptable range for ideology is much narrower in the U.S. than
in Europe.
• Reinforced by the single member district and the two-party system.
– Some beliefs can be contradictory. Equality and capitalism, for
example.
Stability in American Politics and
Policy Making
 Political stability is a key element of our overall
national stability. We have operated under the
same constitutional arrangement since 1789
(oldest continuously operating system with a
written constitution).
– Although many more groups have been included, the
basic rules have remained the same.
• Structure that promotes a two-party system, Presidential
election by electoral college, Senate based on state
representation, House apportioned to population.
• Reinforced by stable beliefs.
Stability in American Politics and
Policy Making
 Policy stability – Political stability produces
policy stability.
– American government and constitutional system not
designed to be quickly responsive to national needs or
desires. Checks and balances, separation of powers
• New Deal conflict, and court packing plan example.
• Clinton’s health plan examples.
– The federal system also operates to delay policy
change.
• Laboratories of innovation versus status quo orientation.
 In many senses, the constitution a
counterrevolutionary document. The Civil War
may have been more revolutionary.
Policy Restraint and Barriers to
Change
 One should not conclude that policy stability is solely the
result of the constitutional structure, or that policy change
can happen. It is just very difficult.
 Barriers to change.
– Ideological and political stability. Shared power, high barriers to
amendment, separation of powers, checks and balances.
– Basic rules and norms.
• The Senate filibuster
• Seniority
• Congressional procedural rules
– Open government and policy restraint.
• Open public meeting laws, Administrative Procedure Act, and the
Freedom of Information Act open up the process for public scrutiny
and by side effect slow down the policy process.
Policy Restraint and Barriers to
Change
 A rationale for stability.
– Deliberation and public participation are at least as
important as rapid and efficient policymaking.
– Rapid policy change can happen with the right
combination of political factors, but during normal
periods, minorities can block policy change and the
system preserves their right to do so.
– Policy does not change rapidly because most of the
public does not support such change except under rare
circumstances.
Policy Restraint and Barriers to
Change
 Fragmentation.
– Fragmentation is a double-edge sword.
• On the one hand, it requires multiple serial
majorities to promote change.
• On the other hand, it offers multiple points of access
to the policy process.
– Dimensions of fragmentation.
• Separation of powers.
• Division of powers (federalism).
Policy Restraint and Barriers to
Change
Table 2.2. The Balance of Power
Function
Congress
President
Courts
Legislative
Make laws
Recommend laws; veto
laws; make regulations
that have the force of
law.
Review laws to
determine legislative
intent, new
interpretations
Executive
Override vetoes;
legislative vetoes of
regulations
Enforce and implement
laws.
Review executive acts;
restrain executive
actions.
Judicial
Impeach judges and the
president, call witnesses
to hearings, set judicial
jurisdiction.
Pardon criminals,
nominate judges
Interpret laws.
Note: The primary function of each branch is indicated in the boxes with the diagonal lines.
Core Values and Beliefs in
American System
 Fundamental Beliefs
– Life The individual's right to life should be
considered inviolable except in certain highly
restricted and extreme circumstances, such as the use
of deadly force to protect one's own or others' lives.
Core Values and Beliefs in
American System
 Fundamental Beliefs
– Liberty The right to liberty is considered an unalterable aspect of the
human condition. Central to this idea of liberty is the understanding that
the political or personal obligations of parents or ancestors cannot be
legitimately forced on people. The right to liberty includes personal
freedom: the private realm in which the individual is free to act, to think
and to believe, and which the government cannot legitimately invade;
political freedom: the right to participate freely in the political process,
choose and remove public officials, to be governed under a rule of law; the
right to a free flow of information and ideas, open debate and right of
assembly; and economic freedom: the right to acquire, use, transfer and
dispose of private property without unreasonable governmental
interference; the right to seek employment wherever one pleases; to
change employment at will; and to engage in any lawful economic
activity.
Core Values and Beliefs in
American System
 Fundamental Beliefs
– The Pursuit of Happiness It is the right of citizens
in the American constitutional democracy to attempt
to attain--to "pursue"--happiness in their own way, so
long as they do not infringe upon rights of others.
– Common Good The public or common good
requires that individual citizens have the
commitment and motivation--that they accept their
obligation--to promote the welfare of the community
and to work together with other members for the
greater benefit of all.
Core Values and Beliefs in
American System
 Fundamental Beliefs
– Justice People should be treated fairly in the
distribution of the benefits and burdens of society,
the correction of wrongs and injuries, and in the
gathering of information and making of decisions.
– Diversity Variety in culture and ethnic background,
race, lifestyle, and belief is not only permissible but
desirable and beneficial in a pluralist society.
Core Values and Beliefs in
American System
 Fundamental Beliefs
– Truth Citizens can legitimately demand that truth-telling as
refraining from Iying and full disclosure by government be
the rule, since trust in the veracity of government constitutes
an essential element of the bond between governors and
governed.
– Popular Sovereignty The citizenry is collectively the
sovereign of the state and holds ultimate authority over public
officials and their policies.
– Patriotism Virtuous citizens display a devotion to their
country, including devotion to the fundamental values and
principles upon which it depends.
Core Values and Beliefs in
American System
 Constitutional principles.
– Rule of Law and Judicial Review Both government
and the governed should be subject to the law.
– Separation of Powers Legislative, executive, and
judicial powers should be exercised by different
institutions in order to maintain the limitations placed
upon them.
– Representative Government The republican form of
government established under the Constitution is one in
which citizens elect others to represent their interests.
Core Values and Beliefs in
American System
 Constitutional principles.
– Checks and Balances The powers given to the different branches
of government should be balanced, that is roughly equal, so that no
branch can completely dominate the others. Branches of
government are also given powers to check the power of other
branches.
– Individual Rights Fundamental to American constitutional
democracy is the belief that individuals have certain basic rights
that are not created by government but which government should
protect. These are the right to life, liberty, economic freedom, and
the "pursuit of happiness." It is the purpose of government to
protect these rights, and it may not place unfair or unreasonable
restraints on their exercise. Many of these rights are enumerated in
the Bill of Rights.
Core Values and Beliefs in
American System
 Constitutional Principles
– Limited Government The powers of government may
not be used to restrict fundamental freedoms including
life, liberty, and property.
– Freedom of Religion There shall be full freedom of
conscience for people of all faiths or none. Religious
liberty is considered to be a natural inalienable right
that must always be beyond the power of the state to
confer or remove. Religious liberty includes the right to
freely practice any religion or no religion without
governmental coercion or control.
Core Values and Beliefs in
American System
 Constitutional Principles
– Federalism Power is shared between two sets
of governmental institutions, those of the states
and those of the central or federal authorities, as
stipulated by the Constitution.
– Civilian Control of the Military Civilian
authority should control the military in order to
preserve constitutional government.
Information Management in Public
Policy
 The environment of public policy in the United
States has changed dramatically because of the
advances in information technology.
 In 1813, Congress made provision for the
protection of state documents by requiring that
copies of important legislative documents be made
available to selected universities, state libraries,
and historical societies.
 The origin of the federal depository library
system.
Information Management in Public
Policy
 By 1945, the USGPO was doing eight
mailings per day to each of 418 depository
libraries.
 Beginning in 1980s and 1990s, the USGPO
began the shift to microfiche.
 In the 2000s, the USGPO is not the only
source of information and most
documentation is now electronic.
The Cold War: Origins of
Information Technology in the U.S.
 Colossus computer – 1943. Codebreaker.
 ENIAC – 1942—1946. Ballistics
calculations.
 UNIVAC 1 – weighed 8 tons. - Sold to
Census Bureau. First commerical sale.
– U.S. Navy and Atomic Energy Commission
bought the next two.
– First private sector purchaser – General
Electric.
The Cold War: Origins of
Information Technology in the U.S.
 Mainframe computing – 1970s
– IBM 370.
– IBM 7090, first transistorized computer.
 Personal computer – 1980s
– Apple
– IBM PC
– Ended the use of mainframe computing by
government.
The Cold War: Origins of
Information Technology in the U.S.
 Advanced Research Projects Agency
(ARPA) within Department of Defense.
– Over 12 years, major governmental initiatives
reconstructed telecommunications
infrastructure and provided foundation for the
Internet.
– Data communications technology (air defense).
– Communications satellite development.
– ARPAnet 1969 – first cooperative computer
network.
Architecture of Federal Information
Policy, 1946-1980
 Administrative Procedures Act of 1946.
– Federal Register, electronic 1993.
 Federal Records Act of 1950.
– Preserve documentation of the organization,
functions, policies, decisions, procedures, and
transactions of agency.
 The Freedom of Information Act of 1966.
– Right to public access to government
information.
Architecture of Federal Information
Policy, 1946-1980
 The Technology Assessment Act of 1972.
– Office of Technology Assessment, later superseded by
other technology agencies.
 The Federal Advisory Committee Act of 1972.
– Timely notice in Federal Register of advisory
committee meetings.
 The Privacy Act of 1974.
– Protects privacy of individuals identified in information
systems maintained by federal agencies.
Architecture of Federal Information
Policy, 1946-1980
 The Government in the Sunshine Act of 1976.
– Public entitled to fullest information on decisionmaking processes of federal agencies.
 Presidential Records Act of 1978.
– Made presidential records public rather than private.
 The Paperwork Reduction Act of 1980.
– Mandated an information resource management (IRM)
approach to federal data.
From BITNET to FIRSTGOV:
Growth of the Internet 1981-2000
 BITNET (1981) – Because It’s Time NETwork.
– Cooperative network situated at CUNY.
 NSFNet (1986) – Managed with IBM, MCI, and
Merit Network. 10,000 hosts by 1987.
 Combination of ARPANet, BITNET, and
NSFNET provided infrastructure for the Internet.
 1984 – Domain Name System (DNS) and Uniform
Resource Locators (URL).
From BITNET to FIRSTGOV:
Growth of the Internet 1981-2000
 Internet Corporation for Assigned Names and Numbers





responsible for civilian URLs, FTP, lookup files. Military
responsible for .mil, Feds responsible for .gov, and foreign
countries for the suffixes.
In 1992, federal government repealed acceptable use
policy allowing commercialization of the Internet.
1992 – Supreme Court prohibited Internet sales taxation
for companies without physical presence in state.
1993 – Federal funding of internet ends.
1995 – Concept of the “Digital Divide.”
2000 – FirstGov.gov, portal to all government information.
Policy Issues in the Information
Age, 1986-2005
 Public access to information.
 Public participation in e-government,
including public comments on federal
regulations.
 Accessibility for disabled. Rehabilitation
Act Amendments of 1986.
 Individual privacy. Matching regulated but
more match since 9/11.
Policy Issues in the Information
Age, 1986-2005
 Modernizing education. Increased technology
infrastructure.
 Regulating e-vice.
– Child Online Protection Act of 1998 (unconstitutional).
– Child Pornography Prevention Act of 1996
(unconstitutional).
– Prosecutorial Remedies and Tools Against the
Exploitation of Children Today Act of 2003
(PROTECT).
– Child Obscenity and Pornography Prevention Act of
2003
– Many more acts.
Policy Issues in the Information
Age, 1986-2005
 Securing intellectual property.
– Digital Millennium Copyright Act of 1998, extended to
digital media. Fair use for universities. Prohibited
removal of copyright management information from
digital media.
 Electronic voting.
– Shift from paper ballots, but many security issues.
 Regulating the outsourcing of IT jobs.
– Raises information security issues.
Securing E-Government, 1986 2005
 Guaranteeing security of cyberspace key issue.
 1980s mostly computer crime.
 1990s, architecture for securing cyberspace
against foreign and domestic enemies.
 Presidential Decision Directive 63 (Protecting
America’s Critical Infrastructure).
 Government Information and Security Reform Act
of 2000.
 9/11.
Securing E-Government, 1986 2005
 USA Patriot Act of 2001.
 Cyber Security and Research Act of 2002.
 Amendments to Fair Credit Reporting Act
to reduce identity theft.
 OMB – Information Systems Council to
coordinate sharing of terrorist information.
 9/11 Commission – separate, secure
network for federal information sharing.
POSC 431 – Public Policy
Analysis
Forecasting Policy Futures
Forecasting in Policy Analysis
 Forecasting is a procedure for producing factual
information about future states of society on the
basis of prior information about policy problems.
 Forms of forecasting.
– Projection.
• A forecast based on the extrapolation of current and historical
trends into the future.
– Prediction.
• A forecast based on explicit theoretical assumptions.
– Conjecture.
• A forecast based on informed or expert judgments about future
states of society.
Forecasting in Policy Analysis
 Aims of forecasting.
– Forecasts provide information about future
changes in policies and their consequences.
– Forecasting permits greater control through
understanding past policies and their
consequences, implying that the future is
determined by the past.
– Forecasting also enables us to shape the future
in an active manner, irrespective of what has
happened in the past.
Forecasting in Policy Analysis
 Limitations of forecasting.
– Forecast accuracy.
• Recent simple forecasting models have had huge errors in
recent years (as much as 50%) in econometric forecasting.
– Comparative yield.
• Both simple and complex theoretical models have been no
more accurate than simple extrapolative models and informed
expert judgment.
– Context.
• Institutional (nonprofit more accurate than business and
government).
• Temporal (long-term less accurate than short-term).
• Historical (modern complexity reduces accuracy).
– “Assumption drag”.
Forecasting in Policy Analysis
 Types of futures.
– Potential futures.
• Future societal states that may occur.
– Plausible futures.
• Future states that, on the basis of assumptions about causation
in nature and society, are believed to be likely if policymakers
do not intervene to redirect the course of events.
– Normative futures.
• Potential and plausible futures which are consistent with an
analyst’s conception of future needs, values, and opportunities.
• The specification of normative futures narrows the range of
potential and plausible futures, thus linking forecasts to
specific goals and objectives.
Forecasting in Policy Analysis
Table 1. Contrast between goals and objectives
CHARACTERISTIC
GOALS
OBJECTIVES
Broadly stated (. . . To
upgrade the quality of
health care)
Concrete (. . . Increase
the number of physicians
by 10 percent)
Definition of terms
Formal ( . . . The quality
of health care refers to
the accessibility of
medical services)
Operational ( . . . The
quality of health care
refers to the number of
physicians per 100,000
persons . . .)
Time period
Unspecified ( . . . In the
future)
Specified ( . . . In the
period 2006-2016)
Nonquantitative
(adequate health care
insurance)
Frequently quantitative ( .
. . The number persons
covered per 1,000
persons)
Broadly defined ( . . .
People in need of care)
Specifically defined ( . . .
Families with incomes
below $19,000)
Specification of purposes
Measurement procedure
Treatment of target groups
Forecasting in Policy Analysis
 Sources of goals, objectives, and alternatives.
– Authority.
– Insight.
– Method.
– Scientific theories.
– Motivation.
– Parallel case.
– Analogy.
– Ethical systems.
Approaches to Forecasting
 Approaches to forecasting.
– Decide what to forecast, or determine the object
of the forecast.
– Decide how to make the forecast, or select one
or more bases for the forecast.
– Choose techniques that are most appropriate for
the object and base selected.
Approaches to Forecasting
 Objects.
– The object of a forecast is the point of reference
of a projection, prediction, or conjecture.
– Objects of forecasting.
•
•
•
•
Consequences of existing policies.
Consequences of new policies.
Contents of new policies.
Behavior of policy stakeholders.
Approaches to Forecasting
 Bases.
– The basis of a forecast is the set of assumptions
or data used to establish the plausibility of
estimates of consequences of existing or new
policies, the content of new policies, or the
behavior of policy stakeholders.
Approaches to Forecasting
 Bases (contd.).
– Bases for forecasts.
• Trend extrapolation (projection).
– The extension into the future of trends observed in the
past.
– Based on inductive logic.
• Theoretical assumptions (prediction).
– Systematically structured and empirically testable sets of
laws or propositions that make predictions about the
occurrence of one event based on another.
– Based on deductive logic.
• Informed judgments (conjecture).
– Knowledge based on experience and insight, rather than
inductive or deductive reasoning.
– Based on retroductive logic.
Methods and Techniques of
Forecasting
APPROACH
BASIS
APPROPRIATE
TECHNIQUES
PRODUCT
Extrapolative
forecasting
Trend extrapolation
Classical time-series analysis
Linear trend estimation
Exponential weighting
Data transformation
Catastrophe methodology
Projections
Theoretical forecasting
Theory
Theory mapping
Causal modeling
Regression analysis
Point and interval estimation
Correlational analysis
Predictions
Judgmental forecasting
Informed judgment
Conventional Delphi
Policy Delphi
Cross-impact analysis
Feasibility assessment
Conjectures
Methods and Techniques of
Forecasting
 Classical time-series analysis.
– Time series are made up four components.
•
•
•
•
Secular trend.
Seasonal variation.
Cyclical fluctuations.
Irregular movements.
– Linear trend estimation.
– Nonlinear time series.
•
•
•
•
•
Oscillations.
Cycles.
Growth curves.
Decline curves.
Catastrophes.
Time – Series Example
Methods and Techniques of
Forecasting
 Linear forecasting example.
– Major disaster data set, 1953-2007.
• Disaster Declarations = 9.851 + .803 * years; R2 = .592.
– Where 1953 is set to zero.
• 1953 – 1971: Disaster Declarations = 13.516 + 0.340 * years;
R2 = .103.
• 1972 - 1988: Disaster Declarations = 71.419 - 1.551 * years;
R2 = .497.
• 1989 - 2007: Disaster Declarations = -9.368 + 1.281* years; R2
= .329.
• Projection for 2008 (55) using whole data set: 54.0.
• Projection for 2008 (55) using post-1988 set: 61.1.
Methods and Techniques of
Forecasting
 Theoretical forecasting.
– Theory mapping.
• Types of causal arguments.
– Convergent arguments.
– Divergent arguments.
– Serial arguments.
– Cyclic arguments.
• Procedures for uncovering the structure of an argument.
– Separate and number each assumption.
– Underline the words that indicate claims (“therefore”,
“thus”, “hence”).
– When specific words are omitted, but implied, supply the
appropriate logical indicators in brackets.
– Arrange number assumptions and claims in an arrow
diagram that illustrates the causal argument or theory.
Methods and Techniques of
Forecasting
 Example for theory mapping from Daniels
and Clark-Daniels.
– http://www.csub.edu/~rdaniels/Pages%20from
%20DanielsClarkDanielsIJMED.pdf.
Methods and Techniques of
Forecasting
 Causal modeling and regression analysis.
– The creation of theoretical models on the basis
of previous research and logic.
– The testing of these models with causal analysis
methods such as path analysis and structural
equation modeling.
– Example:
• http://www.csub.edu/~rdaniels/Model%20of%20Pre
sidential%20Disaster%20Decision-Making.xls.
Methods and Techniques of
Forecasting
 Judgmental forecasting.
– Delphi technique.
• Principles.
–
–
–
–
–
–
Selective anonymity.
Iteration.
Informed multiple advocacy.
Polarized statistical response.
Structured conflict.
Computer conferencing.
Methods and Techniques of
Forecasting
 Judgmental forecasting (contd.).
– Delphi technique (contd.).
• Steps.
– Issue specification.
– Selection of advocates.
– Questionnaire design.
• Forecasting items (probability of occurrence).
• Issue items (ranking of importance).
• Goal items (desirability and feasibility).
• Option items (alternatives).
– Analysis of first-round results.
– Development of subsequent questionnaires.
– Organization of group meetings.
– Preparation of final report.
Methods and Techniques of
Forecasting
 Delphi article.
– http://www.csub.edu/~rdaniels/ch3b1.pdf.
PPA 503 – The Public
Policy-Making Process
Lecture 3a – Official and Unofficial
Actors and Their Roles in Public
Policy
Introduction
 Political science traditions.
– Institutionalism – focus on texts of constitutions, laws,
and other written statements of policies and the
relationships between formal government institutions.
– Behaviorism – focus on political motivations of
individuals, acting singly and in groups, often through
polling, game theory, and statistical techniques.
– Neo-institutionalism – focus on organizations and
systems in which individuals interact and achieve
political and policy goals through explicit or implicit
rules and operating procedures.
Introduction
 Main categories of actors in the policy
process.
– Official actors – statutory or constitutional
responsibilities.
• Legislative, executive, and judiciary.
– Unofficial actors – participation with no
explicit legal authority.
• Interest groups, media.
Legislatures
 First listed branch in the federal and most
state constitutions.
– Source of considerable research.
 Primary function is lawmaking. Number of
bills and resolutions gives some idea of how
busy legislatures are.
Legislatures
Jurisdiction
House or
Assembly
Bills
House or
Assembly
Concurrent
Resolutions
House or
Assembly
Joint
Resolutions
Senate Bills
Senate
Concurrent
Resolutions
Senate Joint
Resolutions
U.S.
Congress
(2001-2002)
5,767
521
125
3,181
160
53
California
Legislature
(2001-2002)
3.283
264
66
2,277
114
56
Legislatures
 Burden eased by staff.
 Bills sifted by committee structure at both
the federal and state level.
– Committee chairs wield significant power.
– Most bills fail to move past their first
committee hurdles because they are largely
symbolic gestures.
Legislatures
 Other critical functions performed by legislators
that affect public policy.
– Casework – activities to help constituents with
government agencies or to gain a privilege or benefit.
• Supports reelection.
– Oversight – Monitor the implementation of public
policy.
• Government Accountability Office – www.gao.gov. Studies
public programs and makes recommendations to improve
efficiency, effectiveness, and accountability.
• Public hearings.
– Help understand issues.
– Reveal shortcomings in current policies.
– Make political capital.
Legislative Organization
Legislative Organization
 California process.
– http://www.leginfo.ca.gov/bil2lawx.html.
Legislative Organization
 What you see on C-SPAN does not represent the
bulk of legislative action on policy.
 Most of the critical work on public policy is done
in committees, which review legislation, propose
and vote on amendments, and, in the end, decide
whether a bill will die at the committee level or be
elevated for consideration by the full body.
 One of the most critical elements of legislative
organization is the organization on party lines.
Legislature – Critiques of Public
Policy Process
 Many people argue that legislatures are out
of touch with the people.
 To understand why legislatures work as
they do, you need to understand two
elements of the legislature: the nature of the
members of the body and the organization
and nature of the branch itself.
Legislature – Critiques of Public
Policy Process
 The primary goal of the typical legislator is
reelection. Casework allows legislators to please
voters.
– Home style and hill style.
 Legislatures are decentralized institutions,
especially Congress.
– Committees and subcommittees.
– Decentralization and centralization of party leadership.
– Issue networks and policy subsystems.
Legislatures – Implications for
Policy Making
 Decentralization and casework focus makes
complex and change-oriented legislation
difficult to pass.
The Executive Branch
 For the sake of discussion, the executive branch
can be considered in two parts: the administration,
staff, and appointees; and the bureaucracy.
 Advantages of an elected executive in the policy
process.
–
–
–
–
Veto power.
Unitary branch of government.
Media and public attention.
Informational advantage over the legislature.
The Executive Branch
 Elected executive limitations.
– “Power to persuade”.
– The size of the Executive Office of the
President.
 Elected executive’s focus on agenda-setting.
Administrative Agencies and
Bureaucrats
 Characteristics of bureaucracy.
– Fixed and official jurisdictional areas.
– Hierarchical organization.
– Written documentation.
– Expert training of staff.
– Career, full-time occupation.
– Standard operating procedures.
 Key complaints about bureaucracy.
– Size.
– Red tape.
Administrative Agencies and
Bureaucrats
 What Do Government Agencies Do?
– Government agencies provide services that are
uneconomical for the private sector (public
goods – free-rider problem).
• Public goods are indivisible and nonexclusive.
– Complaints tend to focus on speed, efficiency,
and effectiveness of public service delivery.
Administrative Agencies and
Bureaucrats
 Bureaucracy and the problem of accountability.
– The key problem is the question of accountability.
Most public employees are appointed on merit, not
accountability to elected officials.
– Early thinking focused on separation of politics and
administration.
– Modern thinking: Agency decisions are political and in
the realm of administrative discretion.
– Problem: no single, agreed-upon definition of the
public interest.
– Administrative discretion: ability to make decisions
with minimal interference.
The Courts
 The ability to interpret legislative and
executive actions: judicial review.
 Courts are the weakest because their
authority rests on the legitimacy of the law
and their ability to argue their case.
 Legislatures and executives initiate public
policy, while courts react to the practical
effects of such policies.
Unofficial Actors and Their Roles in
Public Policy
 Individual citizens.
– Low political participation.
• Voting.
• Other forms of participation: campaigning,
contacting, etc.
– Despite this, citizens can be mobilized:
• Recall election in California.
– Generally speaking, individuals want the most
services for ourselves while paying the least
taxes for those services.
Unofficial Actors and Their Roles in
Public Policy
 Interest groups.
– Interest groups have been part of the political
scene since the founding.
• Madison and the dangers of faction.
– Since the 1960s the number of groups has
greatly expanded.
• Transportation, mass communication, expansion of
government.
– Few legal barriers to the creation of groups.
Unofficial Actors and Their Roles in
Public Policy
 Interest groups.
– The power of interest groups varies.
• Knowledge, money, information.
• Group size, peak associations.
• Intensity, direct economic interest, ideological
commitment.
• Social movements (combinations of interest groups).
Unofficial Actors and Their Roles in
Public Policy
 Types of interest groups.
– Institutional versus membership groups.
– Economic (private) versus public interest versus
ideological groups.
• Benefits, free-rider problems.
 Activities of interest groups.
– Lobbying.
– Campaign contributions.
– Access (well-off).
– Mass mobilization, protest, and litigation.
– Riots and protest marches.
Unofficial Actors and Their Roles in
Public Policy
 Political parties.
– Functions.
•
•
•
•
Voting cues.
Transmission of political preferences.
Creation of packages of policy ideas.
Organization of the legislative branch.
 Think tanks and other research organizations.
– Brookings, Cato, Urban Institute, Rand, American
Enterprise Institute.
• Ideological, scholarly, and methodological distinctions.
Unofficial Actors and Their Roles in
Public Policy
 Communications media.
– The news media are important actors in the policy
process.
• Newspapers – National versus regional versus local.
• TV is the central news medium. Older population, networks;
younger population, cable news.
– Entertainment programming can be equally important.
• Movies, t.v., videogames.
– Media’s primary function in policy process is agendasetting. Media coverage correlates with institutional
attention.
Unofficial Actors and Their Roles in
Public Policy
 Communications media.
– News media are not just passive actors.
•
•
•
•
Interest try to arouse media focus.
Time and space constraints require discretion.
Profit-driven businesses.
Competitive biases of news gathering: dramatic and
narrative qualities of the story.
Unofficial Actors and Their Roles in
Public Policy
 Subgovernments, issue networks, and domains.
– Policy domain is the substantive area of policy over
which participants in policy-making compete and
compromise.
– The political culture and legal environment influence
the domains.
– Policy community inside the domain consists of the
actors actively involved in policy making in that
domain.
• Iron triangles one way of organizing the policy community.
• Issue networks may be more accurate description.
Unofficial Actors and Their Roles in
Public Policy
 Subgovernments, issue networks, and domains.
– Prying open policy networks (major corporate interests
usually dominate).
– But, policy change is possible by prying open a domain.
•
•
•
•
Focusing events.
Social movements and mobilization.
Exploiting the decentralization of American government.
Going public.
POSC 431 – Public Policy
Analysis
Recommending Policy Actions.
Recommendation in Policy
Analysis
 The policy-analytic procedure of
recommendation enables analysts to
produce information about the likelihood
that future courses of action will result in
consequences that are valuable to some
individual, group, or society as a whole.
Recommendation in Policy
Analysis
 The procedure of recommendation involves
the transformation of information about
policy futures into information about policy
actions that will result in valued outcomes.
 Policy recommendations are normative
(advocative) rather than empirical
(descriptive) or evaluative.
Recommendation in Policy
Analysis
 Characteristics of advocative claims.
– Actionable: Advocative claims focus on actions that
make be taken to resolve a policy problem.
– Prospective: Advocative claims occur prior to the time
that actions are taken.
– Value laden: Advocative claims require both that
actions have the predicted consequences, but also that
those consequences have value for society.
– Ethically complex: Advocative claims can be intrinsic
(valued as ends in themselves) or extrinsic (valued
because they will produce some other value).
Recommendation in Policy
Analysis
 Simple model of choice.
– Advocative claims are only possible when the
analyst is confronted by a situation of choice
between two or more alternatives.
– Simple model.
• The definition of a problem requiring action.
• The comparison of consequences of two or more
alternatives to resolve the problem.
• The recommendation of the alternative that will
result in a preferred outcome.
Recommendation in Policy
Analysis
 Simple model of choice.
A1
O1
A2
O2
O1
>

O2
A1
 Conditions for choice.
– Single decision maker.
– Certainty.
– Immediacy of consequences.
Recommendation in Policy
Analysis
 Complex model of choice.
– Conditions.
• Multiple stakeholders.
• Uncertainty about outcomes.
• The passage of time between actions and
consequences.
– Results.
• Intransitivity of choice.
Recommendation in Policy
Analysis
 Forms of rationality.
– Given the conditions of complex choice, there
are multiple forms of rationality.
•
•
•
•
•
Technical rationality.
Economic rationality.
Legal rationality.
Social rationality.
Substantive rationality.
Recommendation in Policy
Analysis
 Rational-comprehensive theory.
– An individual or collective decision-maker must
identify a policy problem on which there is a consensus
among all relevant stakeholders.
– An individual or collective decision-maker must define
and consistently rank all goals and objectives whose
attainment would represent a resolution of the problem.
– An individual or collective decision-maker must
identify all policy alternatives that may contribute to the
attainment of each goal and objective.
Recommendation in Policy
Analysis
 Rational-comprehensive theory.
– An individual or collective decision-maker must
forecast all consequences that will result from the
selection of each alternative.
– An individual or collective decision-maker must
compare each alternative in terms of its consequences
for the attainment of each goal and objective.
– An individual or collective decision-maker must choose
that alternative that maximizes the attainment of
objectives.
Recommendation in Policy
Analysis
 Disjointed incremental theory.
– Consider only those objectives that differ incrementally
from the status quo.
– Limit the number of consequences forecast for each
alternative.
– Make mutual adjustments in goals and objectives, on
the one hand, and alternatives on the other.
– Continuously reformulate problems – and hence goals,
objectives, and alternatives – in the course of acquiring
new information.
POSC 431 – Public Policy
Analysis
Monitoring Policy Outcomes
Introduction
 The consequences of policy actions are
never fully known in advance and, for this
reason, it is essential to monitor policy
actions after they have occurred.
 In fact, policy recommendations may be
viewed as hypotheses about the relationship
between policy actions and policy
outcomes: if action A is taken at time t1,
outcome O will result at time t2.
Monitoring in Policy Analysis
 Monitoring is the policy-analytic procedure used
to produce information about the causes and
consequences of public policies.
 Monitoring, since it permits analysts to describe
relationships between policy-program operations
and their outcomes, is the primary source of
knowledge about policy implementation.
 Monitoring is primarily concerned about with
establishing factual premises about public policy.
Monitoring in Policy Analysis
 Monitoring performs at least four major
functions:
– Compliance: monitoring helps determine whether
the actions of program administrators, staff, and
other stakeholders are in compliance with standards
and procedures imposed by legislatures, regulatory
agencies, and professional bodies.
– Auditing: monitoring helps determine whether
resources and services intended for certain target
groups and beneficiaries have actually reached
them.
Monitoring in Policy Analysis
 Monitoring performs at least four major
functions:
– Accounting: monitoring produces information
that is helpful in accounting for social and
economic changes that follow the
implementation of broad sets of public policies
and programs over time.
– Explanation: monitoring also yields
information that helps to explain why the
outcomes of public polices and programs differ.
Monitoring in Policy Analysis
 Sources of information.
– Information must be:
• Relevant.
– Macronegative versus micropositive.
• Reliable.
– Observations precise and dependable.
• Valid.
– Information about policy outcomes should measure what
we think it is measuring.
Monitoring in Policy Analysis
 Sources of information.
– Information on policy outcomes.
Historical Statistics of the United States
Current Opinion
Statistical Abstract of the United States
United States Census of Population by States
County and City Data Book
Congressional District Data Book
Census Use Study
National Opinion Research Center General Social
Survey
Social and Economic Characteristics of Students
Survey Research Center National Election Studies
Educational Attainment in the United States
National Clearinghouse for Mental Health
Information
Current Population Reports
National Clearinghouse for Drug Abuse Information
The Social and Economic Status of the Black
Population in the United States
National Criminal Justice Reference Service
Female Family Heads
Child Abuse and Neglect Clearinghouse Project
Monthly Labor Review
National Clearinghouse on Revenue Sharing
Handbook of Labor Statistics
Social Indicators
Congressional Quarterly
State Economic and Social Indicators
Law Digest
Monitoring in Policy Analysis
 Sources of information.
– When information is not available from existing
sources, monitoring may be carried out by
some combination of questionnaires,
interviewing, field observation, and the use of
agency records.
Monitoring in Policy Analysis
 Types of policy outcomes.
– Consequences:
• Policy outputs – the goods, services, or resources received by
target groups and beneficiaries.
• Policy impacts – actual changes in behavior and attitudes that
result from policy outputs.
– Populations:
• Target groups – individuals, communities, or organizations on
whom policies and programs are designed to have an effect.
• Beneficiaries – groups for whom the effects of policies are
beneficial or valuable.
Monitoring in Policy Analysis
 Types of policy actions. Policy actions have
two major purposes.
– Regulation – actions designed to ensure
compliance with certain standards or
procedures.
– Allocation – actions that require inputs of
money, time, personnel, and equipment.
– Regulative and allocative actions may have
consequences that are distributive or
redistributive.
Monitoring in Policy Analysis
 Types of policy actions.
– Policy inputs – the resources – time, money,
personnel, equipment, and supplies – used to
produce outputs and impacts.
– Policy processes – the administrative,
organizational, and political activities and
attitudes that shape the transformation of policy
inputs into policy outputs and impacts.
Monitoring in Policy Analysis
POLICY ACTIONS
POLICY OUTCOMES
ISSUE AREA
Inputs
Processes
Outputs
Impacts
Criminal Justice.
Dollar
expenditures for
salaries,
equipment,
maintenance.
Illegal arrests as
percentages of
total arrests.
Criminals arrested
per 100,000
known crimes.
Criminals
convicted per
100,000 known
crimes.
Municipal
Services.
Dollar
expenditures for
sanitation workers
and equipment.
Morale among
workers.
Total residences
served.
Cleanliness of
streets.
Social welfare.
Number of social
workers.
Rapport with
welfare recipients.
Welfare cases per
social worker.
Standard of living
of dependent
children.
Monitoring in Policy Analysis
 Definitions and indicators.
– Variables versus constants.
– Definitions.
• Constitutive definitions – gives meaning to words used to
describe variables by using synonymous words.
– Provide no concrete rules or guidelines for actually
monitoring changes.
• Operational definitions – gives meaning to a variable by
specifying the operations necessary to experience and measure
it.
• Indicators – directly observable characteristics.
Approaches to Monitoring
APPROACH
TYPES OF
CONTROL
TYPE OF
INFORMATION
REQUIRED
Social systems accounting
Quantitative
Available and/or new
information
Social experimentation
Direct manipulations and
quantitative
New information
Social auditing
Quantitative and/or
qualitative
New information
Research and practice
synthesis
Quantitative and/or
qualitative
Available information
Approaches to Monitoring
MANIPULABLE ACTIONS
POLICY
INPUTS
POLICY
PROCESSES
CONTROLLED OUTCOMES
POLICY
OUTPUTS
POLICY
IMPACTS
In1
P1
O1
Im1
In2
.
.
.
Ini
P2
.
.
.
Pj
O2
.
.
.
Om
Im2
.
.
.
Imn
PC1
E1
SES1
PC2
.
.
.
PCp
E2
.
.
.
Eq
SES2
.
.
.
SESr
PRECONDITIONS
UNFORESEEN
EVENTS
UNMANIPULABLE CAUSES
SIDE-EFFECTS
AND
SPILLOVERS
UNCONTROLLED EFFECTS
Social Systems Accounting
 An approach and set of methods that permit
analysts to monitor changes in objective and
subjective social conditions over time.
Social Systems Accounting
 The major analytic element of social
systems accounting is the social indicator.
– Statistics that measure social conditions and
changes therein over time for various segments
of a population. By social conditions, we mean
both the external (social and physical) and the
internal (subjective and perceptional) contexts
of human existence in a given society.
Social systems accounting
 Representative social indicators.
AREA
INDICATOR
Health and illness
Persons in state mental hospitals
Public safety
Persons afraid to walk alone at night
Education
High school graduates aged 25 and older
Employment
Labor force participation by women
Income
Percent of population below the poverty line
Housing
Households living in substandard units
Leisure and recreation
Average annual paid vacation in manufacturing
Population
Actual and projected population
Government and politics
Quality of public administration
Social values and attitudes
Overall life satisfaction and alienation
Social mobility
Change from father’s occupation
Physical environment
Air pollution index
Science and technology
Scientific discoveries
Social Systems Accounting
 Must assume that the reasons for indicator change
are related to policy actions. This is not always
the case.
 Advantages.
– Identify areas of insufficient information.
– When indicators provide reliable information, possible
to modify policies.
Social Systems Accounting
 Limitations.
– Choice of indicators influenced by values of analysts.
– Social indicators frequently do not cover manipulable
policy actions.
– Most social indicators are based available data about
objective (rather than subjective) social conditions.
– Social indicators provide little information about how
inputs are transformed into outcomes.
Social Systems Accounting
Social Systems Accounting
Social Systems Accounting
Model Summarya
Breaks in disaster patterns
1950 - 1971
1972 - 1988
1989 - 2006
a
Predictors: (Constant), Number of years
*p<.05; **p<.01; ***p<.005
R
R Square Sig.
0.322
0.103
0.705
0.497 ***
0.525
0.275 *
Coefficientsa
Breaks in
disas ter patterns
1950 - 1971
Model
1
1972 - 1988
1
1989 - 2006
1
(Cons tant)
Number of years
(Cons tant)
Number of years
(Cons tant)
Number of years
Uns tandardized
Coefficients
B
Std. Error
13.516
2.561
.340
.243
71.419
11.060
-1.551
.403
-6.745
22.133
1.218
.494
a. Dependent Variable: Total Dis as ter Declarations
Standardized
Coefficients
Beta
.322
-.705
.525
t
5.277
1.400
6.458
-3.849
-.305
2.465
Sig.
.000
.179
.000
.002
.764
.025
Social Systems Accounting
Social Experimentation
 Use of social indicators often leads to
random innovation rather than systematic
change.
 Social experimentation is the process of
systematically manipulating policy actions
in way that permits more or less precise
answers to questions about the sources of
change in policy outcomes.
Social Experimentation
 Characteristics and procedures.
– Direct control of experimental treatment.
– Comparison (control) groups.
– Random assignment.
 Goal: internal validity – the capacity of
experiments and quasi-experiments to make
valid causal inferences about the effects of
actions on outcomes.
Social Experimentation
 Threats to internal validity.
– History.
– Maturation.
– Instability.
– Instrumentation and testing.
– Mortality.
– Selection.
– Regression artifacts.
Social Experimentation
 Social experimentation is weakest in the
area of external validity or generalizability.
 Neglects policy processes.
Seattle-Denver Income Maintenance
Experiment
 A social experiment is a field test of one or
more social programs — or, to use the
phraseology of the natural sciences, a test of
one or more "treatments."
 A social experiment is a field test in the
sense that families or individuals are
actually enrolled in a pilot social program
offering some type of special benefit or
service.
Seattle-Denver Income Maintenance
Experiment
 It is "experimental" in the sense that
families or individuals are enrolled in each
of the tested programs on the basis of a
random assignment process, for example,
the flip of a coin.
Seattle-Denver Income Maintenance
Experiment
 To draw conclusions about the effects of the
treatment, it is necessary to collect
information about the people who are
enrolled in each experimental program and
about those who receive no special
treatment (called the control group), and
then to compare them on the basis of the
collected information.
Seattle-Denver Income Maintenance
Experiment
 The primary focus of SIME/DIME — the
effect of income maintenance on work
effort or labor supply — was a good subject
for experimentation .
 An increase in unearned income or a
decrease in the effective wage rate were
hypothesized to lead to decreases in labor
supply.
Seattle-Denver Income Maintenance
Experiment
 In the case of SIME/DIME, the experiment
was in fact designed to test the effect of two
different kinds of social programs on
participant work effort.
 The two policies were a variety of cash
transfer or negative income tax programs
and several combinations of job counseling
and education or training subsidy programs.
Seattle-Denver Income Maintenance
Experiment
 Any observed effects of the experiment was
interpreted as the differential effects of the
experimental treatment compared to
existing government programs.
 Thus any observed experimental-control
differences in outcomes must be interpreted
as estimates of the effect of replacing the
early 1970s status quo with the
experimental programs.
Seattle-Denver Income Maintenance
Experiment
Table 4.
Labor Supply Response of Husbands:
A. Overall NIT response
(Percentage difference in annual hours worked)
Year
Sample Group
1
2
3
4
5
3-year sample
-1.6
-7.3
-7.3
-0.5
-0.2
5-year sample
-5.9 -12.2 -13.2 -13.6 -12.3
Total Sample
-3.1
-9
-9.3 —
—
Seattle-Denver Income Maintenance
Experiment
Table 5.
Labor Supply Response of Wives:
A. Overall NIT response
(Percentage difference in annual hours worked)
Year
Sample Group
1
2
3
4
5
-4
-16.5 -15.2 -2
+13.4
3-year sample
-15.1 -26.5 -21.6 -27.1 -24
5-year sample
-8.1 -20.1 -17.4 —
—
Total Sample
Seattle-Denver Income Maintenance
Experiment
Table 6.
Labor Supply Response of Female Heads:
A. Overall NIT response
(Percentage difference in annual hours worked)
Year
Sample Group
1
2
3
4
5
-5.5 -14.1 -21.6 -8.9
-7.7
3-year sample
-7.9 -15
-21.2 -28.3 -31.8
5-year sample
-6.3 -14.3 -21.4 —
—
Total Sample
Social Auditing
 One of the limitations of social systems
accounting and social experimentation is
that both approaches neglect or
oversimplify policy processes.
Social Auditing
 Social auditing explicitly monitors relations
among inputs, processes, outputs, and
impacts in an effort to trace policy inputs
from disbursement to final recipient.
 Social auditing helps to determine whether
ineffective outcomes are the result of
inadequate inputs or processes that divert
resources or services from intended target
groups or beneficiaries.
Social Auditing
 Processes monitored of two main types:
– Resource diversion (from target or beneficiary
groups).
– Resource transformation (changes in meaning
of policy actions from administrator to
recipient).
Social Audit Steps
 The three phases of a social audit
 Phase 1: Design and data collection
– Clarify the strategic focus
– Design instruments, pilot test
– Collect information from households and key
informants in a panel of representative
communities
Social Audit Steps
 Phase 2: Evidence-based dialogue and
analysis
– Link household data with information from
public services
– Analyze findings in a way that points to action
– Take findings back to the communities for their
views about how to improve the situation
– Bring community members into discussion of
evidence with service providers/planners.
Social Audit Steps
 Phase 3: Socialization of evidence for
public accountability
– Work-shopping
– Communication strategy
– Evidence-based training of planners and service
providers
– Media training
– Partnerships with civil society
Research and practice synthesis.
 An approach to monitoring that involves the
systematic compilation, comparison, and
assessment of the results of past efforts to
implement public policies.
 Two primary sources of available information.
– Case studies of policy formulation and implementation.
– Research reports that address relations among policy
actions and outcomes.
Research and practice synthesis.
 Two methods.
– Case survey method – a set of procedures used
to identify and analyze factors that account for
variations in the adoption and implementation
of policies.
• Requires case coding scheme, a set of categories
that capture key aspects of policy inputs, processes,
outputs, and impacts.
Research and practice synthesis
 Two methods.
– Research survey method – a set of procedures
used to compare and appraise results of past
research on policy actions and outcomes.
• Yields empirical generalizations about sources of
variation in policy outcomes, summary assessments
of the confidence researchers have in these
generalizations, and policy alternatives or action
guidelines that are implied in these generalizations.
• Requires the construction of a format for extracting
information about research reports.
Research and practice synthesis
 Advantages.
– Comparatively efficient way to compile and
appraise an increasingly large body of cases
and research reports on policy implementation.
– The case survey method is one among several
ways to uncover different dimensions of policy
processes that affect policy outcomes.
– The case survey method is a good way to
examine both objective and subjective
conditions.
Research and practice synthesis
 Limitations.
– All related to reliability and validity of
information.
Research and Practice Synthesis:
Example
 David H. Greenberg; Charles
Michalopoulos; Philip K. Robins.
 Do Experimental and Nonexperimental
Evaluations Give Different Answers
about the Effectiveness of GovernmentFunded Training Programs?
Research and Practice Synthesis:
Example
 Abstract
 This paper uses meta-analysis to investigate
whether random assignment (or experimental)
evaluations of voluntary government-funded
training programs for the disadvantaged have
produced different conclusions than
nonexperimental evaluations.
 Information includes several hundred estimates
from 31 evaluations of 15 programs that operated
between 1964 and 1998.
Research and Practice Synthesis:
Example
 The results suggest that experimental and
nonexperimental evaluations yield similar
conclusions about the effectiveness of
training programs, but that estimates of
average effects for youth and possibly men
might have been larger in experimental
studies.
POSC 431 – Program
Evaluation
Source
 Basic Guide to Program Evaluation
– Written by Carter McNamara, MBA, PhD,
Authenticity Consulting, LLC. Copyright 19972006.
– Adapted from the Field Guide to Nonprofit
Program Design, Marketing and Evaluation.
A Brief Introduction ...
 Note that the concept of program evaluation can
include a wide variety of methods to evaluate
many aspects of programs in nonprofit or forprofit organizations.
 However, personnel do not have to be experts in
these topics to carry out a useful program
evaluation. The "20-80" rule applies here, that
20% of effort generates 80% of the needed results.
It's better to do what might turn out to be an
average effort at evaluation than to do no
evaluation at all.
Some Myths about Program
Evaluation
 Many people believe evaluation is a useless
activity that generates lots of boring data
with useless conclusions.
 More recently (especially as a result of
Michael Patton's development of utilizationfocused evaluation), evaluation has focused
on utility, relevance and practicality at least
as much as scientific validity.
Some Myths about Program
Evaluation
 Many people believe that evaluation is about
proving the success or failure of a program.
 This myth assumes that success is implementing
the perfect program and never having to hear from
employees, customers or clients again -- the
program will now run itself perfectly.
 This doesn't happen in real life.
 Success is remaining open to continuing feedback
and adjusting the program accordingly.
 Evaluation gives you this continuing feedback.
Some Myths about Program
Evaluation
 Many believe that evaluation is a highly unique and




complex process that occurs at a certain time in a certain
way, and almost always includes the use of outside
experts.
Many people believe they must completely understand
terms such as validity and reliability.
They don't have to.
They do have to consider what information they need to
make current decisions about program issues or needs.
And they have to be willing to commit to understanding
what is really going on.
So What is Program Evaluation?
 First, we'll consider "what is a program?"
 Typically, organizations work from their mission to
identify several overall goals which must be reached to
accomplish their mission.
 In public agencies and nonprofits, each of these goals often
becomes a program.
 Public and nonprofit programs are organized methods to
provide certain related services to constituents, e.g.,
clients, customers, patients, etc.
 Programs must be evaluated to decide if the programs are
indeed useful to constituents.
So What is Program Evaluation?
 Program evaluation is carefully collecting
information about a program or some aspect of a
program in order to make necessary decisions
about the program.
 Don't worry about what type of evaluation you
need or are doing -- worry about what you need to
know to make the program decisions you need to
make, and worry about how you can accurately
collect and understand that information.
Where Program Evaluation is
Helpful
 Understand, verify or increase the impact of
products or services on customers or clients.
 Improve delivery mechanisms to be more
efficient and less costly.
 Verify that you're doing what you think
you're doing.
Other Reasons to Conduct
Evaluations
 Facilitate management's really thinking
about what their program is all about,
including its goals, how it meets it goals and
how it will know if it has met its goals or
not.
 Produce data or verify results that can be
used for public relations and promoting
services in the community.
Other Reasons to Conduct
Evaluations
 Produce valid comparisons between
programs to decide which should be
retained, e.g., in the face of pending budget
cuts.
 Fully examine and describe effective
programs for duplication elsewhere.
Basic Ingredients: Organization
and Program(s)
 You Need An Organization:
– This may seem too obvious to discuss, but before an
organization embarks on evaluating a program, it
should have well established means to conduct itself as
an organization.
 You Need Program(s):
– To effectively conduct program evaluation, you should
first have programs. That is, you need a strong
impression of what your customers or clients actually
need.
– Next, you need some effective methods to meet each of
those goals. These methods are usually in the form of
programs.
Programs
 Inputs are the various resources needed to
run the program, e.g., money, facilities,
customers, clients, program staff, etc.
 The process is how the program is carried
out.
Programs
 The outputs are the units of service, e.g., number
of customers serviced, number of clients
counseled, children cared for, artistic pieces
produced, or members in the association.
 Outcomes are the impacts on the customers or on
clients receiving services, e.g., increased mental
health, safe and secure development, richer artistic
appreciation and perspectives in life, increased
effectiveness among members, etc.
Planning Your Program
Evaluation
 Depends on what information you need to
make your decisions and on your resources.
– Your program evaluation plans depend on what
information you need to collect in order to make
major decisions.
– But the more focused you are about what you want to
examine by the evaluation, the more efficient you can
be in your evaluation, the shorter the time it will take
you and ultimately the less it will cost you.
Key Considerations
 For what purposes is the evaluation being
done, i.e., what do you want to be able to
decide as a result of the evaluation?
 Who are the audiences for the information
from the evaluation, e.g., customers,
bankers, funders, board, management, staff,
customers, clients, citizens, etc.
Key Considerations
 What kinds of information are needed to make the
decision you need to make and/or enlighten your
intended audiences, e.g., information to really
understand the process of the product or program
(its inputs, activities and outputs), the customers
or clients who experience the product or program,
strengths and weaknesses of the product or
program, benefits to customers or clients
(outcomes), how the product or program failed
and why, etc.
Key Considerations
 From what sources should the information be
collected, e.g., employees, customers, clients,
groups of customers or clients and employees
together, program documentation, etc.
 How can that information be collected in a
reasonable fashion, e.g., questionnaires,
interviews, examining documentation, observing
customers or employees, conducting focus groups
among customers or employees, etc.
Key Considerations
 When is the information needed (so, by
when must it be collected)?
 What resources are available to collect the
information?
Goals-Based Evaluation
1. How were the program goals (and objectives, is
applicable) established? Was the process
effective?
2. What is the status of the program's progress
toward achieving the goals?
3. Will the goals be achieved according to the
timelines specified in the program
implementation or operations plan? If not, then
why?
4. Do personnel have adequate resources (money,
equipment, facilities, training, etc.) to achieve the
goals?
Goals-Based Evaluation
5. How should priorities be changed to put more focus on
achieving the goals? (Depending on the context, this
question might be viewed as a program management
decision, more than an evaluation question.)
6. How should timelines be changed (be careful about
making these changes - know why efforts are behind
schedule before timelines are changed)?
7. How should goals be changed (be careful about making
these changes - know why efforts are not achieving the
goals before changing the goals)? Should any goals be
added or removed? Why?
8. How should goals be established in the future?
Process-Based Evaluations
 1. On what basis do employees and/or the
customers decide that products or services
are needed?
 2. What is required of employees in order to
deliver the product or services?
Process-Based Evaluations
 3. How are employees trained about how to
deliver the product or services?
 4. How do customers or clients come into
the program?
 5. What is required of customers or client?
 6. How do employees select which products
or services will be provided to the customer
or client?
Process-Based Evaluations
 7. What is the general process that
customers or clients go through with the
product or program?
 8. What do customers or clients consider to
be strengths of the program?
 9. What do staff consider to be strengths of
the product or program?
 10. What typical complaints are heard from
employees and/or customers?
Process-Based Evaluations
 11. What do employees and/or customers
recommend to improve the product or
program?
 12. On what basis do employees and/or the
customer decide that the product or services
are no longer needed?
Outcome-Based Evaluations
 Identify the major outcomes that you want to
examine or verify for the program under
evaluation.
 Choose the outcomes that you want to examine,
prioritize the outcomes and, if your time and
resources are limited, pick the top two to four
most important outcomes to examine for now.
 For each outcome, specify what observable
measures, or indicators, will suggest that you're
achieving that key outcome with your clients.
Outcome-Based Evaluations
 Specify a "target" goal of clients, i.e., with
what number or percent of clients you
commit to achieving specific outcomes?
 Identify what information is needed to show
these indicators.
 Decide how can that information be
efficiently and realistically gathered.
 Analyze and report the findings.
Methods to Collection Information
 Questionnaires, surveys, checklists.
 Interviews.
 Documentation review.
 Observation.
 Focus groups.
 Case studies.
Methods to Collection Information
Methods to Collection Information
Methods to Collection Information
Ethics: Informed Consent from
Program Participants
 Note that if you plan to include in your evaluation,
the focus and reporting on personal information
about customers or clients participating in the
evaluation, then you should first gain their consent
to do so.
 They should understand what you're doing with
them in the evaluation and how any information
associated with them will be reported.
Ethics: Informed Consent from
Program Participants
 You should clearly convey terms of
confidentiality regarding access to
evaluation results.
 They should have the right to participate or
not.
 Have participants review and sign an
informed consent form.
Selecting Which Methods to Use
 The overall goal in selecting evaluation method(s) is to get
the most useful information to key decision makers in the
most cost-effective and realistic fashion.
 Note that, ideally, the evaluator uses a combination of
methods, for example, a questionnaire to quickly collect a
great deal of information from a lot of people, and then
interviews to get more in-depth information from certain
respondents to the questionnaires.
 Perhaps case studies could then be used for more in-depth
analysis of unique and notable cases, e.g., those who
benefited or not from the program, those who quit the
program, etc.
Selecting Which Methods to Use
1. What information is needed to make current
2.
3.
4.
5.
decisions about a product or program?
Of this information, how much can be collected
and analyzed in a low-cost and practical manner,
e.g., using questionnaires, surveys and checklists?
How accurate will the information be (reference
the above table for disadvantages of methods)?
Will the methods get all of the needed
information?
What additional methods should and could be used
if additional information is needed?
Selecting Which Methods to Use
6. Will the information appear as credible to
decision makers, e.g., to funders or top
management?
7. Will the nature of the audience conform to the
methods, e.g., will they fill out questionnaires
carefully, engage in interviews or focus groups,
let you examine their documentations, etc.?
8. Who can administer the methods now or is
training required?
9. How can the information be analyzed?
Four Levels of Evaluation:
 There are four levels of evaluation information
that can be gathered from clients, including
getting their:
1. reactions and feelings (feelings are often poor
indicators that your service made lasting impact)
2. learning (enhanced attitudes, perceptions or
knowledge)
3. changes in skills (applied the learning to enhance
behaviors)
4. effectiveness (improved performance because of
enhanced behaviors)
Four Levels of Evaluation:
 Usually, the farther your evaluation
information gets down the list, the more
useful is your evaluation. Unfortunately, it
is quite difficult to reliably get information
about effectiveness. Still, information
about learning and skills is quite useful.
Analyzing and Interpreting
Information
 Always start with your evaluation goals:
– When analyzing data (whether from
questionnaires, interviews, focus groups, or
whatever), always start from review of your
evaluation goals, i.e., the reason you undertook
the evaluation in the first place. This will help
you organize your data and focus your analysis.
Analyzing and Interpreting
Information
 Basic analysis of "quantitative" information (for
information other than commentary, e.g., ratings,
rankings, yes's, no's, etc.):
1. Make copies of your data and store the master copy
away. Use the copy for making edits, cutting and
pasting, etc.
2. Tabulate the information, i.e., add up the number of
ratings, rankings, yes's, no's for each question.
3. For ratings and rankings, consider computing a mean,
or average, for each question.
4. Consider conveying the range of answers, e.g., 20
people ranked "1", 30 ranked "2", and 20 people
ranked "3".
Analyzing and Interpreting
Information
 Basic analysis of "qualitative" information
(respondents' verbal answers in interviews, focus
groups, or written commentary on questionnaires):
1. Read through all the data.
2. Organize comments into similar categories.
3. Label the categories or themes, e.g., concerns,
suggestions, etc.
4. Attempt to identify patterns, or associations and causal
relationships in the themes.
5. Keep all commentary for several years after completion
in case needed for future reference.
Interpreting Information:
1. Attempt to put the information in perspective.
a)
b)
c)
d)
e)
f)
Compare results to what you expected;
Management or program staff
Common standards for your services
Original program goals (program evaluation)
Indications of accomplishing outcomes (outcomes evaluation)
Description of the program's experiences, strengths, weaknesses,
etc. (process evaluation).
2. Consider recommendations to help program staff improve
the program, conclusions about program operations or
meeting goals, etc.
3. Record conclusions and recommendations in a report
document, and associate interpretations to justify your
conclusions or recommendations.
Reporting Evaluation Results
1. The level and scope of content depends on
to whom the report is intended,
2. Be sure employees have a chance to
carefully review and discuss the report.
Translate recommendations to action plans,
including who is going to do what about the
program and by when.
Reporting Evaluation Results
3. Bankers or funders will likely require a report that
includes an executive summary (this is a summary of
conclusions and recommendations, not a listing of what
sections of information are in the report -- that's a table of
contents); description of the organization and the program
under evaluation; explanation of the evaluation goals,
methods, and analysis procedures; listing of conclusions
and recommendations; and any relevant attachments,
4. Be sure to record the evaluation plans and activities in an
evaluation plan which can be referenced when a similar
program evaluation is needed in the future.
Pitfalls to Avoid
1. Don't balk at evaluation because it seems far too
"scientific." It's not. Usually the first 20% of effort
will generate the first 80% of the plan, and this is
far better than nothing.
2. There is no "perfect" evaluation design. Don't
worry about the plan being perfect. It's far more
important to do something, than to wait until
every last detail has been tested.
Pitfalls to Avoid
3. Work hard to include some interviews in your evaluation
methods. Questionnaires don't capture "the story," and the
story is usually the most powerful depiction of the
benefits of your services.
4. Don't interview just the successes. You'll learn a great
deal about the program by understanding its failures,
dropouts, etc.
5. Don't throw away evaluation results once a report has
been generated. Results don't take up much room, and
they can provide precious information later when trying to
understand changes in the program.
Research and Practice Synthesis:
Example
 The results also suggest that variation
among nonexprimental estimates of
program effects is similar to variation
among experimental estimates for men and
youth, but not for women (for whom it
seems to be larger), although small sample
sizes make the estimated differences
somewhat imprecise for all three groups.
Research and Practice Synthesis:
Example
 The policy implications of the findings are
discussed.
 © 2006 by the Association for Public Policy
Analysis and Management
Techniques for Monitoring
TABULAR
DISPLAYS
INDEX
NUMBERS
INTERRUPTED
TIMESERIES
ANALYSIS
CONTROLSERIES
ANALYSIS
REGRESSION DISCONTINUITY
ANALYSIS
APPROACH
GRAPHIC
DISPLAYS
Social systems
accounting
X
X
X
X
X
O
Social auditing
x
x
x
x
x
o
Social
experimentation
x
x
x
x
x
x
Research and
practice
synthesis
x
x
o
o
o
o
Recommendation in Policy
Analysis
 Disjointed incremental theory.
– Analyze and evaluate alternatives in a sequence of
steps, such that choices are continuously amended over
time, rather than made at a single point in time.
– Continuously remedy existing social problems, rather
than solve problems completely at one point in time.
– Share responsibilities for analysis and evaluation with
many groups in society, so that the process of making
choices is fragmented or disjointed.
Recommendation in Policy
Analysis
 Arrow’s impossibility theorem.
– It is impossible for democratic decision makers
in a democratic society to meet conditions of
the rational comprehensive model.
• Individual choices cannot be aggregated through
majority voting procedures to create a collective
decision that will produce a single best solution for
all parties.
The Voter’s Paradox
COMMITTEE MEMBER
Brown (criterion: risk)
PREFERENCE
A (solar) preferred to B (coal)
B (coal) preferred to C (nuclear)
A (solar preferred to C (nuclear)
Jones (criterion: feasibility)
B preferred to C
C preferred to A
B preferred to A
Smith (criterion: efficiency)
C preferred to A
A preferred to B
C preferred to B
Majority (intransitive and cyclic)
A preferred to B
B preferred to C
C preferred to A
Recommendation in Policy
Analysis
 Arrow’s impossibility theorem.
– Reasonable conditions for democratic decision
procedures.
•
•
•
•
•
Nonrestriction of choices.
Nonperversity of collective choice.
Independence of irrelevant alternatives.
Citizen’s sovereignty.
Nondictatorship.
Recommendation in Policy
Analysis
 Bounded rationality.
– Decision makers engage in satisficing behavior
(identify courses of action that are “good
enough.”).
• Consider the most evident alternatives that produce
a reasonable increase in benefits.
 Rationality as constrained maximization.
– Rational choice within the boundaries of
constraint.
Recommendation in Policy
Analysis
 Criteria for policy recommendation.
– Effectiveness – does a given alternative result in the
achievement of a valued outcome (technical
rationality).
– Efficiency – the amount of effort needed to produce a
given level of effectiveness (economic rationality).
– Adequacy – the extent to which any given level of
effectiveness satisfies the needs, values, or
opportunities that gave rise to the problem.
• Fixed costs and variable effectiveness (type I, maximize
effectiveness).
• Fixed effectiveness and variable costs (type II, minimize
costs).
• Variable costs and variable effectiveness (type III, efficiency).
• Fixed costs and fixed effectiveness (type IV, do nothing).
Recommendation in Policy
Analysis
 Criteria for policy recommendation.
– Equity – the distribution of effects and effort among
different groups in society (legal and social rationality).
•
•
•
•
Maximize individual welfare.
Protect minimum welfare.
Maximize net welfare.
Maximize redistributive welfare.
– Responsiveness – satisfies the needs, preferences, or
values of particular groups.
– Appropriateness – the value or worth of a program’s
objectives and the tenability of assumptions underlying
these objectives.
Approaches to Recommendation
 Public versus private choice.
– Nature of public policy processes.
• Numerous stakeholders with conflicting values.
– Collective nature of public policy goals.
• Multiple conflicting criteria for choice.
– Nature of public goods.
• Specific, collective, and quasi-collective goods.
 Supply and demand (market mechanisms).
– Discuss.
Approaches to Recommendation
 Public choice.
– Problems with supply – demand models of
public policy.
•
•
•
•
Multiple legitimate stakeholders.
Collective and quasi-collective goods.
Limited comparability of income measures.
Public responsibility for social costs and benefits.
Approaches to Recommendation
 Benefit-cost analysis.
– Characteristics.
• Measure all costs and benefits to society of a program
including intangibles.
• Traditional benefit-cost analysis emphasizes economic
rationality: net benefits are greater than zero and higher than
alternative uses.
• Traditional benefit-cost analysis uses the private marketplace
as the point of departure in recommending programs.
• Social benefit-cost analysis also measures redistributional
benefits.
Approaches to Recommendation
 Types of costs and benefits.
– Inside versus outside costs and benefits.
– Tangible versus intangible costs and benefits.
– Direct versus indirect costs and benefits.
– Net efficiency versus redistributional benefits.
Approaches to Recommendation
 Tasks in benefit-cost analysis.
– Problem structuring.
– Specification of objectives.
– Identification of alternative solutions.
– Information search, analysis, and interpretation.
– Identification of target groups and beneficiaries.
– Estimation of costs and benefits.
– Discounting of costs and benefits.
– Estimation of risk and uncertainty.
– Choice of decision criterion.
– Recommendation.
Benefit-cost and Costeffectiveness Analysis
 Benefit-cost analysis is an applied branch of
economics that attempts to assess service
programs by determining whether the total societal
welfare has increased (in aggregate more people
have been made better off) because of the project
or program.
 Steps.
– Determine the benefits of a proposed or existing
program and place a dollar value on those benefits.
– Calculate the total costs of the program.
– Compare the benefits and costs.
Benefit-cost and Costeffectiveness Analysis
 Simple steps pose real challenge, especially
estimating intangible benefits.
 Procedure still useful in uncovering
assumptions and estimating value of
intangibles.
Benefit-cost and Costeffectiveness Analysis
 Cost-effectiveness analysis.
– The major costing alternative to benefit-cost analysis.
• Relates the cost of a given alternative to specific measures of
program objectives.
– For example, dollar per life saved on various highway
safety programs.
– Often the first step in a benefit-cost analysis.
– Especially useful if analyst cannot quantify benefits,
but has fairly specific program objectives.
– Key problem: situation where there are multiple
benefits. Results often very subjective.
– 2nd key problem: does not produce a bottom line
number.
Benefit-cost and Costeffectiveness Analysis
 A private sector analogy.
– Benefit-cost analysis similar to financial analysis in
private sector.
• Should the firm have done the project at all, i.E., Is the project
producing a satisfactory rate of return?
• Public version: is the program a success, i.E., Has it improved
social welfare?
• What other options are there for the use of the firm’s
resources?
• Public version: should the program be continued when
weighed against alternative uses for the government’s funds.
Benefit-cost and Costeffectiveness Analysis
 A private sector analogy.
– Benefits and costs do not occur simultaneously
in either private or public sector.
• R&D costs, marketing, capital investment, training.
• Government not priced, thus benefits are more
broadly defined.
– Alternative uses of resources (opportunity
costs).
Benefit-cost and Costeffectiveness Analysis
 Benefit-cost illustration.
Costs
Year
R&D
Capital
O&M
Total
Benefits
1
$1500
$2000
$0
$3500
$0
2
500
2000
2000
4500
3500
3
2000
2500
4500
5500
4
2000
3000
5000
6500
5
2000
3500
5500
8500
$10000
$11000
$23000
$24000
Totals
Year
$2000
Benefit-Costs
Present Value, Benefits-Costs (10%)
1
($3500)
($3500)
2
(1000)
(909)
3
1000
826
4
1500
1127
5
3000
2049
$1000
($407)
Totals
Net Present Value @ 10% ($407)
Net Present Value @ 5%
219
Benefit-cost and Costeffectiveness Analysis
 Formula for net present value.
y2
y2
y3
y3
yx
yx
B

C
B

C
B

C
NPV  B y1  C y1 

 ... 
2
1 r
(1  r )
(1  r ) x1
 Formula sensitive to choice of rate of return.
 Could also use return on investment.
– Discount rate that would make the present value zero.
Benefit-cost and Costeffectiveness Analysis
 Continuing or not continuing the project.
Costs
Year
R&D
6
Capital
$500
O&M
Total
Benefits
$400
$4500
$5400
$9000
7
400
5000
5400
8500
8
400
6000
6400
8500
9
400
7500
7900
8000
10
400
8000
8400
7500
$2000
$31000
$33500
$41500
Totals
Year
$500
Benefit-Costs
Present Value, Benefits-Costs (10%)
6
$3600
3600
7
3100
2818
8
2100
1736
9
100
75
(900)
(615)
$8000
$7614
10
Totals
Net Present Value @ 10% $7614
Net Present Value @ 5%
$7803
Benefit-cost and Costeffectiveness Analysis
 Total versus marginal benefits and costs.
– When considering the overall profitability of a project,
an agency will consider the total costs involved in
getting the project started through its operation’s cycle.
– But at any point in time, when an agency is continuing
or discontinuing a project or program, it only considers
the marginal costs or benefits.
– Definitions.
• Marginal cost: incremental cost of producing one more unit of
output.
• Marginal benefit: incremental benefit of that one unit of output.
– Public sector usually does not do it on the basis of a
single unit, but what are the benefits being generated
now versus the costs now?
Benefit-cost and Costeffectiveness Analysis
 Public-private sector differences and
similarities.
– Similarities – alternative uses for funds, onetime costs, recurring costs, land, labor, capital.
– Differences.
• Distributional considerations.
• Spillovers.
Framework for Analysis
Benefit
Indicator
Measure
Dollar value
Assumptions
Indicator
Measure
Dollar value
Assumptions
Real
Direct
Tangible
Intangible
Indirect
Tangible
Intangible
Cost
Real
Direct
Tangible
Intangible
Indirect
Tangible
Intangible
Transfers
Framework for Analysis
 Real benefits and costs versus transfers.
– Real: net gains and losses to society.
– Transfers: merely alter the distribution of
resources in society.
Framework for Analysis
 Direct and indirect benefits and costs.
– Direct costs and benefits are closely related to
the primary objectives of the project.
• Direct costs – personnel, facilities, equipment,
material, project administration.
– Indirect costs and benefits are byproducts,
multipliers, spillovers, or investment effects.
• Indirect costs are unintended costs that result from
government action.
• Indirect benefits might include benefits of space
exploration.
Framework for Analysis
 Tangible and intangible benefits and costs.
– Tangible benefits and costs can be converted
readily into dollar figures.
– Intangible benefits and costs are those things
that cannot be directly assigned an explicit
price.
 Determining the geographic scope of
analysis.
– Spillover effects may determine true
geographic jurisdiction.
Framework for Analysis
(Agricultural Dam Example)
Real Benefits
Nature of Benefit/Cost
Direct
Tangible
Increased farm output
New supply of water
Intangible
Maintaining family farms
Indirect
Tangible
Reduced soil erosion
Intangible
Preservation of rural society
Real Costs
Direct
Tangible
Construction material, labor, operations and maintenance, direct program supervision by agency
Intangible
Loss of recreational value of land or river
Indirect
Tangible
Administrative overhead of government
Diversion of water and its effects
Increased salinity
Intangible
Loss of wilderness area
Transfers
Relative improvement of profit for farm implement industry
General taxpayers may be subsidizing farmers.
Measuring Benefits
 Evaluation problem difficult for government
because of multiple benefits and
intangibles.
Measuring Benefits
 Sources of data.
– Existing records and statistics kept by agency.
– Feedback from clients.
– Ratings by trained observers.
– Experience of other governments or private or
nonprofit corporations.
– Special data gathering.
 Whenever possible analyst should use
market value or willingness to pay.
Measuring Benefits
 Valuing benefits.
– Cost savings.
– Time saved.
– Lives saved.
– Increased productivity or wages.
– Recreational benefits.
– Land values.
– Alternatives to market prices.
 See handout.
Measuring Costs
 Cost categories.
– One-time, fixed, or up-front costs.
– Ongoing investment costs.
– Recurring costs.
– Compliance costs.
– Mitigation measures.
Measuring Costs
 Valuing indirect costs.
– Flat overhead figure.
• Does the project actually costs increased administrative
burden?
–
–
–
–
Costs to the private sector.
Valuing the use of capital.
Valuing the damage effects of government programs.
Other cost issues.
• Sunk costs.
• Interest costs.
Analysis of Benefits and Costs
 Framework.
– Retrospective.
– Snapshot.
– Prospective.
Analysis of Benefits and Costs
 Importance of using present value.
– Choice of an appropriate discount rate.
• Private sector rate.
• Low social discount rate.
• Long-term treasury bill rates.
– Adjustment for inflation.
Analysis of Benefits and Costs
 Presenting the results.
– Net present value (B-C).
– Benefit cost ratio (B/C).
– Return on investment (discount rate to reduce
present value to zero).
 Appropriate perspective.
– Costs and benefits vary across stakeholders.
– May conduct analyses from several
perspectives.
Analysis of Benefits and Costs
 Sensitivity analysis.
– Use spreadsheet analysis to vary the assumptions.
 Intangibles.
– Relate intangibles to dollar results: if there are net
costs, do the intangible benefits overcome the deficit?
 Particular problems.
– Equity concerns (weighting of values).
– Multiple causation and co-production problems.
PPA 503 – The Public
Policy-Making Process
Lecture 8d – How to Deliver Oral
Testimony Based on a Written
Statement
Introduction
 Goal:
– To speak authoritatively and to answer
questions responsively in public deliberation.
 Objective:
– Skill of writing speakable text, skill of speaking
easily from written text, and readiness to
answer anticipated and unanticipated questions.
Introduction
 Scope:
– Pinpointed topic pertinent to hearing’s purpose and the
witness’s role.
 Product:
– Two expected communication products:
• Short oral summary.
• Full written statement.
 Strategy:
– Confident and useful public testimony resulting from
advance preparation.
PPA 503 – The Public
Policy-Making Process
Lecture 8c – How to Inform Policy
Makers in a Briefing Memo or
Opinion Statement
Introduction
 Goal:
– Recognition of meaningful information in a mass of details
and representation highlighting the significance of
information for a user.
 Objective:
– Skills of distilling, listening, recording, observing,
evaluating sources, relating details to context, interpreting
details accurately in context, and selecting details according
to relevance; capability of stating informed opinion that is
aware of and responsive to other opinions.
Introduction
 Product:
– One- to two-page written memo, or one- to-two
paragraph written statement, possibly with
attachments.
 Scope:
– Only essential topics in an identified context to
target a specific information need.
Introduction
 Strategy:
– Why is this communication necessary?
– What is the subject?
– What is the purpose?
– What will this communication do? What can
happen as a result?
– What is the context?
– What is the situation of reception?
– Why are you providing the information?
Task #1. Develop the Information
 From meetings.
– Attendance, notes, handouts, statements.
 From varied sources.
– Publications and materials
– Consultation.
 From informed reflection and analysis.
– Update, summarize, consult.
Task #2, Write the Memo or Statement.
 Chapter 2 method.
 Quick comprehension and ready use.
 Choose the right presentation.
 Stationery, header, overview, etc.
Developing a Testimony
 Present concisely and responding credibly
to questions.
 Know the context.
 Know your message.
 Know your role.
 Know the communication situation.
 Rehearse your delivery.
Task #1. Write the Testimony
 Use the method in Chapter 2 for both the
oral and written forms.
 Write out your oral summary, even if only
talking points.
 Everything in an oral testimony is public
record.
Task #1. Write the Testimony
 Template.
– Title page or header to identify organization and witness, the
agency holding the hearing, the topic, the date, and the location of
the hearing.
– Greeting to thank the organizers for the opportunity to testify and
to state why the topic is important to the witness.
– Message to state the main information the testimony provides.
– Support (evidence, grounds) for the message.
– Relevance of the message to the hearing’s purpose.
– Optional: discussion or background to add perspective on the
message (only if relevant).
– Closing to conclude the testimony and invite questions.
Task #2. Write the Full Statement
 Might use the same organization as the oral
summary.
 May be longer, more details, and include
appendices.
 Must be well organized to ensure use.
Task #3. Present the testimony.
 Summarize.
 State the message clearly and emphatically.
 Stay within time limits.
 Listen.
 Answer credibly.
Task #3. Present the testimony.
 Questions-and-answer.
–
–
–
–
–
–
Listen to the questions of other witnesses.
Make sure you hear each question correctly.
Answer the question that is asked.
Stop when you have answered the question.
Do not lie or invent information.
Handle the following situations carefully:
• You are asked your personal opinion.
• You don’t know the answer.
• Your credentials are challenged or your credibility is
attacked.
PPA 503 – The Public
Policy-Making Process
Lecture 2c – APA Editorial Style
Punctuation
 Period.
– Use a period to end a complete sentence (also abbreviations,
quotations, numbers, and references).
 Comma.
Use a comma
– Between elements (including before and and or) in a series of three
or more items.
• The height, width, or depth.
– To set off a nonessential or nonrestrictive clause, that is, a clause
that embellishes a sentence but if removed would leave the
grammatical structure and meaning of the sentence intact.
• Switch A, which was on a panel, controlled the recording device.
Punctuation
 Comma (contd.)
– To separate two independent clauses joined by
a conjunction.
• Cedar shavings covered the floor, and paper was
available for shredding and nest building.
– To set of the year in exact dates.
• April 18, 1992, was the correct date.
• But, April 1992 was the correct date.
– To separate groups of three digits in most
numbers of 1,000 or more.
Punctuation
 Comma (contd.).
Do not use a comma
– Before an essential or restrictive clause, that is, a clause
that limits or defines the material it modifies. Removal of
the clause would alter the meaning.
• The switch that stops the recording device also controls the
light.
– Between the two parts of a compound predicate.
• The results contradicted Smith’s hypothesis and indicated
that the effect was nonsignificant.
– To separate parts of measurement.
• 8 years 2 months.
Punctuation
 Semicolon.
Use a semicolon
– To separate two independent clauses that are
not joined by a conjunction.
• The participants in the first study were paid; those in
the second were unpaid.
– To separate elements in a series that already
contains commas.
• The color order was red, yellow, blue; blue, yellow,
red; or yellow, red, blue.
Punctuation
 Colon.
Use a colon
 Between a grammatically complete introductory clause
(one that could stand as a sentence) and a final phrase or
clause that illustrates, extends, or amplifies the preceding
thought. If the clause following the colon is a complete
sentence, it begins with a capital letter.
– For example, Freud (1930/1961) wrote of two urges: an urge
toward union with others and an egoistic urge toward happiness.
– They have agreed on the outcome: Informed participants perform
better than do uninformed participants.
Punctuation
 Colon (contd.).
Do not use a colon
– After an introduction that is not a complete
sentence.
• The policy alternatives included
The status quo, which reflected the current policy
choices,
Alternative A, which required direct intervention,
and
Alternative B, which required indirect intervention.
Punctuation
 Dash
– Use a dash to indicate only a sudden interruption in the
continuity of a sentence. Do not overuse.
• These two alternatives—reducing benefits and
disqualifying recipients—significantly reduced the size of
the program.
 Quotation marks
– Use double quotation marks
– To introduce a word or phrase used as an ironic comment,
as slang, or as an invented or coined expression. Use only
the first time cited.
• Considered “normal” behavior.
• The “good-outcome” variable . . . The good-outcome
variable.
Punctuation
 Quotation marks (contd.)
– To reproduce material from a test item or
verbatim instructions to participants.
• The first question was “what is your gender?”
Use italics and not double quotation marks
– Identify the anchors of a scale.
– To cite a letter, word, phrase, or sentence as a
linguistic example.
– To introduce a technical or key term.
Punctuation
 Parentheses
Use parentheses
– To set off structurally independent elements
• The patterns were significant (see Figure 5).
– To set off reference citations in text.
• Kingdon (2003) suggests
– To introduce an abbreviation
• The Federal Emergency Management Agency (FEMA)
– To set off letters that identify items in a series within a
sentence or paragraph.
• The policies include (a) welfare policy, (b) energy policy,
and (c) defense policy.
Punctuation
 Parentheses (contd.)
Do not use parentheses
– To enclose material within other parentheses
(use brackets to enclose material within
parentheses).
• (the Department of Housing and Urban
Development [DHUD]).
– Back to back.
• (e.g., policy learning; May 1990).
Punctuation
 Brackets
Use brackets
– to enclose parenthetical material that is already within
parentheses.
• (The results for the control group [n=8] appear in Figure 2.)
• Exception: do not use brackets if the meaning is clear using
commas.
– Not (as Imai [1990] later concluded)
– But (as Imai, 1990, later concluded)
– to enclose material inserted in a quotation by someone
other than the author.
• “when [the author’s] words are quoted” (Dummy, 1995, p.
151).
Punctuation
 Slash
Do not use a slash
– When a phrase would be clearer.
• Not: Smith acted as a supervisor/mentor.
• But: Smith acted as a supervisor or mentor.
– For simple comparisons. Use a hyphen or short
das (en dash) instead.
• Test-retest reliability
• Not: test/retest reliability.
Spelling
 Merriam-Webster’s Collegiate Dictionary is the standard
spelling reference for APA journals and books.
 The more comprehensive version is the Webster’s Third
New International Dictionary.
 Plural forms of Latin or Greek origin
Singular
Plural
Appendix
appendices
Cannula
cannulas
Datum
data
Matrix
matrices
Phenomenon
phenomena
Schema
schemas
Spelling
 Hyphenation
– Use the dictionary to determine the use of
hyphens in compound words.
• Follow-up is a noun or adjective, but follow up is a
verb.
• If a compound is in a dictionary, it is considered a
permanent compound (e.g., high school, caregiver,
and self-esteem).
• Spelling can also change (life-style became lifestyle;
data base became database).
Spelling
 General principles of hyphenation
– Do not use a hyphen unless it serves a purpose. If a compound
adjective cannot be misread, do not use a hyphen.
• Grade point average.
• Health care reform.
– In a temporary compound that is used as an adjective before a
noun, use a hyphen if the term can be misread or if the term
expresses a single thought (all words modify the noun).
• Different-word lists (lists of different words).
• Different word lists (different lists of words).
– Most compound adjective rules are applicable only when the
compound adjective precedes the term it modifies. If it follows the
term, do not use a hyphen.
• Client-centered advice.
• But: the advice was client centered.
Spelling
 General principles of hyphenation.
– Write most words with prefixes as one word;
however, there are exceptions.
– When two or more compound modifiers have a
common base, this base is sometimes omitted in
all except the last modifier, but the hyphen is
retained.
Capitalization
 Words beginning a sentence
– The first word of a complete sentence.
– The first word after a colon that begins a complete
sentence.
 Major words in titles and headings
– Not conjunctions, articles, or short prepositions, but all
words four letters or longer. Capitalize all verbs, nouns,
adjectives, adverbs, and pronouns. When a capitalized
word is hyphenated, capitalize both words. Capitalize the
first word after a colon or dash in the title.
– Major words in article headings and subheadings.
– Major words in table titles and figure legends.
– References to titles of sections within the same article.
Capitalization
 Proper nouns and trade names.
– Proper nouns and adjectives and words used as
proper nouns.
– Names of departments if they refer to a specific
department.
– Trade and brand names of drugs, equipment,
food, programs, etc.
– Do not capitalize names of laws, theories,
models, or hypotheses (except retain uppercase
in proper names).
Capitalization
 Nouns followed by numerals or letters.
– On Day 2 of Experiment 4.
– Do not capitalize nouns that denote common
parts of books or tables followed by numerals
or letters.
 Titles of tests
– Capitalize complete, exact titles of published
and unpublished tests.
– Do not capitalize shortened, inexact, or general
titles of tests
Italics
 Use italics for
– Titles of books, periodicals, and microfilm
publications.
– Genera, species, and varieties.
– Introduction to a new, technical, or key term or
label (do not italicize after the first use).
– Letter, word, or phrase used as a linguistic
example.
– Words that could be misread.
– Periodical volume numbers in reference lists.
Italics
 Do not use italics
– Foreign phrases and abbreviations (ad lib, et al.,
per se, vis-à-vis.
– Greek letters.
– Mere emphasis.
Abbreviations
 Use abbreviations sparingly.
– Do not overuse because it creates confusion.
– Do not underuse. If you introduce an abbreviation, and only use it
two or three times subsequently, you are better spelling it out in all
cases.
 Explain the abbreviation the first time, and use the
abbreviation subsequently.
 Some abbreviations are in dictionaries. They can be used
without explanation.
– IQ, REM, ESP, AIDS, HIV, NADP, ACTH.
 Use the standard Latin abbreviations only inside
parentheses. Spell out the English equivalent in the main
text (e.g., use and so forth for etc.).