Documenting and Communicating
Progress on Policy Change
COMMUNITY HEALTH AND WELLNESS
CONFERENCE
APRIL, 2013
Agenda
 How does policy change progress, so I know what to
look for?
 What are some process and outcome measures that
are appropriate for policy change?
 What impact measures are appropriate for policy
change?
 What should I report and when?
Use data at
every stage of
policy work
Chuck: “Some date we
use to get started, some
data we use to know if
we are on the right
track, and some data
lets us know how we
did.”
Theory
of
change
Impact
Report
out
Progress:
outcome
Progress:
process

Global theory: Large Leaps

Policy change
theories
This document
identifies six theories of
policy change. Each is
summarized and a
theory of change is
provided for each
theory with strategies
and outcomes.
Stachowiak, S. (2009).
Pathways for change;
Six theories about hoe
policy change happens.
Seattle, WA:
Organizational
Research Services.
Funding from The
California Endowment.

Global theory: Coalition


Significant changes in policy and institutions can occur when the right
conditions are in place. Conditions are: issue is defined differently or
new dimensions of the issue get attention; new actors get involved; the
issue becomes more salient and receives heightened attention. Promising
strategies include issue framing, mobilizing supporters, and media
advocacy. Focus on changing policy, not administrative or rule changes.
Issue definition and agenda setting are key to mobilizing new people.
Policy changes happens through coordinated activity among a range of
individuals with the same core policy beliefs. Coalitions are held together
by agreement over core beliefs about policies; core beliefs are unlikely to
change. Strategies include influencing decision-makers, changing
incumbents, affecting public opinion, changing perceptions about
policies.
Global theory: Policy Windows

Policy can be changed during a window of opportunity when advocates
successfully connect two or more components of the policy process: the
way a problem is defined, the policy solution to the problem, or the
political climate surrounding the issue. Three streams in the policy
process are problems (including definition and salience), policies, and
politics. Two streams must converge to create a policy window.
Strategies include impacting problem definition, developing policy
options, and influencing the political climate (through coalition building,
demonstrations, or media).
 Strategy theory: Messaging and Frameworks
Based on work with
tobacco and underage
drinking prevention
coalitions
Consistent with
research (including that
done by Organizational
Research Services,
Michael Quinn Patton,
etc.).

Individuals’ policy preferences or willingness to accept
them will vary depending on how options are framed or
presented. Strategies include issue framing and message
development, typically embedded in a broader campaign.
 Strategy theory: Power Politics

Policy change is made by working directly with those with
power to make decisions or influence decision making;
power is concentrated in the hands of a few. Strategies
include relationship building, focusing on decisionmakers, and establishing credibility.
 Strategy theory: Grassroots

Policy change is made through collective action by
members of the community who work on changing
problems affecting their lives. Power is changeable and
dynamic and not held by a few elites. Strategies including
training, capacity building, community mobilizing, etc.
Coalition strategies were consistent with CDC
 Coalitions utilized strategies consistent with Centers
for Disease Control and Prevention guidance (2009).
The most common strategies were:



educating community and decision makers through
presentations and one-on-one meetings;
followed by monitoring, assessment, or local surveillance to
collect data about the issue;
and then advocating directly to decision-makers through
presentations or one-on-one meetings.
Comparing prevalence of strategies: successful
and unsuccessful efforts
<---less prevalent ----- more prevalent---->
SWAT involvement
Partner involvement
Education/presentations
Surveillance
Presented to decision-maker
Personal relationships
Successful policy efforts
Source: Coalition interviews
Not yet
Unsuccessful policy efforts
<---less prevalent -------------------more prevalent ---->
Share contacts
Distribute information
The prevalence of
actions taken by
Coalition members
to move a policy
forward.
Attend advocacy meetings
Participate in events
Make presentations
Complete tasks
Conduct local research
Promote the partnership
Source: Coalition interviews
Start with a theory of change
 What steps do you expect to see?
 What strategies are you using to move your policy
work forward?
Documenting progress
 There are many types of evaluation, but thinking about evaluation in
terms of process, outcome, and impact categories can be helpful.

Process evaluation is “a study of what goes on while a program (or policy) is in
progress. Process evaluation relates to the phase of the program studied – in this case,
program implementation “(Weiss, 1998, p 335).

Outcome evaluation is “a study of whether or not the program produced the intended
program effects. Outcome evaluation relates to the phase o f the program studies - in
this case, the end result of the program” (Weiss, 1998. p. 335).

Impact is “the net effects of a program (i.e., the gain in outcomes for program
participants minus the gain for an equivalent group of nonparticipants). Impact may
also refer to program effects for the larger community” (Weiss, 1998, p. 331).
Weiss, C. (1998) Evaluation: methods for studying programs and policies (2nd ed.). New Jersey: Prentice Hall.
Exercise: process measures for policy change
• Specific: the objective must communicate exactly what is
expected, who’s involved, and where is it going to happen.
• Measurable: the objective should include concrete criteria or
data for measuring progress toward the attainment of the goal.
• Attainable: the goals are neither out of reach nor below
standard performance as these may be considered meaningless.
• Relevant: the objective should be worthwhile, happening at the
right time, and match other efforts and needs.
• Timely: ground objectives within a time frame, giving them a
target date.
How did Coalitions move policy forward?
 Interim steps are an indicator of future success: 35
percent of the coalitions that achieved a policy
reported at least one interim step; in contrast, when
not successful, coalitions did not achieve an interim
step.
What types of interim steps occurred?
A champion stepped forward. Sometimes a school board member,
county commissioner, city council person, or administrator became
the champion for the policy and offered to guide it to adoption.
Policy was developed. Prior to adoption, representatives from the
coalition and the adopting entity often prepared drafts or worked on
the actual policy components.
Supports were developed. The coalition may have assisted with
implementation supports prior to the actual policy being adopted /
going into effect. This included helping to obtain signs, coordinating
meetings between cessation service providers and the entity,
providing other resources, or working on ancillary policies such as
citations.
Interim steps (continued)
Policy was placed on the policy agenda. This occurred when the issue or policy is
publicly placed on the policy agenda of your target. This could be when the school
board or county commission allows a presentation or when a landlord schedules
a presentation at a tenant meeting.
Verbal commitment from decision-maker was received. Prior to the actual
adoption of a policy, the coalition may receive a verbal commitment to adopt a
policy. For example, a landlord may state he/she wishes to move forward and
then schedules the next meeting to review draft policies with representatives
from the coalition.
A non-coalition task force or committee was formed to adopt/implement policy.
Often a task force or committee (particularly in schools) will be created to work
on the policy and guide its adoption and implementation. Other examples would
be a legislative committee working on a proclamation, resolution, or ordinance.
Other interim steps from the literature
 Policy development, policy proposal, demonstration
of support, adoption, funding, and implementation
 Shift in social norms: changes in awareness,
attitudes, values, or behaviors
 Increased alliances: increased number of partners,
increased strength of collaborations
 Strengthened base of support: increased public
involvement, increased advocacy activities, # of
partners, increased media coverage, increased
awareness among groups, increased visibility of issue
Exercise: Short and intermediate outcome
measures
• Direction of change: identify if the area of change is going to be
maintained, increase, decrease, improve, rise, or lower.
• Area of change: identify what will be measured.
• Target population: clarify the target population that will experience
the change.
• Degree of change: identify the percentage or actual number.
• Time frame: provide the deadline by which the change will occur.
• As measured by: include the tool or source of data.
• Baseline: document the baseline.
Strategy: social host ordinance

As noted, there may be additional interim and long-term outcomes associated with this strategy, such as ordinance
adoption, reduction in citations, reduction in social access, and reduction in consumption.
Outcome
Measure, instrument, or
question
Data collection methodology
Analysis
Process: Youth to present
information on the need for
policy change and its positive
impact to 5 municipalities
between 7/1/2012 and
6/30/2013.
Agendas
Source: Municipality
How: Collect or download
agenda
When: At time of presentation
Who: Coalition staff
Compare the number of
presentations planned to the
number of presentations
completed.
Outcome: By 6/30/2013, at least
two interim steps toward a social
host ordinance will have been
achieved.
Policy planner checklist (see
Resources section).
Source: Coalition
How: Document the date that
each interim step is achieved
When: Quarterly
Who: Coalition staff
Compare the number of steps
planned to the number of steps
completed.
Students report
social access to
alcohol
Long-term
impact
Number of policies
obtained
Number of residents
impacted
Students report
less social access
to alcohol
Policy progress:
progress
measures
Behaviors of interest
Others from prior
studies
Policy achieved:
XX,XXX
students
Policy progress:
interim steps
Reporting plan content
 Who is your target audience (e.g. funder, partner, policymaker)?
 When do they need information (e.g. when planning an evaluation,
when conducting an evaluation, or when you have results)?
 What message do you need to communicate (e.g. progress, results,
impact)?
 What format will you use to communicate your message (e.g. in
person, formal report, email)?
 How often will you communicate with this target audience (e.g.
monthly, quarterly, annually)?
Policy and advocacy resources
 California Endowment. (n.d). Advocacy progress planner [computer
software]. Available at www.planning.continuousprogress.org
 Sponsored by the California endowment, the component definitions
provided as part of the planner are based on two reports prepared by
Guthrie, K., Louie, J., David, T., and Foster, C. of Blueprint Research &
Design, Inc. in 2005 and 2006 although this document is not attributed to
any author.
 Interim steps towards policy include policy development, placement on the
policy agenda, policy adoption, policy implementation, policy monitoring
and evaluation, policy maintenance, and policy blocking. Document lists
audiences, capacity building and preparation/planning inputs, contextual
factors, policy and communication activities, and capacity and policy
interim outcomes and benchmarks. Users of the system create their own
models from these choices.
Policy and advocacy resources
 Reisman, J., Gienap, A., and Stachowiak, S. (2007).
A guide to measuring advocacy and policy. Seattle,
WA: Organizational Research Services. Funded by
the Annie E. Casey Foundation.
 Reisman, J., Gienap, A., and Stachowiak, S. (2004).
A practical guide to documenting influence and
leverage in Making Connections communities.
Seattle, WA: Organizational Research Services.
Funded by the Annie E. Casey Foundation.
Wrap up
 How does policy change progress, so I know what to look
for?

Theory of change
 What are some process and outcome measures that are
appropriate for policy change?


Process measures
Outcome measure
 What impact measures are appropriate for policy
change?

Impact measure
 What should I report and when?

Outline of a reporting plan