Iowa Strategic Prevention Framework
State Incentive Grant:
Evaluation Primer and Guide
Acknowledgements
This Guide is the result of a collaborative effort by the members of the Iowa Strategic Prevention
Framework State Incentive Grant (SPF SIG) Evaluation Workgroup. The workgroup is made up of the
following members and organizations:
Angie Asa-Lovstad, Kossuth Connections
Phyllis Blood, Consultant
Derek Clark, Clinton Substance Abuse Council
Dr. Ousmane Diallo, Iowa Department of Public Health
Julie Hibben, Iowa Department of Public Health
Jennifer Husmann, Area Substance Abuse Council
Martha McCormick, Next Step, Inc.
Linda McGinnis, Iowa Department of Public Health
Patrick McGovern, Iowa Consortium for Substance Abuse Research and Evaluation
Linda Phillips, Siouxland CARES
Deb Rohlfs, Community and Family Resources
Debbie Synhorst, Iowa Department of Public Health
Sections of this guide were adapted from material developed by the following organizations/sources:
Community Anti-Drug Coalitions of America (CADCA). Evaluation Primer: Setting the Context for a
Community Anti-Drug Coalition Evaluation (2010).
Maine SPF SIG Program, Maine’s Strategic Prevention Framework Guide to Evaluation and Planning
(2009).
South Carolina SPF SIG Program, Evaluation Planning for CAST Counties (2008).
1
Table of Contents
Acknowledgements............................................................................................................................................... 1
Table of Contents ................................................................................................................................................... 2
Introduction ............................................................................................................................................................ 3
Directions for Completing your SPF-SIG Evaluation Plan ....................................................................... 5
Appendix 1: ODSS Codes ...................................................................................................................................10
Appendix 2: CADCA Seven Approaches........................................................................................................11
2
Introduction
Over the last year or so, you and your coalition have been participating in the Strategic
Prevention Framework State Incentive Grant (SPF SIG) and working with the Iowa
Department of Public Health (IDPH) to implement the Strategic Prevention Framework
(SPF) process. The fifth step of this process is evaluation of your SPF-SIG implemented
strategies. This guidance document provides a brief overview of how evaluation fits within
the SPF process and how evaluation can be useful to your county. This guide will then
provide an overview of the required county-level evaluation. While this guide is organized
around the Strategic Prevention Framework principles, it can be used as a framework to
evaluate any substance abuse prevention strategy.
The SPF-SIG evaluation is intended to help your county monitor strengths, weaknesses, and
effectiveness of your project so that you may make informed decisions about future
prevention efforts within your county.
If you have any questions at any time regarding this process, please feel free to contact
Patrick McGovern, lead evaluator, your capacity coach, or IDPH project staff.
Why evaluate?
Evaluation is a systematic way of assessing your initiative. It helps you understand
whether there has been positive impact on your county’s priority issue(s), as well as the
associated intervening variables. The results of evaluation may be used to refine strategy
implementation, concretely illustrate progress toward project goals and objectives, and
even solicit funding for additional prevention efforts. The goal of the SPF is to implement
strategies that “fit” with your population’s needs, as identified during the needs assessment
phase. Evaluating your progress can help you to determine whether the strategies do, in
fact, address your county’s needs and whether they have been effective.
In addition to monitoring the effectiveness of strategies in your county, evaluation can also
provide you with information on how to proceed with prevention strategies in the future.
For example, did any unforeseen circumstances or needs prevent you from implementing a
strategy as planned? Did any strategies face a substantial amount of reluctance or
excitement by participants? Did the plan result in improvements in your priority issue?
The answers to these questions can help you prepare for future prevention efforts.
The language of evaluation comes from many disciplines, and can be confusing. Here is a
list of some common evaluation terms:
3
Common Words and Terms
(Excerpted from “A Word About Words” in CADCA’s Evaluation Primer)
What you want




Aim
Goal
Objective
Target
What you do to
get there
 Activity
 Input
 Approach
 Initiative
 Method
 Policy
 Practice
 Program
 Strategy
Are you getting there?







Output
Benchmark
Indicator
Measure
Milestone
Short-term
Outcome
Intermediate
Outcome
Did you get there?



Impact
Outcome
Results
Five functions of evaluation
A sound evaluation serves five key functions, including improvement, coordination,
accountability, celebration, and sustainability. During the regional trainings, the capacity
coaches provided an overview of each function. A summary is shared again here:
IMPROVEMENT: Volunteers, leaders, and supporters should get better at the work of
county problem solving because of what they learn.
COORDINATION: SPF efforts are made up of many partners working on different parts of
an overall response to county alcohol problems. Keeping these partners and activities
pointing in the same directions can be difficult unless the evaluation fosters coordination.
The information should help members know what others are doing, how this work fits with
their own actions and goals and what opportunities exist for working together in the
future.
ACCOUNTABILITY: Volunteers want to know if their time and creativity make a difference.
Funders want to learn how their money factors in county improvements. Everyone
involved in the process want to see outcomes. A good evaluation allows the coalition to
describe its contribution to important population–level change.
CELEBRATION: A stated aim of any evaluation process should be to collect information
that allows for the celebration of genuine accomplishments.
SUSTAINABILITY: The path to reduced alcohol use can be long, often requiring years of
hard work to see movement in population–level indicators. Evaluation should help your
group stay “in the game” long enough to make a difference by sharing information with key
stakeholders and actively reinforcing their continued support.
4
Cultural competence and evaluation
A culturally competent evaluation calls attention to questions of diverse stakeholders and
cultures, and includes representatives of the population served in the design process. The
data collection process should be culturally appropriate for all members of the target
community. Once data are collected, representatives from the target population should be
involved in the interpretation and dissemination of results.
Sustainability and evaluation
Evaluation is a key ingredient of sustainability. Evaluation data tracks trends and can be
used to modify or update your strategic plan. Evaluation data may also be used to
demonstrate additional needs, in turn justifying more funding. Evaluation also provides
information to raise awareness and to strengthen community support for your actions.
Evaluation provides a system for continuous improvement by identifying what is working
and what needs to be revised.
Directions for Completing your SPF-SIG Evaluation Plan
The SPF-SIG Evaluation Plan is a required component of your implementation of the SPF
process. The following directions are intended to help and guide your work. However, you
are encouraged to develop and add to your plans to ensure that you collect meaningful and
useful data for you, your coalition, and your county.
Overview of Evaluation Guidance Document
Please find the Evaluation Guidance Document on the Iowa SPF-SIG website in the SPF
Resources: Evaluation: Evaluation Resources folder. This document is organized by
strategy (the orange rows), with at least two questions (found in column A) per strategy.
The first questions are cross-cutting questions that should be applied for all strategies;
these questions address cultural competence, sustainability, use of the media campaign(s),
and a summary of how the strategy has worked within your community. Column B
contains suggestions as to where the data may be found. Much of the data needed to
answer the questions can be found in the Online Documentation and Support System
(ODSS); the key for these codes may be found in Appendix 1 or refer to the ODSS User
Manual. Column C contains the approach type, as defined by CADCA. See Appendix 2 for
additional information about these approaches. The Approaches column is included to
help you identify the fit; these are recommendations and may not be accurate for all
situations across all counties. Column D contains the measure, a description of the data to
be used.
Reporting Requirements
All questions included for a strategy must be included in your evaluation plan. You may
add additional items to your plan that you would find useful within your county, but you
must at least include the questions associated with your identified strategies. To complete
this process, you should copy the strategy-specific questions from the Evaluation Guidance
document into Table 1 beginning on page 7. This table already includes the cross-cutting
questions that should apply to almost all strategies; if not applicable (e.g. no modifications
5
made to fit unique cultural groups) then the cross-cutting question(s) may be deleted. You
may choose to copy the Measure(s) and Data Source(s), but this is not necessary. Once you
have discussed and decided upon your county-specific measures and data sources, you may
enter them in their respective boxes in the table (if different).
The Measure(s) should be a clear and concise description of the data that will be used to
answer the question. When possible, this should include an estimated timeline for data
collection, who collects it, and (where applicable) the number of times the strategy was
implemented (i.e. the number of server trainings offered.) For example, for the Alcohol use
Restrictions in Public Places strategy, one measure might read, “The number of alcohol-free
signs displayed during community festivals by calendar year, as reported by the LEW in the
Community Assessment Workbook.”
The Data Source(s) should be completed with one source listed whenever possible. If some
uncertainty exists about whom or what organization collects those data, multiple sources
may be listed. If this occurs, you will need to clearly identify the data source used when
you submit your data or reports.
If you are implementing more than six strategies, please copy and paste the table into a
blank page to accommodate additional strategies. Once you insert your questions, you may
delete any blank rows. For assistance doing either of these options, or with any questions
about this process, please contact Pat McGovern at: patrick-mcgovern@uiowa.edu.
Once you have completed the table, please write a short paragraph describing how you
intend to collect information to address the questions for each strategy. This information
should be inserted below to replace the line that begins with, “Insert paragraph.” As
implementation continues, these paragraphs will be modified to include a summary of the
data collected and any changes to your plan.
Strategy (Insert Strategy Name):
Insert paragraph description. Repeat for each strategy.
6
Table 1. County Evaluation Plan
Question
Measure(s)
Strategy: Insert Strategy Name
What modifications or adaptations were applied to address unique cultural groups?
What efforts were undertaken to sustain this strategy?
What media advocacy was used with this strategy?
Provide a brief summary of the process taken on each component of the strategy.
Strategy: Insert Strategy Name
What modifications or adaptations were applied to address unique cultural groups?
What efforts were undertaken to sustain this strategy?
What media advocacy was used with this strategy?
Provide a brief summary of the process taken on each component of the strategy.
7
Data
Source(s)
Table 1. County Evaluation Plan (continued)
Question
Measure(s)
Strategy: Insert Strategy Name
What modifications or adaptations were applied to address unique cultural groups?
What efforts were undertaken to sustain this strategy?
What media advocacy was used with this strategy?
Provide a brief summary of the process taken on each component of the strategy.
Strategy: Insert Strategy Name
What modifications or adaptations were applied to address unique cultural groups?
What efforts were undertaken to sustain this strategy?
What media advocacy was used with this strategy?
Provide a brief summary of the process taken on each component of the strategy.
8
Data
Source(s)
Table 1. County Evaluation Plan (continued)
Question
Measure(s)
Strategy: Insert Strategy Name
What modifications or adaptations were applied to address unique cultural groups?
What efforts were undertaken to sustain this strategy?
What media advocacy was used with this strategy?
Provide a brief summary of the process taken on each component of the strategy.
Strategy: Insert Strategy Name
What modifications or adaptations were applied to address unique cultural groups?
What efforts were undertaken to sustain this strategy?
What media advocacy was used with this strategy?
Provide a brief summary of the process taken on each component of the strategy.
9
Data
Source(s)
Appendix 1: ODSS Codes
Code Title
Definition
DA
Developmental
Activity
New or modified programs, policies, or practices
in the community facilitated by SPF-SIG and
related to your goals and objectives
Activities performed by members of the initiative
or group to bring about a new or modified
program, policy, or practice in the community
related to your goals and objectives (acting
directly to make changes in your community,
actively lobbying, or advocating with change
agents. Ex. personal contacts, phone calls,
demonstrations, petitions, and letter writing).
The delivery of information, training, materials,
or other valued goods or activities by project
team members to people in the community. (Ex.
classes, programs, screenings, workshops,
communications).
Actions taken to prepare or enable the project to
address its goals and objectives (e.g., attending a
training session for self-development, working on
a strategic plan, receiving CSAP training).
M
Media
Coverage of the project or its activities through a
media outlet
RG
Resources
Generated
O
Other
CC
CA
SP
Community Change
Community Action
Services Provided
Acquisition of funding or other resources for the
project through grants, donations, or in-kind gifts
Additional activities that are recorded for which
no code or definition has been created.
10
Criteria:
Have occurred; related to priority area; new or
modified strategy in different part of the community;
facilitated by SPF-SIG representatives; must be first
occurrence
Have occurred; related to priority area; taken to
bring about community change; facilitated by SPFSIG representatives
Have occurred and/or ongoing; conducted by SPFSIG representatives; delivered to community;
information training, material goods, or other
services
Have occurred; provided to members of the project
or those acting on project's behalf; are not (or not
yet) other codable steps
Occurred; instance of coverage; feature the SPF-SIG
project or its activities OR coverage is facilitated by
the project
Occurred; in the form of money, materials, or
donated time; used to facilitate activities related to
the SPF-SIG priority areas; allocated to the SPF-SIG
project
Appendix 2: CADCA Seven Approaches
1. Provide information—
 Educational presentations
 workshops or seminars
 data or media presentations
 public service announcements
 brochures
 billboard campaigns
 community meetings
 town halls
 forums
 Web-based communication
2. Enhance skills—
 Workshops, seminars or activities designed to increase the skills of participants,
members and staff
 Training
 technical assistance
 distance learning
 strategic planning retreats
 parenting classes
 model programs in schools
3. Provide support—
 Creating opportunities to support people to participate in activities that reduce risk
or enhance protection
 providing alternative activities
 mentoring
 referrals for services
 support groups
 youth clubs
 parenting groups
 Alcoholics or Narcotics Anonymous
4. Enhance access/reduce barriers**—
 Improving systems and processes to increase the ease, ability and opportunity to
utilize systems and services
 access to treatment
 childcare
 transportation
 housing
 education
11


special needs
cultural and language sensitivity
**Note: This strategy also can be utilized when it is turned around to reducing
access/enhancing barriers.
 Establish barriers to underage drinking or other illegal drug use, they decrease its
accessibility.
 When more resources (money, time, etc.) are required to obtain illegal substances,
use declines.
 Mandating the placement of pseudoephedrine-based products behind the pharmacy
counter- significant decrease in local clandestine meth labs.
5. Change consequences (incentives/disincentives)—
 Increasing or decreasing the probability of a specific behavior that reduces risk or
enhances protection by altering the consequences for performing that behavior
 increasing public recognition for deserved behavior
 individual and business rewards
 taxes, citations and fines
 revocations/loss of privileges
6. Change physical design—
 Changing the physical design or structure of the environment to reduce risk or
enhance protection
 parks
 landscapes
 signage
 lighting
 outlet density
7. Modify/change policies—
 Formal change in written procedures, by-laws, proclamations, rules or laws with
written documentation and/or voting procedures
 workplace initiatives
 law enforcement procedures and practices
 public policy actions
 systems change within government, communities and organizations
12