Cheryl Staats
September 2010
Stage 2 - Defining Your Work: Thinking about the Work to be Evaluated

Section 2a – The outcomes you want to achieve
o Guiding questions
 “1. What specific changes does the Group expect to make in the community through
its chosen strategies?
 2. Does the group have a clear understanding about how these changes are related
to the community outcomes it cares about?
 3. Has the group discussed and come to an understanding about reasonable
expectations for the degree or type of change that might be observed at this time?
Has it considered what different audiences might expect and how to manage those
expectations?”
o Teminology:
 Outcomes are desired states of well-being or desired end states. An example
outcome would be the “formation of effective collaborative structures and
relationships.”
 Indicators measure conditions and can help you assess the progress you are making
towards desired outcomes.
 Characteristics of good indicators: They are meaningful (have face validity),
powerful (proven at predicting outcome change), actionable (possible to
change through the organization’s strategies), measurable (defined
precisely and clearly), reliable (high inter-rater reliability), available or
feasible to collect, universal (data are available for all members of a
population), and used to reveal differences (can be disaggregated into
relevant categories).
 An evaluation guidebook by the Sierra Health Foundation can be found here:
http://www.evaluationtoolsforracialequity.org/evaluation/tool/doc/PagesSierraGui
de02.pdf

Section 2b – The theory behind your work
o Guiding questions
 “1. Has the group linked its analysis of the problem, community change goals, and
strategies?
 2. Has the group developed an understanding about how change is expected to
happen – how its strategies are expected to influence the community?
 3. Has the group assessed the appropriateness, feasibility, and likely effectiveness of
its strategies? Are the strategies powerful and broad enough to influence the
community’s outcomes?
 4. Has the group identified any external conditions or internal capacities that need
to be in place so that these strategies will work? Likely barriers or areas of
resistance to change?”
Page 1 of 6
Cheryl Staats
September 2010
o
o
o
“Evaluators often ask groups to create two tools: one is called ‘a theory of change’ and the
other is called a ‘logic model.’”
 Theory of change- a pathway or series of steps that culminate in change. The group
has to understand the assumptions they are making behind their theory of change.
Group members are able to see whether they have included all of the steps
necessary for change, allowing for dialogue regarding what steps are in fact
necessary.
 “Once a theory of change is developed for a strategy, one of its main
benefits is helping identify expected results along the way to the hoped-for
community change.”
 A logic model – “is usually in the form of a chart. The chart also lists the steps that
the group is planning to implement, usually as particular activities or strategies. The
chart also lists what the group thinks the important results of those steps will be,
usually in the form of short-term, intermediate and long-term outcomes.” This
helps groups working towards social equity, antiracism, and inclusion goals a way to
discuss different groups’ assumptions about what is success so that reasonable
expectations can be created.
 Logic models help evaluation because they lay out outcomes in varying
intervals that can be used to measure progress.
 Logic Model development guide from the W.K. Kellogg Foundation:
http://www.evaluationtoolsforracialequity.org/evaluation/tool/doc/pub366
9.pdf
 Online logic model building tool from University of Wisconsin:
http://www.uwex.edu/ces/lmcourse/#
 Being able to measure interim results/progress is important since much anti-racism
work is long-term.
Strategies – “sets of activities, services, or programs that are intended to accomplish the
same ends.”
 Example: “Conducting a public awareness and education campaign about racial
disparities in educational opportunities and outcomes is a strategy. Activities that
may be part of that strategy include public service announcements on radio and
television, editorials and commentary in print media, a web site, presentations at
public meetings, distribution of a community report card, etc.”
Questions to help the group pick strategies and activities that have the best likelihood of
success and the biggest potential impact:
 “Are the specific ways in which our strategies address the identified problems and
contribute to the desired outcomes clearly spelled out? Is there a well-described
pathway from each activity or program to its expected immediate effects, and from
these effects to the next expected change, and so on?
Page 2 of 6
Cheryl Staats
September 2010



o
o

Do we have evidence -- from research, from best practices, or from experience -that suggests that the activities or programs of our strategies (if they are
implemented well), and their effects, will make a difference in the ways we expect?
Is the strategy going to be implemented broadly, with sufficient quality, intensity
and duration, to make a difference in what you will be able to observe about the
expected effects?
Have external factors that might weaken our strategies or lessen their effects been
identified? What has been done to address these factors, or to reduce their
consequences, if that is possible? If not possible, what information will be available
to take these factors into consideration when evaluating the success of our
strategies?”
 Six Stages of Change1:
Community change is rarely linear; it’s important monitor trends, use multiple methods of
observation and change documentation, and evaluate strategies at multiple points in time.
“The theory of change for anti-racism work should include strategies for anticipating,
identifying and addressing resistance in the community. That will not only improve the
work's overall effectiveness, but also allow the evaluation to include these strategies in its
assessment.”
Section 2c – The evaluation questions to be answered
o Guiding questions:
 “1. Who are the most important audiences for evaluation findings and what does
each want to know?
 2. What questions does the group need to answer about how well or broadly its
chosen strategies are being carried out?
1
“Long Term Anti-Racism Strategies: A Guide to Developing Effective Community Projects.” (2001) Affiliation of Multicultural
Societies and Service Agencies of BC, http://www.evaluationtoolsforracialequity.org/evaluation/tool/doc/ltars.pdf
Page 3 of 6
Cheryl Staats
September 2010

o
o
o
o
3. What questions does the group need to answer about whether and how its
strategies are making a difference in the specific things it is trying to change
immediately in the community?”
Why measure a group’s programs and activities?
 Descriptive – for a report or historical record
 Check on its operations and use of resources
 “Assess whether it was possible to carry out the intended activities and programs
and reach and engage the intended individuals or groups.”
Data that you may want/need to include:
 Activities – Which ones occurred? “When? In what order? By whom? Using what
resources? Targeted to which individuals or groups, and to how many? Reaching
which individuals or groups, and how many?”
 Services or programs – Which were delivered? “How many? By whom? To whom?
When? Where? How many of the intended set of individuals or groups received
any services or participated in any way? What was the intensity or ‘dose’ of what
they received? That is, how much service or support did they receive on average, or
how long did they participate on average? How many were considered to have
completed the program or received the appropriate amount of services?”
“Evaluation of implementation and participation should be planned for those activities and
programs that meet one or more of the following conditions:
 Are critical to the overall success of the strategy
 Require substantial resources
 Are highly visible
 Are innovating or involve significant changes to a proven effort
“Evaluation questions about the implementation of activities and programs tend to focus on
the “why” of implementation success or failure and how this relates to the difference the
programs or activities made on the things the group hopes to change.”
Stage 3 - Designing: Evaluation Design and Plan

Section 3a – Evaluation design
o Guiding questions:
 “1. What kinds of information are needed to answer the evaluation questions?
What information about the community outcomes of interest does the group need
to track?
 2. What type of evidence is needed to:
 a. demonstrate that the desired community changes are happening?
 b. evaluate the success of the group’s chosen strategies and activities?
 c. assess whether or to what extent the group’s strategies are contributing
to community change?
Page 4 of 6
Cheryl Staats
September 2010

o
o
o
o
3. Have issues of cultural competence been considered in developing the evaluation
design?”
Evaluation can be expensive, so there’s a need to compromise between the evaluation goals
and rigor and evaluation cost.
 General rule of thumb: Evaluations should cost approximately 10% of total project
budget
Things to consider when deciding how much effort and resources to expend on evaluation:
 What evaluation questions need answered now? It can be important to measure
implementation (make sure strategies are reaching the right people), but if you have
to choose, devoting resources to measuring results rather than implementation is
likely to be preferable.
 “What decisions will be made from evaluation information and what are the
consequences of those decisions for your constituencies?
 Have the strategies of the group been in place long enough for their implementation
to be assessed and/or to make it likely that their effectiveness can be evaluated?
 What do key stakeholders expect regarding the type and quality of evidence from
the evaluation?
 What research or evaluation expertise and skills do members of the group have or
can draw on from within the community? What data are already available for the
evaluation?”
Many factors can combine to create community change, so when changes in the community
are observed, it is both important and difficult to judge whether the change was the result
of a strategy that was implemented or other factors.
Although a control group evaluation design is often the “gold standard” in the social
sciences, it is often not possible to create. But there are other evaluation designs:
 Comparison group design – Matching subjects who are involved in an
intervention/strategy with people of similar characteristics who are not involved.
This isn’t as good as a control group design, because other differences between the
groups may interfere in the analysis.
 Pre-post observations – This design measures change in a group, but it can be
unclear whether the same change would have occurred without the intervention or
strategy. Or, if there is no sign of change (or even a reversal), it is still possible that
the strategy had an effect at slowing the process, though this would be hard to
defend without other evidence.
 Inferential designs – These designs draw conclusions based on finding consistent
patterns of results (make inferences from evidence related to underlying
assumptions). “Using a theory of change, and using it to guide evaluation, lends
itself well to this approach.”
 “A related approach is called ‘results mapping’ in which in-depth data
collection and analysis of a number of specific instances of the observed
pattern of change are conducted. The focus of this approach is to create a
“map” of strategies and actions and their results in individual cases,
Page 5 of 6
Cheryl Staats
September 2010


accumulating evidence across a series of cases to understand the extent to
which planned activities lead to the expected changes in outcomes. This
approach is especially helpful when the outcomes are a result of a complex
set of events and/or involve healing, transformation or prevention. See
http://www.pire.org/resultmapping.”
Additional document: “Principles for Evaluating Comprehensive Community
Initiatives,” prepared by The Association for the Study and Development of
Community on behalf of the National Funding Collaborative on Violence Prevention.
June 2001.
http://www.evaluationtoolsforracialequity.org/evaluation/resource/doc/CVP06200
1.pdf
Section 3b – Evaluation plan
o Guiding questions:
 “1. Is there a work plan and budget for the evaluation? Are the resources available
consistent with the design requirements? If not, will the design be modified, or will
the evaluation wait until more resources are found?
 2. Have the responsibilities for the evaluation been assigned? Will outside assistance
be needed at any point in the evaluation, and if so, have technical assistance
providers (including external evaluators) been recruited?
 3. Specifically how does the group want to involve community members in the
evaluation – at what stages and in what capacities? Is the group willing and able to
commit the time, attention, and resources that may be needed to do this?
 4. Is there a schedule with benchmarks so that progress on the evaluation can be
tracked? Is there a team responsible for monitoring the evaluation and keeping the
whole group informed about its progress?”
o Other questions that generally fall in this area:
 What is the clearly-articulated purpose of the evaluation? What questions will be
the focus of the evaluation?
 Who will be the audience for the evaluation?
 What type of evaluation will be performed?
 In regards to cultural competency, is the evaluation model we are using meaningful
and relevant to our target population?
 What indicators will be used?
 What will serve as the data source(s)?
 How will the data be collected? What resources are available for this collection?
 How will the data be interpreted, and by whom?
 How will the evaluation results be shared?
o Additional resource:
 Taylor-Powell, Ellen, Sara Steele, and Mohammad Douglah. “Planning a Program
Evaluation” University of Wisconsin-Extension. (February 1996).
http://www.evaluationtoolsforracialequity.org/evaluation/tool/doc/G3658_1.PDF
Page 6 of 6