Leadership Development Evaluation Handbook
First Draft – August 2005
Theory of Change Approach to Leadership
Development Evaluation
Manuel Gutiérrez & Tania Tasse
1
Theory of Change Approach to Leadership
Development Evaluation
Manuel Gutiérrez and Tania Tasse
OMG Center for Collaborative Learning
Background of Theory of Change Evaluation
The theory of change approach to evaluation gained
popularity and wide acceptance in the 1990’s through its
innovative use in the evaluation of comprehensive community
initiatives.
Given the inherent complexity of these initiatives
– they are typically neighborhood-based and seek multiple-level
outcomes across several programmatic areas – evaluators found
serious limitations in traditional evaluation approaches and
were forced to come up with other approaches that would be
suitable for its application to these initiatives.
The work of
the Aspen Institute Roundtable on Comprehensive Community
Initiatives for Children and Families was extremely important in
legitimizing and expanding the understanding of the
applicability and limitations of the theory of change evaluation
approach (see Connell, Kubisch, Shorr, and Weiss, 1995;
Fulbright-Anderson, Kubisch, and Connell, 1998).
Notably, the
Aspen Roundtable provided a forum for funders, evaluators, and
2
practitioners to refine the approach by sharing tools and
lessons learned from conducting theory of change evaluations.
The basic description of a theory of change approach to
evaluation was defined by Carol Weiss in her seminal paper,
Nothing as practical as good theory, included in the first Aspen
Institute Roundtable publication on the evaluation of community
change initiatives(Weiss, 1995).
Essentially, Weiss proposed
that a theory of change approach requires that the designers of
an initiative articulate the premises, assumptions, and
hypotheses that might explain the how, when, and why of “the
processes of change”. As part of this approach, designers and
key stakeholders are asked to identify key programmatic elements
and to indicate how these interventions might lead to the
anticipated short-term, intermediate, and long-term outcomes.
This process is commonly referred to as articulating an
initiative’s or a program’s theory of change.
Once the theory
of change is made explicit, then it becomes possible for the
evaluator to test the assumptions that underlie the initiative
or program and to assess its outcomes.
3
Distinction between Theory of Change and Logic Models
Logic models have been used in planning and evaluation for
a long time, preceding the popularization of theory of change
evaluation.
However, they have gained increased attention in
the evaluation field, as they are essential tools in the
articulation of a program’s theory of change. While linked and
complementary, these terms are often used interchangeably,
leading to unnecessary confusion. In Table 1, we provide our
understanding of fundamental distinctions between the terms
theory of change and logic models (see Kellogg Foundation, 2003,
for an excellent guide on the development and use of logic
models).
Table 1. Basic distinctions between theory of change and
logic models.
Theory of change
Logic models
Overarching set of premises
Systematic visual
and hypotheses that seek to
representations of the
explain why and how change
relationships between
occurs
program elements and the
sequence of events that will
lead to change
Higher order explanation;
More detailed description;
4
panoramic view
frame by frame view
Provides reasoning to guide
Provide roadmap, based on
program design, planning,
theory of change, for
implementation, and
program design, planning,
evaluation
implementation, and
evaluation
Conceptual framework with
Evaluation and planning
explanatory power
tools for articulating and
describing the theory of
change
Describes the principal
Describe the anticipated
assumptions that will be
short-term, intermediate,
tested in the evaluation
and long-term outcomes that
will be assessed in the
evaluation
Theory of Change Evaluation and Leadership Development:
A Good Fit
There are many valid approaches for conducting an
evaluation of a leadership development program.
Some of those
approaches are presented by other authors in this Handbook.
Often, factors such as the characteristics of the specific
5
program, the nature of the evaluation questions, and the
resources designated for the evaluation are likely to influence
the type of evaluation that will be selected. We believe that,
in most cases, the theory of change approach would be a good fit
for evaluating leadership development programs for the following
reasons:
Leadership and leadership development are terms with multiple
meanings and definitions. Given the various definitions of
leadership and the wide array of existing leadership development
programs, the theory of change approach focus on articulating
premises and assumptions is an effective process for clarifying
a program’s view of leadership and how this view shapes program
activities.
Leadership development is a complex psychological and social
process. Describing the change process for individuals
participating in leadership development programs may be as
complex and challenging as describing the neighborhood change
process in comprehensive community initiatives.
For that
reason, a theory of change evaluation represents a promising
approach for systematically tracking and understanding the
personal change process in individuals.
6
Leadership development programs typically involve multiple
components and use various interventions.
In this case, the
observed complexity derives from the nature and layering of the
interventions.
Given this programmatic complexity, the theory
of change evaluation approach can help ascertain how and to what
extent different program components contribute to the attainment
of anticipated outcomes.
Leadership development programs typically hold the expectation
that individual-level changes will lead to organization-level,
system-level, and society-level outcomes.
In this context, the
theory of change approach provides a framework that articulates
the anticipated pathways of change and allows evaluators to
gather data in order to test out whether, to what extent, and in
what contexts individual-level change leads to broader outcomes.
Pathway Mapping and Pathway Maps
Staff at the OMG Center for Collaborative Learning (OMG)
have been conducting theory of change evaluations for over ten
years.
Over this time, through multiple engagements involving
the evaluation of comprehensive community initiatives and
single-focus programs, we have used a process called pathway
mapping to engage stakeholders and articulate the initiative’s
or program’s theory of change.
More specifically, pathway
7
mapping is the process of specifying a program’s desired
outcomes and linking those outcomes to program actions and
strategies. The pathway mapping process also requires making
program assumptions explicit, challenging them when they appear
inconsistent or unclear, and reaching consensus on those program
assumptions and sequence of events that describe why and how a
program works – the pathways to change.
The pathway mapping process produces pathway maps.
Pathway
maps are logic models that place emphasis on the surfacing of
assumptions linking strategies, activities, and outcomes.
We
prefer using the term “pathway map” rather than “logic model”
because it connects the process (mapping) to the product
(map)and emphasizes the concept of “pathways of change” that
links to the program’s theory of change.
For us, then, a
pathway map is a particular type of logic model.
Figure 1 provides an example of a pathway map for a
community-based employment program.
In this example we present
the basic elements of a pathway map: a contextual analysis for
the program, the program’s strategic focus, its
actions/activities, and its intended outcomes (short-term and
long-term improvements).
Very importantly, we also include the
program’s core assumptions that underlie the connections between
the basic elements of the pathway map. In the next section, we
8
provide more detailed examples of a theory of change evaluation
approach and pathway mapping.
Insert Figure 1 about here
Pathway maps and leadership programs
Representing the theory of change
When designing evaluations for leadership programs,
evaluators are likely to be challenged by broad and ambitious
program objectives, which makes it difficult to know exactly
what to look for when assessing program success.
In addition,
staff and stakeholders of the programs may not be totally clear
-or in agreement about- how and why their program activities
should lead to the objectives, and what may be the signs of
progress along the road to success.
To help overcome these challenges, OMG relies on a pathway
mapping process as the first step in a theory of change
evaluation of a leadership program.
As we indicated previously,
in pathway mapping, staff and stakeholders of a program make
explicit their theories and assumptions about how the program
works from start to finish.
The final product of this process,
a pathway map, is a special type of logic model or flowchart
9
documenting how program activities (inputs) are believed to
create results (outcomes) over time.
Over the past few years, OMG has used a theory of change
approach to evaluate three established leadership development
and recognition programs.
These programs are: the Rockefeller
Foundation’s Next Generation Leaders (NGL), the Eisenhower
Fellowships (EF)’s international exchange programs, and the Ford
Foundation’s Leadership for a Changing World (LCW). OMG was
hired to conduct impact assessments of these three programs.
The LCW and NGL evaluations have included implementation
studies, as well.
These three programs are quite diverse in their views on
leadership, their goals and intended outcomes, the
characteristics of program participants, and the strategies used
to achieve their goals (for more information on these programs,
go to their websites listed in the References section).
Nevertheless, in spite of their diversity, we found that the
theory of change approach has provided a flexible framework for
evaluating each program (for more information on the NGL and EF
evaluations, see Gutiérrez and Stowell, 2004, and Gutiérrez and
Tasse, 2005).
10
Details of a pathway map
The Eisenhower Fellowships’ pathway map shown here includes
program inputs, such the nomination and selection processes that
precede the fellowship, joint program planning, the actual
fellowship experience, and resources provided to alumni.
The
pathway map also details outcomes that are expected to occur for
the individual fellows (for example, enhanced professional
knowledge), for their organizations (improvements and/or growth
in the organization), and for their communities or societies
(new programs, institutions, policies being put in place).
Ideal outcomes were included in this pathway map to convey the
vision that the program is ultimately striving to contribute to;
however, they are formatted differently in the pathway map to
indicate that these are ideals and that the program does not
necessarily expect to be able to measure progress towards them.
The arrows from left to right reflect the order in which the
Eisenhower Fellowships believe outcomes develop.
Insert Figure 2 about here
Pathway maps may vary extensively from client to client,
not only in terms of content but often in terms of format and
layout.
For example, while EF chose to depict outcomes
according to the level at which they occur (individual-level,
organizational-level, and societal-level), many programs prefer
to state their outcomes according to the time period over which
11
they are expected to occur, and thus they use the category
headings Short-term, Intermediate, and Long-term outcomes.
Another common variation is that, in addition to program inputs
and outcomes, some pathway maps include a Problem Statement or
Contextual Statement describing why the program was first
started, under what conditions, and to address what problem/s.
For example, Leadership for a Changing World’s pathway map
begins with a Contextual Statement that includes the following
language: “There is a lack of conviction in the public that
leadership exists and that local/community leaders can impact
social issues.”
The length and degree of detail shown by a pathway map are
other variables to be decided upon.
OMG has worked with
programs to produce in-depth pathway maps and also to produce
simpler ones.
To fully describe the conditions that led to a
program, the activities and resources that comprise the program,
and the assumptions delineating how and why outcomes are thought
to occur, requires a significant amount of time.
The resulting
document is generally several pages long and requires the full
attention of a reader to grasp.
This type of pathway map is
valuable for some purposes, including building consensus around
program theory when establishing a new program, and re-thinking
program design.
Regardless of the length of the document,
evaluators working with program staff need to balance the level
12
of detail provided on a pathway map with the level of
information required to understand the program.
Typically, the
inclination is to add as much detail as possible to the pathway
map.
However, very detailed pathway maps will look cluttered
and will be hard to interpret.
Shorter pathway maps (like the Eisenhower Fellowships
pathway map) are generally 1-2 pages long and highlight only the
key activities and expected outcomes of a program.
These
pathway maps may be less helpful in presenting why the program
exists, or the details and logic behind the program and
activities.
On the other hand, they are simpler to understand
and thus they can be used to illustrate a program’s logic to
audiences that are unfamiliar with the program, or to those who
do not require all of the details behind it.
In general, the
content, format, and layout of a pathway map should be presented
in whatever style best reflects the program at hand; and the
degree of detail should be decided based on why the program is
doing the pathway map and how they are likely to use it in the
future.
Even when pathway maps are presented in a more simple
layout, key elements of the theory of change such as program
context and assumptions are presented as accompanying text.
No matter what the finished document looks like, if a
pathway map effectively captures the logic behind a program,
then it can be used as a roadmap to guide the evaluation.
It
13
specifies what results evaluators should look for, and it often
specifies at what time periods those results should occur.
Pathway maps also provide a list of key program activities,
which is critical if the evaluation includes an assessment of
the program’s implementation.
Once the pathway map is
completed, the evaluator’s next step is to select methods and
design data collection activities.
It is important to note that, in the absence of an
experimental design (which is not a realistic design for the
evaluation of leadership programs) or a comparison group design
(which is difficult to implement), the theory of change
evaluation approach provides a reasonable alternative to the
limitation of the lack of a counterfactual.
If an evaluation
can show that program activities were fully implemented and that
outcomes developed in the way envisioned by staff and
stakeholders and documented on the pathway map, then credibility
is built for the program theory.
In other words, it becomes
easier to believe that the program contributed to outcomes in
the way depicted by the pathway map.
Usefulness of pathway maps
In addition to providing a framework for assessment
activities, OMG finds that pathway maps also provide a useful
structure for evaluation reports.
The first section of a report
14
would describe the logic and assumptions behind the program, as
well as the inputs and expected outcomes.
The following
sections would describe whether or not the program was
implemented according to its design, and whether outcomes
occurred in the way envisioned by the staff and stakeholders in
the pathway map.
Beyond their usefulness to evaluators, pathway maps can be
valuable tools for program staff and administrators because they
provide a level of clarity and explicitness about program theory
that many programs do not capture in writing.
Pathway maps can
be used for program planning, management, and development
purposes, as well as to present and explain programs to those
who are not familiar with them.
For example, pathway maps are
useful when orienting new staff to the mission and purpose of a
program, or to inform critical programming decisions like
whether or not “new” activities that are being considered make
sense in light of the other inputs and goals.
The inputs
section of a pathway map can be used as a programming checklist, against which a program manager may ask the questions, are
these activities in place, and if so, are they in place at the
right intensity to lead to these outcomes?
Or, if certain
activities are not in place, then which outcomes are not likely
to occur?
15
Additionally, when a program uses a pathway mapping process
as part of the program design phase, it increases the likelihood
that everyone involved has a shared understanding about how the
program will look and work, and what is expected to happen as a
result of it.
OMG conducted pathway mapping with one program
during the design phase, and the staff of the program continue
to use the document to re-examine and refine program activities
and logic.
In this respect, it is important to note that a
pathway map should be considered a dynamic tool for
incorporating evaluation findings, reflecting mid-course
corrections, and capturing other program evolutions.
A pathway
map should not be considered as “etched in stone”, but more as a
roadmap that is likely to show some changes over time, given new
knowledge about the program.
Those changes, of course, would be
an important part of the story that a theory of change
evaluation should be able to tell.
[SIDEBAR]: Creative Pathway Maps
Some organizations have developed creative ways to present
their programs to others.
ARISE, a grassroots organization in
Alamo, TX, that combines personal development, leadership, and
community organizing training for women living in southern Texas
colonias, created a visual and kinesthetic representation of
their work consisting of multiple footprints and hearts spread
16
across the floor (each footprint or heart is drawn on a piece of
paper so that they can be used repeatedly).
Inside the
footprints and hearts are ARISE’s guiding principles and values
statements.
The footprints “walk” across the floor, scattered
among the hearts.
They lead to the organization’s vision
statement, which is displayed on the wall.
Although ARISE doesn’t use the term “logic model” or
“theory of change” to describe this depiction of their program,
it serves a similar purpose as a pathway map: to communicate the
program’s logic and to connect the everyday activities of ARISE
to the organization’s vision.
Sister Gerrie Naughton, founder
and Executive Director of ARISE, came up with the idea for this
tool, and the program participants created the footprints,
hearts, and vision statement.
Sister Gerrie says that the
guiding principles, values, and vision statement already existed
on paper, but that the tool was developed because there was a
need to see – to visually connect – how the everyday work of
ARISE (the inputs) lead to the vision (outcomes).
ARISE uses it
for training purposes when they hire new staff.
The Pathway Mapping Process
Preparing for the mapping sessions
Assuming that OMG is asked to evaluate a program already
under implementation (as opposed to one that is being designed),
17
the pathway mapping process would begin with a review of written
program documents, including the program’s website, application
or nomination forms, participant handbooks or other materials
describing the program’s activities, annual reports, newsletters
and publications, and any prior evaluations that have been
conducted.
When reviewing these documents, OMG tries to
identify the key activities of the program; short-term,
intermediate, and long-term goals; and assumptions about how and
why the program should cause the desired goals to occur.
The next step is to interview (in person or via telephone)
key stakeholders of the program.
Interviews may be conducted
with different types of stakeholders for different purposes.
For example, it is often useful to interview the individual/s
who originally designed a program in order to gather in-depth
contextual information about how and why the program was
started.
Also, interviews should be conducted with individuals
who are influential to the program —for example, those involved
in program planning, implementation, or decision-making (staff,
executive directors, board of trustee members, etc.)— but who
may not be available to participate in the next step of the
process, the working sessions.
Lines of questioning for these
interviews include why the program was originally designed, why
or how the particular set of program activities was decided
upon, what are the program impacts that the interviewee has
18
witnessed or heard about from those who have participated in the
program, and what are the perceived strengths and weaknesses of
the program.
After the interviews, OMG schedules a series of working
sessions with key staff and any other individuals who the
program decides to include in the development of the pathway
map.
This may include individuals who have participated in the
program (alumni), board members, or former staff.
Since the
working sessions are highly interactive and their success
depends on participation from all group members, OMG recommends
that no more than 10 people be included.
Also, since the
sessions are intense, we recommend at least two separate
sessions.
In OMG’s experience, two 3-4 hour sessions are
generally sufficient to produce a basic, 1 or 2-page pathway
map.
More detailed maps may require additional sessions or
longer sessions.
The mapping sessions
Based on the initial document review and stakeholder
interviews, OMG drafts a preliminary pathway map and presents it
at the first working group meeting.
The draft serves two
purposes: first, since most people are unfamiliar with pathway
maps, it provides an example of what the document looks like;
second, concerning content, the draft gives the working group
19
something to react to instead of having to start from a blank
slate.
It is important when presenting the draft to frame it as
a preliminary document whose contents are entirely open to
change, since the working group sessions are the most in-depth
and critical component of the pathway map development process.
Over the course of the working group sessions, the group
should review the program inputs (or activities) and come to
agreement on a list of critical activities.
discuss outcomes.
Next, they should
Since most programs already have mission
statements outlining their long-term goals, it generally makes
sense to start with these and work backwards to the short-term
and intermediate outcomes.
Short-term and intermediate outcomes
are changes or results that one would expect to occur in the
interim between the program’s implementation and its ultimate
goal.
They are markers of progress along the larger road to
success.
During discussions about short-term and intermediate
outcomes, it is critical that evaluators not only identify the
outcomes but also that they surface and examine the group’s
assumptions concerning why certain activities will lead to
certain outcomes.
These assumptions can be considered the
pathways in the pathway map – they are the logic behind the
arrows leading from program inputs to program outcomes.
20
For example, it is not uncommon during pathway mapping for
staff of a program to list a set of short-term and intermediate
outcomes that make sense given the long-term goals of the
program but do not follow logically from the programs’
activities.
For example, if a program’s activities focus on
strengthening the capacity of individual leaders to do their
work, but the program lists greater public knowledge and
awareness of leadership as an outcome, then it would be
important for an evaluator to examine the assumptions of the
group.
Why do they believe activities being conducted with
individuals will affect public opinion?
Did they forget to list
key activities in their pathway map, or are the program’s
activities misaligned with its vision?
Another question for consideration during pathway mapping
is whether the pathways between activities and outcomes are
based on research results (tested theory) or represent a new
approach being tried by the program.
In some pathway maps, it
may be important to distinguish between these types of links,
and possibly to focus evaluation resources on examining the
experimental pathways since such examinations can contribute to
further learning in the field.
During the course of the working sessions, evaluators
should expect to encounter disagreement among participants.
There are at least two different types of disagreement that can
21
occur: those concerning the wording and organization of the
pathway map, and those concerning the contents and assumptions
within it.
Disagreements reflecting the first category are
fairly easy to resolve.
For example, staff of a program may
argue about whether the 15 key activities should be grouped into
three, four, or five categories/headings.
Some staff may see
three natural categories, while others see four, or five.
However, all participants agree that the 15 activities are the
key activities for the program.
A sharp evaluator can recognize
and diagnose this type of disagreement, and once participants
see that their differences are not fundamental to their
understanding of the program, it is usually resolved by
compromise.
Sometimes, these differences point to a more
fundamental disagreement about the nature or goals of the
program.
Disagreements about program content, goals, and assumptions
are a more serious challenge to the pathway mapping process.
These disagreements are more likely to arise when a program is
producing a detailed pathway map, because the more details being
depicted in the document, the greater the number of things that
staff must discuss and agree upon. When disagreements occur
around core goals or assumptions underlying a program, it is
important to hear all viewpoints and allow participants to
engage in discussion.
Sometimes, what seems like a disagreement
22
is actually the result of individuals expressing the same thing
differently.
In some cases, participants are able to work through
disagreements themselves.
(The fact that pathway mapping is a
time-intensive process usually means that participants are open
to compromises that will move the process forward.)
However, in
cases when true disagreement exists around the key activities or
goals of a program, it is necessary to seek out the input of
individual/s who designed the program and/or those who are most
knowledgeable and ultimately responsible for the program’s
direction.
Participants can look to these individuals for
clarity about the purpose and vision of the program.
Following the working sessions, OMG revises the draft and
provides the participants with a second draft.
Ideally, all the
members of the working group will come together for another
session to review, discuss, and comment on the document.
However, if not all workgroup members are able to meet, OMG will
make sure to get everyone’s comments before or after the
meeting.
OMG receives all comments and makes revisions to the
pathway map.
Assuming that there are no further issues to
clarify or resolve, the pathway map is then finalized. If any
unresolved issues emerge during the revision process, we
continue to work with the stakeholders until we are able to
reach a consensus.
23
Evaluator roles
Evaluators need to play different roles when engaging
stakeholders in the pathway mapping process.
These roles
include being a facilitator, an expert researcher, a challenger
or devil’s advocate, a mediator, a consensus builder, and a
synthesizer.
While it may be possible for an individual to
effectively fulfill all of these roles, we have found that it
works much better to have a team of two evaluators involved in
that process.
That way, roles can be shared and/or balanced, as
needed, by the evaluation team members.
As it should be evident by now, conducting effective
pathway mapping sessions with stakeholders requires sound
knowledge of program design, program implementation, and
evaluation issues as well as strong skills in group dynamics.
Since the pathway map is a critical product for, both,
evaluators and program staff, it is extremely important that the
mapping process be perceived by program staff as an opportunity
to engage in reflective practice and not just as a mechanical
exercise to satisfy evaluator demands.
Lessons Learned with Pathway Mapping
OMG’s experience provides some lessons learned for others
who may use pathway mapping in the future.
The first lesson
concerns the usefulness of benchmarking, or indicating to what
24
degree an outcome is expected to occur.
This is important
because it allows evaluators to interpret findings related to
program outcomes.
For example, EF’s pathway map lists “new
and/or improved activities, programs, or institutions benefiting
a broad constituency” as a program outcome. During OMG’s
evaluation, data collection focused on assessing the extent to
which fellows had contributed to such improvements.
We found
that approximately 50% of fellows had contributed to this
outcome; however, we didn’t know whether 50% was low,
acceptable, or high.
finding.
We did not know how to interpret the
As a result, we recommend that the pathway mapping
process include some discussion of how much or to what extent a
particular outcome is expected.
Another word of caution concerns the level of outcomes that
programs include in a pathway map. As mentioned previously,
leadership programs tend to have very broad and idealistic goal
statements, for example “mutual understanding and peace” or
“changing the way leadership is understood.”
While these may be
effective visioning statements for galvanizing an organization,
they should not automatically be incorporated into a pathway
map.
The outcomes listed in a pathway map should be ones that
can serve as accountability measures.
They should be specific
enough and realistic enough that a program would feel
25
comfortable being held accountable to them.
For evaluation
purposes, the outcomes need to be measurable.
Finally, there is a lesson about having clients understand
the value of pathway mapping.
Since accountability is the
primary concern of most clients who seek evaluation services,
they are most interested in assessment activities that generate
findings related to program impacts and sometimes view pathway
mapping as an unnecessary part of the evaluation. They do not
realize that pathway mapping is a planning and evaluation
tool/process that allows for outcomes to be collected in a more
focused and productive way later on in the evaluation. In the
past, when OMG has described their proposed evaluation
activities to prospective clients (pathway mapping followed by a
combination of surveys and focus groups or interviews), some
clients have responded that they would like to skip the pathway
mapping and go straight to the data collection activities.
When and if this happens, it is important to let clients
know that the pathway mapping produces a framework which is
critical to a systematic evaluation of program outcomes. In
addition, we have found that it is very useful to offer clients
good descriptions of the various uses of pathway maps and the
valuable reflections that are generated by the pathway mapping
process.
26
Another argument for including pathway mapping in the
evaluation design is that, unless the evaluator is already
familiar with the program, time must be devoted up-front to
informing the assessment team about the program.
Since this
level of effort is required, it makes sense to fully invest in
the pathway mapping process and have the evaluator document what
s/he is learning about the program in a way that will serve both
the evaluation and the program.
In other words, the process of
pathway mapping needs to happen to a greater or lesser extent
for an evaluator to become familiar enough with a program to
conduct an evaluation.
If the process is already happening, it
makes sense to have it formalized and documented so that the
program reaps the benefits as well.
Concluding Thoughts: Challenges to Theory of Change Evaluation
Throughout this chapter we have endeavored to discuss the
usefulness of a theory of change evaluation approach and have
argued that this type of approach would be a good fit for
evaluating leadership programs.
But, one may ask, when would
the use of a theory of change approach not represent a good fit
for a leadership program?
We think that there are two program
contexts where the theory of change evaluation would not be a
good fit.
The first one would be a stable program that
undergoes periodic evaluation and that has already identified,
27
with good clarity, its objectives, interventions, and intended
outcomes.
This type of program shows internal cohesion, its
anticipated outcomes appear reasonable, and measures have
already being identified and used to assess its anticipated
outcomes.
A theory of change evaluation, then, would not add
much value and a more traditional approach would be completely
suitable.
A second type of program context not amenable to a
theory of change evaluation approach would involve an
exploratory, pilot, program where program designers are not
ready to specify clear pathways of change and need to better
understand the effects of specific program components and/or
their interactions with different types of participants.
In
this case, a more open and exploratory approach for collecting
and interpreting data would be more suitable.
A common challenge to theory of change evaluation is how to
reconcile multiple theories of change representing the views of
distinct stakeholders.
Although differences among stakeholders
(staff, participants, alumni, Board) of leadership programs
focusing on individuals come up, it is generally possible to
reach consensus as the program represents a narrow slice of
experience.
When evaluating that type of leadership programs,
we push for consensus on the theory of change whenever possible.
If not possible, we will work with multiple theories of change,
although this has been rare.
This situation becomes more
28
complicated when the leadership program is set up as a placebased program or initiative (for instance, a neighborhood-based
program or a city-wide initiative) and the program directly
targets, both, individual and community change.
This type of
program or initiative represents a much broader slice of
experience.
In this situation, program participants who are
community members may hold quite different views than the
program’s funder about the nature and sequence of change in
their communities.
While consensus should also be sought in
this situation, it would be realistic to expect to have multiple
theories of change representing the perspectives of distinct
groups of stakeholders.
The evaluation should allow for these
differences and attempt to test the different theories if
possible.
The theory of change evaluation approach has been often
criticized for representing a linear progression of change in a
world where change is not linear.
This is a complex
epistemological issue that we do not intend to solve in this
chapter.
However, we would argue that a good theory of change
evaluation needs to be open to change in whatever form or
sequence occurs, whether anticipated or not, and would urge
evaluators using this approach to remain flexible during the
evaluation and not to grow overly confident on the accuracy of a
program’s articulated theory of change or the details of a
29
pathway map or logic model.
After all, a theory is just a
theory.
30
References
Connell, J.P., Kubisch, A.C., Schorr, L.B., and Weiss, C.H.
New
Approaches to Evaluating Community Initiatives: Concepts,
Methods, and Contexts. Vol. 1. Washington, DC: The Aspen
Institute, 1995.
Eisenhower Fellowships.[http://www.eisenhowerfelloships.org]
Fulbright-Anderson, K., Kubisch, A.C., and Connell, J.P. New
Approaches to Evaluating Community Initiatives: Theory,
Measurement, and Analysis. Vol. 2. Washington, DC: The Aspen
Institute, 1998.
Gutiérrez, M. and Stowell, B. Next Generation Leadership
Program: Final Assessment Report. Philadelphia, PA: OMG Center
for Collaborative Learning, 2004.
Gutiérrez, M. and Tasse, T.
Evaluation Report.
Eisenhower Fellowships: Final
Philadelphia, PA: OMG Center for
Collaborative Learning, 2005.
Leadership for a Changing World. [http://www.leadershipfor
change.org]
Next Generation Leaders. [http://www.nglnet.org]
W. K. Kellogg Foundation. Logic Model Development Guide.
Battle
Creek, MI: W.K. Kellogg Foundation, 2003.
Weiss, C.H.
“Nothing as practical as good theory: Exploring
Theory-Based Evaluation for Comprehensive Community Initiatives
31
for Children and Families.”
Schorr, L.B., and Weiss, C.H.
In Connell, J.P., Kubisch, A.C.,
New Approaches to Evaluating
Community Initiatives: Concepts, Methods, and Contexts. Vol. 1.
Washington, DC: The Aspen Institute, 1995.
32