ASC 607: PROGRAM EVALUATION
Over the years I have become convinced that we
learn best—and change—from hearing stories that
strike a chord within us…Those in leadership
positions who fail to grasp or use the power of
stories risk failure for their schools and for
themselves.
–John Kotter
HOW PLANNING AND
EVALUATION INTERACT
Strategic Planning
Goals
Objectives
Evaluation
Inputs
Activities
Time Frame,
Budget,
Responsibility
Outputs
Outcomes
Short
Long
Mid
THE ART AND SCIENCE OF EVALUATION
Learning Intentions
To make a personal and emotional
connection to change or impact.
To recognize the steps and purpose
for conducting program
evaluations.
Associations
Certainty
Curiosity
Fear
Indifference
Confusion
Excitement
Hope
Panic
Associations
Certainty
Curiosity
Fear
Indifference
Confusion
Excitement
Hope
Panic
Associations
Certainty
Curiosity
Fear
Indifference
Confusion
Excitement
Hope
Panic
“Evaluation is a systematic and intentional process of
gathering and analyzing data to inform learning,
decision-making, and action.”
PROGRAM EVALUATION DEFINED
WHY EVALUATE
To determine whether a program is
working efficiently in term of the resources
used to perform the program’s work.
To determine whether the program
operates as planned.
To determine whether programs are
effective in achieving their intended goals.
EVALUATION MISCONCEPTIONS
Program evaluation is something we already do.
PROGRAM EVALUATION AND ACCOUNTABILITY
Program evaluation only tells me if my program
worked or not.
Effective evaluation is not an ‘event’
that occurs at the end of a project,
but is an ongoing process that helps
decision makers better understand
their programs.
Evaluation is forward-looking and
directly tied to upcoming
decisions…it reports back in real
time and allows for midcourse
corrections.
It may not be possible to know
definitively whether programs yield
impact, but leaders must do their
best to develop as much
information as possible to help
them understand whether
programs achieve their objectives.
Program evaluation is too costly, complicated,
and best left to experts.
“The key factor is for teachers to have mind
frames in which they seek feedback about their
influences on students and thus change,
enhance, or continue their teaching methods.
This process begins with the question: ‘How do I
know that this is working.’”
-John Hattie
THE BENEFITS OF EVALUATION
BENEFITS
How would you articulate the benefits of program evaluation to
your school community based on reading Kim’s story? What did
Kim learn that you could share as a rationale for doing program
evaluation?
EVALUATION FOCUS
Learning Intentions
To understand the components of a
program logic model and how to
plan using one.
To know what the different levels of
program outcomes are and how
they differ from program outputs.
To experience building different
elements of a program logic
model given several situations
including a case study.
To know how to facilitate a process
for creating a logic model for a
local program.
What is a logic model?
SIMPLE LOGIC MODELS
Inputs
Outputs
Outcomes
LOGIC MODEL TERMINOLOGY
Why use logic models?
Helps in planning and evaluation
Identifies gaps in program logic
Promotes consensus about how program works
Clarifies what is appropriate to evaluate
Summarizes complex programs for stakeholders
EXPANDED LOGIC MODELS
What is an outcome?
“Outcomes must relate to the benefits program
actions, services, and products might have for
the participants, not simply their receipt.”
Rossi, Lipsey, & Freeman
Outputs vs. Outcomes
Outputs vs. Outcomes
Outputs ask “What does
the program do?” And
“Who does the program
reach?”
Outputs typically refer to
counts of things like the
number of people
trained.
Outcomes refer to “What
difference does the
program activity make?”
Outcomes relate to
changes that occur in
participants’ knowledge
and behaviors and
larger-scale changes.
Outputs vs. Outcomes
Outputs ask “What does
the program do?” And
“Who does the program
reach?”
Outputs typically refer to
counts of things like the
number of people
trained.
Outcomes refer to “What
difference does the
program activity make?”
Outcomes relate to
changes that occur in
participants’ knowledge
and behaviors and
larger-scale changes.
What is NOT an outcome?
“The Workforce Development Agency develops a
flexible, innovative, and effective workforce by
assisting the structurally unemployed with financial
independence, advocating for the integration of
workforce development into the K-12 school
system, and supporting the alignment of workforce
development with economic development efforts.”
“The mission of the San Diego County Office of
Education, as a world-class educational leader and
trusted partner, is to transform public education and
guarantee high levels of student achievement. In
partnership with local school districts and the global
learning community, we will research and apply
innovative 21st century practices; leverage resources;
develop strategic alliances; inspire powerful leadership;
and provide exemplary customized services to districts,
communities and the students in all County Officeoperated programs.”
Logic Model Check-in
Rich Uncle
Three compelling reasons why I
need the car (and the difference it
will make in my life)…
Before I get my car, I will need...
I will typically use my car to...
The evidence I will send my Uncle
every six months...
In three years, my car has made the
following changes in my life...
Inputs
Resources:
What we invest
Outputs
Actions:
What we do
Products/
Participants:
What we
produce and
who we reach
Outcomes
Short-term:
Learning
Intermediate:
Action
Long-term
Conditions
LOGIC MODEL
Inputs
Resources:
What we invest
Outputs
Actions:
What we do
Products/
Participants:
What we
produce and
who we reach
Outcomes
Short-term:
Learning
Intermediate:
Action
Long-term
Conditions
LOGIC MODEL
Inputs
Resources:
What we invest
Outputs
Actions:
What we do
Products/
Participants:
What we
produce and
who we reach
Outcomes
Short-term:
Learning
Intermediate:
Action
Long-term
Conditions
LOGIC MODEL
Inputs
Resources:
What we invest
Outputs
Actions:
What we do
Products/
Participants:
What we
produce and
who we reach
Outcomes
Short-term:
Learning
Intermediate:
Action
Long-term
Conditions
LOGIC MODEL
Inputs
Resources:
What we invest
Outputs
Actions:
What we do
Products/
Participants:
What we
produce and
who we reach
Outcomes
Short-term:
Learning
Intermediate:
Action
Long-term
Conditions
LOGIC MODEL
Inputs
Resources:
What we invest
Outputs
Actions:
What we do
Products/
Participants:
What we
produce and
who we reach
Outcomes
Short-term:
Learning
Intermediate:
Action
Long-term
Conditions
LOGIC MODEL
Inputs
Resources:
What we invest
Outputs
Actions:
What we do
Products/
Participants:
What we
produce and
who we reach
Outcomes
Short-term:
Learning
Intermediate:
Action
Long-term
Conditions
LOGIC MODEL
Inputs
Resources:
What we invest
Outputs
Actions:
What we do
Products/
Participants:
What we
produce and
who we reach
Outcomes
Short-term:
Learning
Intermediate:
Action
Long-term
Conditions
LOGIC MODEL
FULL LOGIC MODELS
The program situation
If I were given one hour to save the planet,
I would spend 59 minutes defining the
problem and one minute resolving it.”
–Albert Einstein
CASE STUDY
FIELDWORK
“The principle vehicle of leadership is the story: the
leader affects individual behavior, thought, and
feelings through the stories that he and she tells. All
successful leaders are successful to the extent that
they tell and embody persuasive stories about where
the institutions they lead should be going and how
they will get there.”
-Howard Gardner
Todd Langager  Shannon Coulter
tlangager@sdcoe.net
scoulter@sdcoe.net