Unit 1:
Getting ready
University of Wisconsin - Extension, Cooperative Extension, Program Development and Evaluation
University of Wisconsin - Extension, Cooperative Extension, Program Development and Evaluation
What comes to mind when you hear the
word evaluation?
University of Wisconsin - Extension, Cooperative Extension, Program Development and Evaluation
University of Wisconsin - Extension, Cooperative Extension, Program Development and Evaluation
What does evaluation mean to you?
On a piece of paper, write down your
definition of evaluation.
University of Wisconsin - Extension, Cooperative Extension, Program Development and Evaluation
University of Wisconsin - Extension, Cooperative Extension, Program Development and Evaluation
Definition of evaluation
Definitions of evaluation may vary by approach and
field of practice.
We use:
Program evaluation is… “the systematic
collection of information about the activities,
characteristics, and results of programs to
make judgments about the program, improve or
further develop program effectiveness, inform
decisions about future program development,
and/or increase understanding.”
– Patton, 2008
Source: Patton, M. Q. (2008). Utilization-focused evaluation (4th ed.). Thousand Oaks, CA: Sage Publications, Inc.
University of Wisconsin - Extension, Cooperative Extension, Program Development and Evaluation
University of Wisconsin - Extension, Cooperative Extension, Program Development and Evaluation
In simplest terms, evaluation answers
3 questions:
What?
So what?
Now what?
University of Wisconsin - Extension, Cooperative Extension, Program Development and Evaluation
University of Wisconsin - Extension, Cooperative Extension, Program Development and Evaluation
An outcome is…
the change that occurs as a result of our
activities and investments.
University of Wisconsin - Extension, Cooperative Extension, Program Development and Evaluation
University of Wisconsin - Extension, Cooperative Extension, Program Development and Evaluation
A new way of thinking…
FROM:
• What are you doing?
• What activities do you
deliver?
• Who participates? How
many?
TO:
• So what?
• What difference are
you making?
• What results were
achieved?
• Who benefited? How?
ACTIVITIES
OUTCOMES
University of Wisconsin - Extension, Cooperative Extension, Program Development and Evaluation
University of Wisconsin - Extension, Cooperative Extension, Program Development and Evaluation
A metaphor for outcomes:
Not how many worms
the bird feeds its young,
but
how well the fledgling flies
–United Way, 1999
Source: United Way of America. (n.d.). Outcome measurement: What and Why?: An overview. Retrieved July 31, 2008,
from http://liveunited.org/Outcomes/Resources/What/upload/TPsOMWhatandWhy.pdf
University of Wisconsin - Extension, Cooperative Extension, Program Development and Evaluation
University of Wisconsin - Extension, Cooperative Extension, Program Development and Evaluation
Create your own “outcome metaphor”
Create your own metaphor or drawing that
will help you remember the meaning of
outcome.
Metaphor: a word or phrase denoting one kind of
object or idea in place of another
University of Wisconsin - Extension, Cooperative Extension, Program Development and Evaluation
University of Wisconsin - Extension, Cooperative Extension, Program Development and Evaluation
What does the term program mean?
Program refers to the object of evaluation:
direct service interventions, community
mobilization efforts, research initiatives,
surveillance systems, policy development
activities, outbreak investigations, laboratory
diagnostics, communication campaigns,
infrastructure-building projects, training and
educational services, administrative systems.
–CDC, 1999
Source: Centers for Disease Control and Prevention. (1999). Framework for program evaluation in public health [Electronic
version]. Morbidity and Mortality Weekly Report, Vol. 48, No. RR-11, (September 17). Retrieved July 31, 2008, from
http://www.cdc.gov/eval/framework.htm
University of Wisconsin - Extension, Cooperative Extension, Program Development and Evaluation
University of Wisconsin - Extension, Cooperative Extension, Program Development and Evaluation
Why am I evaluating outcomes?
“Don’t accept your
dog’s admiration as
conclusive proof that
you are wonderful.”
–Ann Landers
University of Wisconsin - Extension, Cooperative Extension, Program Development and Evaluation
University of Wisconsin - Extension, Cooperative Extension, Program Development and Evaluation
What’s in this for you?
Benefits of evaluating outcomes:
• Program/organizational improvement
• Improved accountability
• Better ability to describe what a program does
– program description for stakeholders
• Data for fund raising
• Data for public relations
• Input for policy decision making
• Better planning
• Satisfaction in knowing how you are doing
University of Wisconsin - Extension, Cooperative Extension, Program Development and Evaluation
University of Wisconsin - Extension, Cooperative Extension, Program Development and Evaluation
Assessing organizational readiness for
evaluating outcomes
Does your organization have a clearly defined
and commonly understood/supported
• Mission – what your organization does
• Vision – what your organization should look like
in the future
• Values/beliefs/principles – what you value/
believe – manifested in how you behave
University of Wisconsin - Extension, Cooperative Extension, Program Development and Evaluation
University of Wisconsin - Extension, Cooperative Extension, Program Development and Evaluation
Assessing organizational readiness for
evaluating outcomes
• Are your identified priorities reflected in
your goals?
• Are the leaders committed to evaluation?
• Is there a general commitment to
evaluation throughout the organization?
University of Wisconsin - Extension, Cooperative Extension, Program Development and Evaluation
University of Wisconsin - Extension, Cooperative Extension, Program Development and Evaluation
Assessing your organization’s
readiness…
Does your organization have the resources
needed to do outcome evaluation?
Human
− Leadership – committed and involved
− Board and staff – understand the purpose
− Time – of leadership, staff, and participants
Fiscal
− Often entails some financial commitment
University of Wisconsin - Extension, Cooperative Extension, Program Development and Evaluation
University of Wisconsin - Extension, Cooperative Extension, Program Development and Evaluation
Assessing your organization’s
readiness…
• Is there a plan and a timeline?
• Is there a common framework/language
for evaluation?
• Do funders/stakeholders expect reporting
of outcomes?
University of Wisconsin - Extension, Cooperative Extension, Program Development and Evaluation
University of Wisconsin - Extension, Cooperative Extension, Program Development and Evaluation
Assessing your organization’s
readiness…
Is your organization going to be reporting to a funder?
If so, the funder may have a specific format or
expectation for reporting outcomes.
• Design outcome evaluations to meet the format or
expectation.
• Remember, mission/vision/values are core to what you do –
don’t compromise by “chasing” dollars that don’t “fit” your
organization.
University of Wisconsin - Extension, Cooperative Extension, Program Development and Evaluation
University of Wisconsin - Extension, Cooperative Extension, Program Development and Evaluation
Assessing your organization’s
readiness…
• Are staff members interested in outcome
evaluation?
• Do they have skills to engage in outcome
evaluation?
• Does the organization support
professional development for staff?
University of Wisconsin - Extension, Cooperative Extension, Program Development and Evaluation
University of Wisconsin - Extension, Cooperative Extension, Program Development and Evaluation
Assessing your organization’s
readiness…
• Is evaluation rewarded in your organization?
• Are evaluation processes, data, and
findings valued?
• Are evaluation data and findings used?
• Do staff know/see how their evaluations are
used?
University of Wisconsin - Extension, Cooperative Extension, Program Development and Evaluation
University of Wisconsin - Extension, Cooperative Extension, Program Development and Evaluation
Assessing your organization’s
readiness…
• Is evaluation data used by the
organization to improve programs, guide
resource allocations, and support
planning?
• Is evaluation data used outside the
organization to enhance its image,
increase funding, and share promising
practices?
University of Wisconsin - Extension, Cooperative Extension, Program Development and Evaluation
University of Wisconsin - Extension, Cooperative Extension, Program Development and Evaluation
Barriers to evaluating outcomes
• Lack of leadership commitment
• Limited time and resources, including
computer technologies
• Limited knowledge and skills in evaluation
• Attitude that programs cannot be
measured
• Fear that evaluation data will hurt
programs
• Lack of use or misuse of evaluation data
University of Wisconsin - Extension, Cooperative Extension, Program Development and Evaluation
University of Wisconsin - Extension, Cooperative Extension, Program Development and Evaluation
Assets in evaluating outcomes
• Committed leadership
• Resources allocated for outcome
measurement
• Evaluation expertise exists or can be
secured
• Evaluations are used
• Incentives exist for doing quality
evaluation
University of Wisconsin - Extension, Cooperative Extension, Program Development and Evaluation
University of Wisconsin - Extension, Cooperative Extension, Program Development and Evaluation