Planning
your
evaluation
This presentation provides an overview of the steps to
planning a quality evaluation. Each step is covered in
greater detail in other sections on the web site.
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
1
An evaluation plan is your roadmap
How do you know which way to go
if you don’t know where you are going?
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
2
Use the booklet,
Planning a Program
Evaluation and the
Worksheet as
background for this set
of slides and to help
with your evaluation
plan.
Links to these
resources are found on
the web page:
http://www.uwex.edu/ces/pdande/evaluation
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
3
There are
5 core
steps
in
planning a
program
evaluation
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
4
These are best practices steps!
Consider these steps regardless if you are
evaluating a single workshop or a
comprehensive program.
However, the level of detail in your evaluation
plan will depend upon the scope and nature of
your evaluation.
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
5
STEP 1
Begin by deciding who should
be involved in helping design and
implement the evaluation.
Or, do you do it alone?
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
6
Think about these 4-H YD stakeholders
as people who might be involved in your
evaluation ( your evaluation stakeholders)
•
•
•
•
•
•
Youth participants
Your program partners
Parents
Teachers
Volunteers
Your funder
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
7
How might they be involved?
What roles might they play?
They might
• Help determine what the evaluation will
focus on –
what you really need to know
• Be part of an evaluation team
• Help write the questions
• Collect data
• Help enter data and/or do the analyses
• Raise funds
• Write a press releases
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
8
Why would you want to do this?
• To build ownership of the evaluation and your
work
• To develop skills in others, e.g., youth who
learn how to conduct a survey or analyze data
• To ensure that the evaluation findings will be
useful and used
• To bring talent and expertise to the evaluation
• To share the work!
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
9
STEP 2
Next, we focus the evaluation.
What program or part of a program are you
going to evaluate?
Use a logic model to describe your program so
you evaluate what is meaningful.
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
10
A logic model helps us clearly describe the
program we want to evaluate. It helps us focus
on what we want to collect information about.
Situation:
INPUTS
Program
investments
OUTPUTS
Activities
Assumptions
Participation
OUTCOMES
Short
Medium
Long-term
External factors
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
11
Be clear about your
purpose for evaluating
We are conducting an evaluation of
______________ (name of program)
because _________________________
in order to ________________________.
Example:
We are conducting an evaluation of the 3 series
Money Quest Program because we want to know to
what extent youth participants increase their
knowledge about money management and use the
recommended practices in order to report the value
of this program to our funder.
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
12
Determine Use and User
WHO?
WHAT
do you/they
want to know?
HOW
will they use the
info?
You – staff
Participants
Funder
Other
stakeholders ??
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
13
What do you want to know –
what data do you want to collect?
Go back to your logic model – the description of your
program and think about what you really want to know
about your program. What data do you want to collect?
Do you want information about:
• Outcomes: to what extent changes occur in
knowledge, skills, attitudes, opinions, behaviors,
practices, policies, social-economic-environmental
quality…
• Reactions: What the participants like/dislike?
Whether they will come again/stay involved/promote
your program to others, etc. ?
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
14
Do you want to know about …
• Participation
Who/how many attended? Who didn’t attend why? What happens for different participants?
• Activities
What was done and how well was it done (quality)
Did everything go according to plan? What
worked well/not so well? Which activities link to
different outcomes for different participants?
• Inputs
What was invested (inputs)? By whom? Who
collaborated? How? What resources were used?
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
15
Do you want to know about…
• Your teaching?
How you did? How you might improve?
• Future programming?
Future interests; needs?
• Costs and returns?
How much it costs to put the program on and
what the return on that investment is?
• Other questions???
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
16
Prioritize:
We can’t and don’t want to collect information
about everything!
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
17
How will you know it? INDICATORS
For each evaluation question, think about the
specific information you need to collect to answer
that question.
Example:
Question
Indicator
To what extent did the
program increase youthadult partnerships?
- #,% of Boards with youth participating
in meetings before and after
- #,% of Boards with youth on
committees before and after
- #,% of Boards with youth in
leadership positions before and after
- Reported change in quality of the
youth-adult interaction
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
18
What is your “evaluation design”
Evaluation design is your overall approach to
collecting data.
Typical evaluation designs include :
– Single point in time (e.g., survey, end-of-program
questionnaire)
– Pre-post program or retrospective post-then-pre
(comparison of before to after)
– Multiple points in time (e.g., pre, post and follow-up
– Comparison group designs (two or more groups)
– Case study design (indepth analysis of a single
case)
– Mixed method (e.g., the use of a survey, observation
and testimonials)
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
19
STEP 3
Now, think about from whom and what method
you will use to collect the information.
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
20
From where or from whom will you get the
information you need? (“Source”)
Do you think that you are most likely to get the
information you need from:
– Existing information – records, reports,
program documents, etc.
– People – participants, parents, volunteers,
etc.
– Pictorial records and observations – video
or photos, observations of events
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
21
What method(s) of data collection is most
appropriate ?
• Survey
(questionnaire)
• Case study
• Interview
• Observation
• Group assessment
• Expert or peer
reviews
• Portfolio reviews
• Testimonials
• Tests
• Photographs,
videotapes, slides
• Diaries, journals, logs
• Document review and
analysis
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
22
Which method do you choose?
As we know, there are many standard methods
of data collection, and more creative ones. Each
has its own strengths and weaknesses.
The ‘art and science’ of data collection is to
select the method appropriate for the purpose of
your evaluation, the audience you are collecting
information from and your resources.
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
23
Test your understanding:
Match the method in the left column to
the definitions on the right
• Case study
• Questionnaire
• Observation
• Focus group
• Interview
1. Collecting standardized
information from people in a
non-threatening way
2. Talking with and listening to
people either face-to-face or by
telephone.
3. Gathering information by
viewing what is occurring
4. Exploring a topic in depth
through group discussion
5. Gaining an in-depth
understanding of someone’s
experience in the program
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
24
STEP 4
While the actual analysis comes once you have
your data, you want to think about analysis when
you are planning the evaluation. That way you will
be sure to collect the type of information to be able
to report as you want.
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
25
What do you want to be able to report?
Will you want to report
Type of analysis
Numbers, percents
Count; percentage
Average number or score
Range of scores
Mean
Range
Changes from before to after
Change score
Comparisons of one group to
another
Cross tab
People’s stories
Qualitative content analysis
Comments
Qualitative content analysis
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
26
Step 5
And, last but not least, is USING and
COMMUNICATING your evaluation findings. Actually,
we should be communicating and using our
information throughout the evaluation…not just at the
end.
Think about the many ways you might use and share
information both during and after your evaluation.
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
27
How will you report, use, and learn from
the findings?
WHO - To whom will you report?
WHAT - What will you report/share?
HOW - How will you share the information?
WHEN - When will you communicate?
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
28
Managing the evaluation
We might think about a management
plan to make sure everything gets done!
– Human subjects protection
Check the decision tree and follow best practice
guidelines. Consult with the Human Subjects
Administrator if you have questions:
http://www.uwex.edu/hsp/
– Timeline
– Responsibilities: who will do what?
– Budget: money needed to do the
evaluation
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
29
Check your evaluation plan against the
Evaluation Standards to ensure a ‘quality’
evaluation
• Utility: Will it be useful and used?
• Feasibility: Is it practical and can be
accomplished given your resources?
• Propriety: Is it respectful and ethical?
• Accuracy: Is it likely to produce accurate
information?
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
30
Remember…
“The more attention you give to
planning the evaluation,
the more effective it will be.”
−The Program Manager’s Guide to Evaluation, 2003
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
31
Reflection time
What is one thing
that you learned from
this presentation that
you didn’t know before?
Good luck
with planning your next evaluation!
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
32