Getting
started
with
evaluation
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
1
Test your evaluation IQ!
QUIZ #1
On the next slides, think about what evaluation is…
then, compare your ideas to a common definition
(slide #4).
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
2
What IS evaluation? (choose one answer)
a.
b.
c.
d.
e.
It is a survey to collect information from
participants
It is an end-of-session questionnaire
It is a success story to document our value
It is statistics and data analysis (that I don’t
like to do!)
A systematic process of collecting
information for a specific purpose and use.
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
3
While many think of evaluation as synonymous
with data collection (a survey or an end-ofsession questionnaire) or reporting a success
story, it actually is much broader than any one of
those things.
The first four answers on the quiz are part of what
make up evaluation…
but evaluation includes more than any one of
these things. The correct answer is E.
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
4
Evaluation is…
the systematic collection of information about the
activities, characteristics, and results of programs
to make judgments about the program,
improve or further develop program effectiveness,
inform decisions about future programming, and/or
increase understanding.
(Patton, 2008:39)
This definition of evaluation comes from Michael Patton in his
book, Utilization-Focused Evaluation. Notice the phrases in
color as key concepts about what evaluation is and why we do it.
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
5
In other words,
“Don’t accept your
dog’s admiration
as conclusive proof
that you are wonderful.”
 Ann Landers
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
6
Remember, evaluation actually is part of our every day lives…
In fact, we engage in “informal” evaluation all the time. We try a
new restaurant and decide whether to go back again. We examine
the meat in the meat case at the grocery store and decide which
cut, if any, to buy. We watch our program participants to see if they
appear to be engaged and learning. We ask our volunteers how
things are going.
However, sometimes we need to be more deliberate and
systematic. To have information that is seen as credible and
trustworthy by others, we engage in what we characterize as
“formal” evaluation. We commit time and resources in a deliberate
process of systematic data collection to have credible information
for making sound decisions.
See the Program Development and Evaluation booklet, Developing a concept
of Extension Program Evaluation; link provided on web site
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
7
Informal
Formal
• Casual
• Deliberate, thoughtful
• Spontaneous
• Takes time
• Information is quickly
collected
• Information is
systematically
collected
• Sometimes
insufficient
• Credibility of
information may be
questioned
• Criteria are explicit
• More credible
information
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
8
Images of Evaluation
• Judging
• Learning
Evaluation is often thought of in terms of judgment . This can
provoke fear and anxiety.
However, evaluation also helps us learn. This might be the
most important and useful way to think about evaluation.
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
9
What is your image of evaluation?
Do you see it as…
1. judgment?
2. hard and difficult?
3. someone else’s responsibility,
not yours?
4. something to fear and loathe?
5. a waste of time?
6. Or, do you see the potential of evaluation to
foster and enhance learning?
Spend a few minutes reflecting on what your image of
evaluation is.
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
10
Quality evaluation
Like most professions, evaluation has a set of
professional standards that guide ethical and proper
practice. These four Evaluation Standards are:
1. Utility: How useful is the evaluation to you and your
audiences? i.e., Will it serve the information needs of the
intended users?
2. Feasibility: How practical is your evaluation? i.e., Will it not
disrupt program operations; will it be worth the costs?
3. Propriety: How appropriate is your evaluation for those
involved? i.e., Does it respect people, attend to human
subjects protection, is it ethical and legal ?
4. Accuracy: How accurate is your evaluation and the
information it conveys? i.e., do the methods and procedures
revel technically adequate information?
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
11
Use the evaluation
standards as guidelines to
help ensure high quality
evaluation.
Quick Tips 9 – Ways to
Improve the Quality of
Your Program Evaluations
will help.
Links to the Quick Tips are
found on this web page:
http://www.uwex.edu/ces/pdande/resources/
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
12
Get clear on language
Impact
Outcomes
Process
Accountability
Outputs
Indicators
Measurement
Data
Inputs
There are a lot of technical terms and jargon in evaluation. Be sure
when you are working with others that you mean the same thing
with the words you use. We often talk past each other. See the
Glossary (pdf listed on web site) for a list of definitions of common
evaluation terms.
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
13
WIIFM - What’s in it for me?
By doing quality evaluation,
•
Your bragging rights are legitimized.
•
You can feel sure that you are making a
difference and using public dollars well.
•
You will have credible information to share with
my committees and participants.
•
Your outcomes will be stronger because
measuring results produces results!
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
14
Continued…
By doing quality evaluation,
•
You will know what needs to be done to
make my programming stronger.
•
You can demonstrate your scholarship.
•
You will increase understanding of your
programming purpose and direction.
•
You will achieve your promotion goals.
What else would you add to this list?
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
15
Make Evaluation Meaningful
• Make evaluation a part of programming – not
an add-on or something that is required.
• Build evaluation in upfront in the planning
process
• Use your resources wisely – don’t expect to
formally evaluate everything.
• Use evaluation to “learn”.
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
16
And, finally…
Practice – practice – practice
We learn evaluation
by doing it!
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
17
Reflection time
What is one thing
you learned (or had reinforced)
from going through this
presentation that you
hope not to forget?
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
18