This is STEP 3 of the five steps of evaluation
(see “Planning an Evaluation”).
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
1

Answer the questions on the next slide… and then find the answers on the following slides.
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
2

a.
There is one best way to collect data b.
Quantitative methods that collect numbers provide more useful information c.
Evaluation data collection involves any established social science research method d.
We often collect data from program participants e.
We should always collect data from as many participants as possible
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
3

a. FALSE.
Unfortunately, there is NOT one best way to collect data. It all depends…upon what you want to know, who the respondents are, your resources and what is culturally appropriate among other things.
Think about your choices and weigh the pros and cons of different ways to collect data b. FALSE. Sometimes numbers are more useful; other times, narrative (qualitative data) are more useful.
Often times, a mix of quantitative and qualitative data provides the most useful information c. TRUE : any and all social science research methods can be used for collecting evaluation data
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
4

d. TRUE: Yes, we often do collect data from program participants BUT, that doesn’t mean they are the ONLY ones or the best ones. There may be others (parents, volunteers, friends) who might be in a better position to provide the information we need. Or, we might be able to use existing documents, record books, journals, photographs as sources of information.
e. FALSE . Sometimes using a sample provides better data.
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
5

Collecting data involves:
1) Sources where you will get the information; and
2) Methods - how you will collect /gather the information
Use this booklet for help http://learningstore.uwex.edu/pdf/G3658-04.pdf
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
6

From where or from whom will you get the information?
• Existing information – records, reports, program documents, logs, journals
• People – participants, parents, volunteers, teachers
• Pictorial records and observations – video or photos, observations of events, artwork
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
7

Common methods include:
• Survey
• Case Study
• Interview
• Observation
• Group assessment
• Expert or peer reviews
• Portfolio reviews
• Testimonials
• Tests
• Photograph, videotape, slides
• Diaries, journals, logs
• Document review and analysis
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
8

Quantitative methods Qualitative methods
Surveys
Questionnaires
Tests
Focus groups
Unstructured interviews
Existing databases Unstructured observations
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
9

"Not everything that counts can be counted."
5 (Quantity) Happy (Quality) Kids
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
10

• Quantitative data collection methods produce numbers.
• Qualitative data collection methods produce words.
• Quantitative and qualitative each has its strengths and weaknesses.
• Quantitative methods are more structured and allow for aggregation and generalization.
• Qualitative methods are more open and provide for depth and richness.
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
11

There is no simple answer
There is no ONE best method
It all depends…
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
12

• The purpose of your evaluation − Will the method allow you to gather information that can be analyzed and presented in a way that will be credible and useful to you and others?
• The respondents − What is the most appropriate method, considering how the respondents can best be reached, how they might best respond, literacy, cultural considerations, etc.?
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
13

• What kind of data your stakeholders will find most credible and useful
• Resources available. Time, money, and staff to design, implement, and analyze the information. What can you afford?
• Type of information you need. Numbers, percents, comparisons, stories, examples, etc.
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
14

• Interruptions to program or participants. Which method is likely to be least intrusive?
• Advantages and disadvantages of each method.
• .The importance of ensuring cultural appropriateness
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
15

Why would this be so?
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
16

When we use several methods we say we are
‘triangulating ’. Triangulation is important in evaluation because we want accurate and trustworthy information.
Triangulation means the use of multiple sources and methods to gain a better understanding.
Each source and each method has inherent biases so using more than one source and/or method provides a more accurate picture.
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
17

How might you mix sources of information in your evaluation?
How might you mix data collection methods to evaluate your program?
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
18

Mix sources of information
For example, you might collect information from program participants AND parents; or from campers and camp leaders.
Mix data collection methods
For example, you might survey participants AND interview a sample of participants. You might conduct focus group interviews with community service participants AND observe the community service projects.
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
19

Thinking back on the examples in the previous slide, what different type of information might you get from the different sources and methods?
Using multiple sources and/or methods means more time and resources.
The choice of data collection method ultimately depends upon the resources you have available .
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
20

What will you use to actually collect the data… a recording sheet, a questionnaire, a video or audio tape?
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
21

The term, “ instrument ”, sounds like we are talking about a dental office, a cockpit or an orchestra.
Actually, we use the term “instrument” to mean the tool on which the data is actually recorded : the questionnaire, the recording form, the video or audio tape, for example.
If you have selected a survey as your method, you automatically know that you will need a questionnaire. But, if you choose a method such as focus group or interview or observation, think about what you will use for recording the information.
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
22

When Will Data be Collected?
– Before and after the program
– At one time
– At various times during the course of the program
– Continuously through the program
– Over time - longitudinally
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
23

Will you collect information from everyone or use a sample?
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
24

A sample is a portion or a subgroup of a larger group called a population.
Sometimes we want to include everyone; other times it is better to collect information from a sample.
See the booklet, Sampling http://learningstore.uwex.edu/pdf/G3658-03.pdf
Also, see Bill Trochim’s Social Science Research Methods at http://www.socialresearchmethods.net/kb/sampling.php
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
25

Probability:
• Why?
Generalize to population.
Some examples:
– Simple random sample
– Stratified sample
– Cluster sample
– Systematic sample
Nonprobability:
• Why? Generalizability not as important. Want to focus on “right cases.”
– Quota sample
– “Purposeful” sample
– “Convenience” or
“opportunity” sample
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
26

True or False?
a.
There is one best way to collect data b.
Quantitative methods that collect numbers provide more useful information c.
Evaluation data collection involves any established social science research method d.
We often collect data from program participants e.
We should always collect data from as many participants as possible
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
27

What is one thing you learned (or had reinforced) from going through this presentation that you hope not to forget?
Good luck with your data collection efforts!
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
28