Using
observation
to collect
evaluation
data
See the PDE booklet,
Collecting evaluation data: Direct observations
http://learningstore.uwex.edu/pdf/G3658-05.pdf
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
1
Checking in…
What do you think?
Answer YES or NO to each of the following
1. Observation involves “seeing” and “listening”
2. People may behave differently when they know
they are being observed so it is better not to tell them
3. Structured observations provide more accurate and
useful information
4. As long as you see it, it doesn’t matter if you record
what you saw
5. You, as someone who “knows” the program and the
participants, are best suited to conduct the observations
6. The same principles of sampling apply to observation as
to other forms of data collection
Check your answers at the end
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
2
Observation…
Involves all 5 senses:
sight, hearing, smell, touch, and taste
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
3
Purpose and benefits of observation
as a data collection method
• It is unobtrusive
• You can see things in their natural context
• You can see things that may escape conscious
awareness, things that are not seen by others
• You can discover things no else has ever really paid
attention to, things that are taken for granted
• You can learn about things people may be unwilling to
talk about
• It is inconspicuous – least potential for generating
observer effects
• It is the least intrusive of all methods
• You can be totally creative – flexibility to yield insight
into new realities or new ways of looking at old
realities
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
4
Observation is useful when…
• You want direct information
• You are trying to understand an ongoing
behavior, process, unfolding situation, or
event
• There is physical evidence, products, or
outcomes that can be readily seen
• Written or other data collection methods
seem inappropriate
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
5
Observations
Advantages
– Most direct
measure of
behavior
– Provides direct
information
– Easy to complete,
saves time
– Can be used in
natural or
experimental
settings
Disadvantages
– May require training
– Observer’s presence
may create artificial
situation
– Potential for bias
– Potential to overlook
meaningful aspects
– Potential for
misinterpretation
– Difficult to analyze
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
6
Is observation culturally appropriate?
Things to consider:
• Discomfort, threat of being observed
• Issue of being an “outsider”
• Observer effect
• Possibilities for
misinterpretations
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
7
Observation – Ethical issues
• Unobtrusiveness is its greatest strength;
also potential for abuse in invasion of
privacy
• You can venture into places and gather
data almost anywhere so be ethical
• Remember our Human Subjects Protection
guidelines
– Consent form for participating in an
observational study
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
8
Types of observation
Structured
Unstructured
Looking for
Looking at
Sometimes we have
something specific we
want to observe –
leadership skills; level of
participation; etc.
We use a structured,
preset guide of what to
observe or a checklist.
Sometimes we want to
see what is naturally
occurring or exists
without predetermined
ideas. We use have an
open-ended approach to
observation and record
all that we observe
Observing what does not happen may be as
important as observing what does happen.
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
9
Practice:
Structured/unstructured observations
Imagine you are sitting in a room where ten
youth are sitting at computers learning about
Web 2.0 applications.
1) If you want to assess to what extent
students are interested and learning, what
specifically would you look (listen) for?
2) If you aren’t sure what specifically indicates
student interest or learning and you want to
see what is going on during the
demonstration, how would you proceed?
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
10
Steps in planning for observation
• Determine who/what will be observed.
• Determine aspects that will be observed
(characteristics, attributes, behaviors, etc.).
• Determine where and when observations
will be made.
• Develop the observation guide
• Pilot test the observation guide
• Train the observers and have them practice.
• Conduct the observations
• Analyze and interpret the collected information.
• Write up and use your findings.
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
11
Who/what can you observe
• People (individuals, groups,
communities)
– Characteristics
– Interactions
– Behaviors
– Reactions
• Physical settings
• Environmental features
• Products/physical artifacts
Use sampling strategies as you do for other methods of data collection
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
12
Example – Observing participation in an
after school program
• Who you will observe:
youth attending the program
• What you will observe:
– Age, gender
– Length of time student stays in the program
– Involvement in activities: which activities
• Level of involvement
– Interactions with other youth; with staff
• When you will observe: all hours the program
is open for one week each month during 2007
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
13
Recording your observations
It is not good enough to just observe,
you need to record your observations.
You might use:
– Observation guide
– Recording sheet
– Checklist
– Field note
– Picture
– Combination of the above
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
14
Sample Observation Guides
Guide for structured observations
Guide for unstructured observations
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
15
Structured observation guide used for pre and post program evaluation
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
16
Who does the observations?
•
•
•
•
•
•
•
You – program staff
Participants - Youth
Parents
Teachers
Volunteers
Other stakeholders
Colleagues
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
17
Training –
preparation/orientation may be necessary
–
–
–
–
To learn what to look for
To learn how to record observations
To practice
To ensure that observations across sites are
consistent: observers use the same
methods, rate an observation in same way
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
18
How well do you observe?
Take 15 seconds and look at the picture below. Move to the next slide or
turn away from the computer screen and write down everything you
observed. Then, come back to the photo and see what you missed (or
thought was there and isn’t!).
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
19
Practice
your observation skills
everyday in everyway!
And,
add observation
to your data collection toolbox.
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
20
Checking back in…Answers
1. Observation involves “seeing” and “listening” YES
2. People may behave differently when they know they are being observed so it
is better not to tell them NO – they often DO behave differently but that is
not a reason not to tell them they are being observed.
3. Structured observations provide more accurate and useful information NO
– unstructured observation also can be very useful
4. As long as you see it, it doesn’t matter if you record what you saw NO – you
must record your observations to have evidence of it existing
5. You, as someone who “knows” the program and the participants, are best
suited to conduct the observations NO – various people might be well suited,
though training may be necessary
6. The same principles of sampling apply to observation as to other forms of
data collection YES
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
21