Data Collection Toolkit I:
Available Data and Observation
Research Methods for Public
Administrators
Dr. Gail Johnson
Dr. G. Johnson,
www.researchdemystified.org
1
Steps in the Research Process
Planning
1. Determining Your Questions
2. Identifying Your Measures and Measurement
Strategy
3. Selecting a Research Design
4. Developing Your Data Collection Strategy
5. Identifying Your Analysis Strategy
6. Reviewing and Testing Your Plan
Dr. G. Johnson,
www.researchdemystified.org
2
Toolkit: Available Data
 Files/records
 Computer data bases
 Other reports
 Census data
 Documents (budgets, policies and
procedures, organizational chart, position
descriptions, staff, etc.)
Dr. G. Johnson,
www.researchdemystified.org
3
Data Collection:Agency Records
 Agencies may have already collected the
data: from clients, community, internal
information systems.
 Agencies may have already summarized
and reported information: internal
management reports, budget documents,
and reports to public or funding agencies.
Dr. G. Johnson,
www.researchdemystified.org
Existing Reports
 Routine and special reports may already exist.
 Fully
cite any reports used.
 Describe the methodology used for each report.
 Sometimes the data is old:
 Decide whether it is too old.
 Decide whether it can be used as a baseline.
 Replicate their data collection strategy using
the same exact measures
Dr. G. Johnson,
www.researchdemystified.org
Available Computer Data
 Obtain a copy of the data files
 Obtain a copy of the database structure (how the
data set is laid out).
 Obtain a copy of the data dictionary (how the data
is labeled and coded).
 Transfer data with a minimum of effort (no retyping)
Dr. G. Johnson,
www.researchdemystified.org
Available Computer Data
 But how good is the data?
 Garbage
in, garbage out applies.
 Find out procedures used to collect data.
 Find out procedures used to verify accuracy.
 Estimate the error rate.
Dr. G. Johnson,
www.researchdemystified.org
Available Data
Which of these are likely to be available?
 Graduation
rates for last year
 Budget expenditures for school construction
 Teacher commitment to teaching
 Student performance
 Student satisfaction with the learning
experience
Dr. G. Johnson,
www.researchdemystified.org
8
Available Data: Key Issues
 If others collected the data, you want to
know enough to determine whether the data
are:
 Valid
 Reliable
 Accurate
Dr. G. Johnson,
www.researchdemystified.org
9
Available Data: Key Questions
 How did they operationalize their
measures?
 How did they collect the data?
 How did they ensure accuracy in collecting
and transcribing data?
Dr. G. Johnson,
www.researchdemystified.org
10
Available Data
 Advantages:
 Often
cheaper and faster than collecting data
yourself.
 Challenges:
 Exact
data you need may not be available
 May have difficulty getting access.
 May have too many data errors
Dr. G. Johnson,
www.researchdemystified.org
Toolkit:
Data Collection Instrument (DCIs)
 Systematically collect data from files, documents,
or real life (observation in the field)
 A form that specifies exactly what data you want
to collect
 Structured, check-a-box form makes it easy to
collect but may not be possible:
 The degree of structure depends on situation
Dr. G. Johnson,
www.researchdemystified.org
12
Data Collection Instrument
 Pre-test the DCI and revise
 Establish final decision rules
 Train coders
 Test for inter-rater reliability
 This means that different coders actually code the same
material in exactly the same way
 This is the same as when coding qualitative data, such
as content analysis
 If too divergent, revise and test again for inter-rater
reliability
Dr. G. Johnson,
www.researchdemystified.org
13
DCI: Example
Describe background and performance of teachers:
 Go through files:

A. highest degree: __BA/BS ___MA ___PhD
 B. date hired:___________
 C. Number of classes taught last year:__________
 D. Performance ratings for past three years:



2009: __Excellent __ Good ___ Poor ___Unknown
2008: __Excellent __ Good ___ Poor ___Unknown
2007: __Excellent __ Good ___ Poor ___Unknown
Dr. G. Johnson,
www.researchdemystified.org
DCI: Example
 E.
Age: __ <25 ___ 26-35 ___ >36
 F. Gender: ___ Male ___ Female
 G. Current salary: _______________
Dr. G. Johnson,
www.researchdemystified.org
Discussion and Exercise
 You want to look at the qualifications of those
admitted to the MPA program
 How would you operationalize
“qualifications”?
 Where is the data likely to reside?
 At this early stage, what information might you
need in order to determine the most efficient
and effective data collection option?
 Begin a design matrix
Dr. G. Johnson,
www.researchdemystified.org
Toolkit: Observation
 Field Work
 Environment
 People: on the street, in a lab, in a classroom, at
work, etc.
 Situations and Process
 Structured observations use checklists
 Unstructured observations capture
impressions
Dr. G. Johnson,
www.researchdemystified.org
17
Observation Examples
 Observe classroom activities to measure the amount of





time spent on hands-on learning activities.
Observe amount of traffic on a road from a suburb to
downtown city.
Observe the number of salmon in local streams during
spawning season
Observe the process used for getting a fighter plane ready
to re-deploy
Observe customer service skills of front-line workers
Observe grocery prices in rich and poor neighborhoods
Dr. G. Johnson,
www.researchdemystified.org
18
Observation Exercise
Take out a piece of paper
1. Look around the room and write down
everything you observe
2. What are the gaps between what you see
and what you expected?
3. Why is there a gap?
4. What changes, if any, would you
recommend?
Dr. G. Johnson,
www.researchdemystified.org
Observation Process
 How can observation be systematic, reliable
and accurate?
 Collaboration
with stakeholders will help to set
realistic standards within the culture of the area.
 Have more than one observer: their observation
reports can be compared.
 Develop a check list.
 Train the observers so they observe and record
the same things.
Dr. G. Johnson,
www.researchdemystified.org
Observation Process
 Develop checklist and decision rules
 Train the observers
 Test for inter-rater reliability
 Observers complete their rating sheets independently
 Their ratings would be compared
 If there are differences, more training and clarification
are needed
 If little difference, then they are ready to go live
Dr. G. Johnson,
www.researchdemystified.org
Discussion: Housing Conditions
 Suppose we want to describe the conditions
of various neighborhoods that have had
housing programs over the past 10 years.
 What
are the conditions you would want to
researchers to assess?
How might they collect the information about
conditions in an unstructured way?
 How might they collect the information about
conditions in an unstructured way?

Dr. G. Johnson,
www.researchdemystified.org
Which Would Be a Better
Strategy?
 You want to determine the effectiveness of the
program’s efforts in engaging stakeholders in
discussions at meetings.
 Observe setting, seating arrangements,
interactions, and whatever else is interesting.
 Use a checklist to capture specific interactions:
who speaks, who does not, who interrupts, who
does not, who shows up, who does not.
Dr. G. Johnson,
www.researchdemystified.org
Discussion: Observe “Friendly
Environment”
 You have been asked to look at whether
your organization is a friendly environment
to women.
 If
an outsider were to visit your organization,
what might they want to observe to determine if
it is “friendly” and how might they go about
observing it?
Dr. G. Johnson,
www.researchdemystified.org
24
Combining Different Approaches



It is not unusual to do a preliminary study (or a
feasibility study) in a fairly unstructured way
Sometimes you do not know what you want to
know until you are in the actual situation
After that initial kicking tires, it is often easier
to develop a more systematic and structured way
for a larger observational study
Dr. G. Johnson,
www.researchdemystified.org
Types of Observation
 Obtrusive: observe with their knowledge
 Unobtrusive: observe without their
knowledge
Dr. G. Johnson,
www.researchdemystified.org
26
Obtrusive Measures
 Observations of behavior with participant
knowledge



Ask their: perceptions, opinions and attitudes gathered
through interviews, surveys, focus groups
Scores on tests related to the program—they know they
are being judged (eg. Dancing with the Stars)
Told: Dr. X will be observing our classroom today
Dr. G. Johnson,
www.researchdemystified.org
27
Limitations of Obtrusive
Observation
 The act of observation is likely to change
behavior:



Would teachers perform differently if they were being
observed?
Would people at meetings be more polite if they were
being observed by an outsider?
Would airplane baggage handlers resist the temptation
to play basketball with passengers’ luggage if they were
being watched?
Dr. G. Johnson,
www.researchdemystified.org
Unobtrusive Observations
 Observation without participant knowledge
 One-way mirrors
 Participant observer
 “Nickel and Dimed”: writer worked at low-wage jobs
and wrote about that experience
 Calling local IRS service center to observe their
knowledge of tax code and customer service
 Other ways to gather unobtrusive measures
Historical/document/archival data

Data already collected for another purpose: secondary sources
Dr. G. Johnson,
www.researchdemystified.org
29
Observation:
Living on Food Stamps
 From august 2008 to august 2009, food
stamp users rose from 29.5 million to 36.5
million. Half were children.
 Living on the equivalent of food stamps for
one month is one way to research some
aspects of that experience.
 Find out the maximum food stamp
allowance for our size family
Dr. G. Johnson,
www.researchdemystified.org
30
Observation:
Living on Food Stamps
 Research: data collection
 Maintaining a record of what is purchased.
 Maintaining a diary of everything that is eaten.
 No eating out at a restaurant
Dr. G. Johnson,
www.researchdemystified.org
31
Observation:
Living on Food Stamps
 What is it like to live on food stamps?
 Is
there enough food to last for the set
amount of time?
 Does the food meet the minimum
standards of what nutritionists believe is a
healthy diet?
 How does it feel to be restricted?
 What else did you observe/experience?
Dr. G. Johnson,
www.researchdemystified.org
32
Observation at Work
 How effective is communication at work?
 If
you were an anthropologist coming into your
workplace as a participant observer, how might
you design the study?
 What are the things you would want to
observe?
 What do you think you are likely to observe ?
 This is your hypothesis
 Decide about measures and scope of your
research
Dr. G. Johnson,
www.researchdemystified.org
33
Grounded Theory Concept
 Grounded theory begins by observing and
gathering data with no acknowledged filters or
coding.
 Researchers ake sense of the data and then
generate a theory to explain what they have
observed.
 Begin with roots in ground, move up to the top
leaves.
 Inductive logic.
Dr. G. Johnson,
www.researchdemystified.org
Grounded Theory Concept
 This is the opposite of traditional research
which uses deductive logic.
 We have an idea about how things work,
design a data collection instrument to gather
that data, and then test our theory.
 This
is the funnel approach.
Dr. G. Johnson,
www.researchdemystified.org
Observation
 Advantages:
 Actual situation or behavior (not self-report)
 Real-time (avoids memory decay)
 Some researchers enjoy this reality
 Challenges:
 Observer bias
 Coding problem
 Can be labor intensive and costly
Dr. G. Johnson,
www.researchdemystified.org
Takeaway Lessons
 Data collection requires tremendous attention to
detail.
 It takes time to nail everything down.
 Available data: cost-effective if it the data you
actually need and is high quality.
 DCIs: often used to collect data from files,
databases, and real life observations.

Need to know a lot before a DCI can be structured
Dr. G. Johnson,
www.researchdemystified.org
37
Takeaway Lessons
 Observations: “kicking the tires.”
 Unobtrusive observation:

closer to reality.
Unlike “reality shows” where they know everything
is being filmed-and maybe some incentive to create
drama.
 But
limited by our “paradigm”
Things can be invisible when does not conform to
our expectations
 Not seen because not looked for

Dr. G. Johnson,
www.researchdemystified.org
38
Creative Commons
 This powerpoint is meant to be used and
shared with attribution
 Please provide feedback
 If you make changes, please share freely
and send me a copy of changes:
 Johnsong62@gmail.com
 Visit www.creativecommons.org for more
information
Dr. G. Johnson,
www.researchdemystified.org