Week 8: Common Data Collection Methods in Public Administration
October 26, 2010
Lecture Notes Prepared by Rebecca Urban
Introduction
Today’s lecture was divided into two sections:
1. Remainder of the Week 7 lecture on Experimental Design, presented by Prof. Tedds.
2. Lecture on Data Collection
a. Use of interviews, documents and observations, presented by Prof. Brady.
b. Use of surveys, presented by Prof. Tedds.
1. Research Design: Experimental Design
Overview:





Introduction
Types of (pre) experimental design
Experimental design of interest to Public Administration
o General features of RCTs
o Diagrammatic explanation of RCTs
o An example: SPP
o Other Examples of RCTs in Public Administration
Natural or Quasi-experiments
o Examples
o Gautreaux Program
Key Issues in Experiments
o Milgram
Introduction:
Experimental design is a difficult and costly undertaking
 Will not be used in 598 reports
 Is important to know in order to evaluate papers (both in this program and in the
‘real world’)
It is attempting to answer explanatory questions and trying to uncover underlying causal
relationship
 Asking if X causes Y
There are three criteria for establishing criteria (pro-variation, no rival hypothesis and
temporal design)
Not all explanatory Questions can be answered using experimental design. Some
concepts are difficult to observe (ie. What does it mean to smoke?)
Types of (Pre) Experimental Design
There are numerous forms of experimental design:




















One-Shot
One-Group
Static Group
Random Group
Pre-Post Randomized Group
Solomon Four Group
Randomized Block
Factorial
One-Shot Repeat
Latin Square
Incomplete Block
Youden Square
Lattice Square
Fractional Factorial
Graeco-Latin square
Split-plot
Covariance
Time-series
Natural
Quasi
This lecture focuses on two designs:


Pre-Post Randomized Group; and
Natural or Quasi
Experimental Designs of Interest to Public Administration
Exploration of two experimental designs:
1. Randomized controlled trial (RCT)
o pre-post design with control group and randomization
o There is a debate on whether this is the “gold standard”
 It does have shortcomings
 Experimental design can easily be manipulated
 It is a research design and nothing more
 Some claim it is great in theory and poor in practical application
2. Natural or Quasi Experimental Design
o Takes the RCT design and removes the ability of the researcher to
randomize people into groups.
 Assignment is by nature, not controlled
o Provides an alternative to RCT when it is unethical or unfeasible to
randomize people into groups
General Features of RCTs

Causal variable has to be subject to manipulation by the researcher

Outcome variable must be observable (both pre-test and post-test)

Treatment effects must be observable

Random assignment of subjects to one of the two groups
o Experimental group (treatment)
o Control group (no treatment)

The ideal goal would be to take a person and observe them, give them treatment,
and then observe the changes. After which you would take that person back in
time, give them no treatment, then observe them
Diagrammatic Representation of RCT (Boot Camp)
Question: Are troubled youth, who participate in a boot camp, less likely to commit
criminal offences?
Treatment Group:
Measure Criminal Behaviour (pre)  Treatment  Measure Criminal Behaviour (post)
Control Group:
Measure Criminal Behaviour (pre)  No Treatment  Measure Criminal Behaviour
(post)
Results:
If the bootcamp has had an effect on the delinquent children there should be a reduction
in criminal offences by those in the treatment group
An Example: Self Sufficiency Project
This is a real example from self-sufficiency project:

How single parents on welfare are vulnerable to the welfare trap because welfare
is clawed back if they earn so much as a cent (so lose more than they gain by
working)

Premiums were generous at $12,000 a year so long as you were a long-term
recipient of welfare (more than a year). No welfare wall because you go out and
earn your money and still get the $12,000

Before treatment both those receiving the premium and those not are acting the
same

With treatment they go work and start working at a faster rate (appears at first that
it has overcome the welfare wall)

After 51 months they are almost identical

Just shifting behaviour, not changing behaviour
Other Examples of RCTs in Public Administration
American Examples:



Perry Pre-School Project: http://en.wikipedia.org/wiki/High/Scope
Project Star: http://www.heros-inc.org/star.htm
Moving to Opportunity: http://en.wikipedia.org/wiki/Moving_to_Opportunity
Canadian Example:

Mincome: http://en.wikipedia.org/wiki/Mincome
Natural or Quasi-experiments

Removes the ability of the researcher to randomize the groups

Randomized experiments are often unethical

Conditions seem similar to randomized experiment

Rely on chance circumstances mimicking the conditions of a randomized
experiment

Real-world events have impact on the separation of treatment and control groups
 Natural Experiments are complicated
Examples of Natural Experiments
Majority of policy/program change can be viewed as natural experiments




Quebec’s Day Care Program
Policy changes to EI in 1996
Iceland’s implementation of PAYE (pay as you earn)(there was no taxation)
Minimum wage changes in a province
Example: Gautreaux Program



One of the first cases in the U.S.
Concerning access to housing and the results
It was the pre-curser to the moving to opportunity program

Between 1954 and 1967, the Chicago Housing Authority (CHA) constructed more
than 10,300 public housing units.
o Most were built in poor, racially segregated neighborhoods.
Gautreaux brought a lawsuit in federal court challenging this practice because her
plaintiffs believed that it was unfair that CHA residents had to live in these poor
areas

Key Issues in Experiments
There are many known issues with experiments:

Selection bias
o Treatment and control groups are not the same, selected to increase the
results

History
o Life intervenes in experiments, ie. 9/11 played havoc on experiments

Maturation
o People mature over period of time for reasons unrelated to the treatment,
ie. grow up out of issues, this could make it appear as though your
treatment is working when in reality it’s not

Mortality or attrition
o They are dropping out of the program, ie. those more likely to commit
crimes are more likely to drop out

Testing/Hawthorne Effect
o Testing influences peoples behaviour, I know you’re watching me so I
may change my behaviour

Instrumentation Effect
o Outcome measure is not well defined, may not measure what you intend to

Diffusion of Treatment
o Occurs when subjects in each of the groups talk to each other, control
group changes behaviour to do what they think they should

Compensatory Behaviour
o Control group compensates for not having the treatment

Experimental effects
o When the researchers involves themselves into the groups and influences
the behaviour, they have a desired outcome

Ethical issues
o Milgrim, how willing people were to obey instruction even if it the action
was wrong or immoral like killing somebody (can cause psychological
distress on the participants)
Milgram Experiment





Study of behavioural obedience
Participants asked a question to a person they could not see, but could hear
If the person got the answer wrong the participant was told to shock them
Some people withdrew, while others gave enough voltage to kill a person and
they could hear the screams following every shock from a wrong answer
Participants suffered psychological stress, some thought they had killed somebody
when in reality they had not
1a. Data Collection: Interviews, Documents and Observations
Overview
Introduce key features of common data collection methods:
 Qualitative interviews;
 Documents;
 Observation
Qualitative Interviews




Interviews are a hermeneutic, post-structuralist or post-positivist epistemology
Demands proximity to participants and provides insights into the life experiences
of those interviewed
Centres on interpersonal exchange (the interviewer must learn from experience to
fully understand)
Interviewing is both a skill and an art
It is learnt through study followed by a lot of practice

Three broad approaches to interviewing:

o Informal conversational interview
 As though you are just meeting in everyday life. It is just a regular
conversation, often don’t understand they are being interviewed
o Interview topic guide
 Listing topics you are going to talk about then go through them in
no particular order
o Standardized open-ended interview
 You have set questions, but not set responses, ask in the sequence
they are ordered. Worded in a way in which the person is invited to
give an open response, not choose a,b,c,d
Special Kinds of Interviews
1. Elite interviews (i.e. Senior Public Servants)
 Well educated and hold positions of relative power
2. Focus Groups
 Group discussions that are organized to explore people’s views and experiences
on specific issues
Problems with Focus Groups

Video highlighted the numerous problems associated with focus groups (which
are commonly used in public policy and political party research)
1. They do not generate substantial results
2. The researcher guides the discussion the way he wants
3.
4.
5.
6.
7.
8.
People do not seem overly interested in the topic
Should have been more clear in his purpose
Seemed like his critiques would promote reluctance in participation
Bunch of people with not very informed information
Does not work well on controversial issues
Could have the group effect if people know each other (i.e. people have roles
where some are dominant and others are submissive)
Combined Approach
The approaches to interviews can often be combined. Could have a standard formal open
ended question approach formulated but might have to move towards a more informal
approach because of the way your participant answers questions or answers before the
question is completely finished.
The schedule you have created could end up somewhat different in reality when looking
at the questions actually asked.
Getting Access
Arranging and conducting interviews can be very time consuming.
It is easiest to contact mid-rank bureaucrats and professionals.
Non-elites
 Many challenges in trying to contact socially marginalized people
o It is hard to find them, you must determine where the frequent in order to
advertise, etc.
o It is hard to conduct follow-ups as they may not have internet or reliable
phone service
 There is a social distance from the research process
o They do not think that their opinions are important
o They often give short answers, i.e. yes or no
o They are not familiar with the process
Elites
 Elites can often have requests for many interviews and are busy people in general.
May have to send many requests.
 They are usually familiar with the interview process and are willing to talk at
length. However, they may have a particular agenda that they want to talk about
that could be irrelevant to your subject
 They often demand assurances and conditions
Asking Questions
There are six broad types of questions outlined in the readings:






Experience & behaviour questions;
Opinion & values questions;
Feeling questions;
Knowledge;
Sensory questions;
Background/demographic questions
In addition each type of question can be asked in three different time frames:
 Past;
 Present;
 Future
How open or closed?
 Elites tend to prefer open ended questions and dislike closed questions
 Exploratory research tends to have more open questions
 It is more time consuming and difficult to record, code and analyze open ended
questions
Respect:
 Elites expect you to be well informed
o Do not ask them anything you can find on the webpage of their ministry
(i.e. mission statement)
o Do your background research
 Make it seem as though their experience or expertise is important to the research
 Do not tape an interviewee without their prior consent
 If you are not recording the interview be sure to type up notes directly following
 Do not type notes on your computer, it makes the interview impersonal
Practical Information You Should Know About Interviews






Dress appropriately
Bring extra batteries for your recorder and pens
Bring food and beverages
Make sure to schedule enough time
Interviews are exhausting (mentally, emotionally, and physically)
Ensure that you are not conducting the interview in a noisy surrounding
Asking Questions: Practical Exercise for 598
Sample Questions:
 “Could you tell me a bit about your current position and role?” How long have
you been working and living in the Fernie area?
 “Are you aware of the recent debates about water service delivery in West Fernie?
What is your perspective on the issue currently?”
 “How have you been connected to this issue in the past?




“How do you feel about the status quo?
“What do you think needs to improve/what is fine?
“West Fernie residents pay the same rate for water service from the City of Fernie
as do City residents. In your opinion is this a fair and equitable rate?
“Do you have any concerns about water quality in West Fernie/the City of
Fernie?”
-These questions appear to be open ended but are not and actually received short answers.
-They did not receive the intended answers
-It is important to formulate and develop your questions carefully
Documents
Collecting Documents - Sampling
Census:
 All of a particular population of documents (i.e. all annual reports from 19911996)
Random Sample:
 Random sample of documents (i.e. random sample of newspaper articles on a
certain topic)
Theoretical Sampling:
 Evaluate a policy (establish its goals and impacts)
 Establish a picture of dominant views on a policy or political subject in a
particular period
 Tracking a process (i.e. tracking a new policy process)
Collecting Documents: Accessing


Publically available documents, i.e. websites, Hansard, etc.
Information not publically accessible
o Sometimes can be accessed by informally asking the agency
o May require an Access to Information Request
Observation




All interviews involve observation;
Concerned here with situations where observations are recorded and reported in
research findings;
It is a resource and time-intensive collection method.
Long term observation can be stressful and the researchers presence alters the
setting.
Types of Observations



Complete participant: conducts research covertly
o You don’t tell anyone you’re a researcher. It would be difficult getting this
through research ethics
Participant as observer: research role is explicit
o more common, clear that you are a researcher, but you are still
participating in the process
Observer as participant and complete observer: observation is clearly
structured, other contact is limited.
o more of just an observer
1b. Data Collection: Surveys
Overview



What is Survey Research?
Essentials of Survey Design
o Deliver
o Format
o Questions
o Pretest
o Coding
Strengths and Weaknesses of Survey Research
What Is Survey Research?

Use of a standardized questionnaire to gather data about unit of analysis

You do not deviate, there is no clarification and no follow up on questions asked

It is Exploratory and causal

The key is to design research instruments to fit the research goal

If interested in a household will ask the women of the house

If interested in a firm will ask managers etc

We can obtain facts, perceptions, opinions, attitudes, behaviours
Essentials of Survey Research
There are 14 steps, but refer to previous weeks because they were covered under research
design (i.e. type of question, unit of analysis, boundaries, time frame, sampling methods,
etc.)
Surveys are often generalized results, but this is not always the case
Survey Delivery Methods




Mail or Drop off Surveys
Email or Internet Based Surveys (survey monkey is a great tool)
Telephone Surveys (i.e. Stats Can)
In Person
Survey Delivery: Strengths and Weaknesses

Important to understand the costs and benefits

Rule of thumb – 80% is a representative response rate

Mail surveys have a response rate of around less than 50% with three follow-ups
up to 80%

Any response rate less than 50% you really need to think seriously about your
questions and who is responding, why are they responding, they tend to have a
vested interest in the topic and you won’t get a generalizable result

Anything less than 20% is extremely low and is not worthwhile

To maximize response rate ask them to participate in advance, include purpose,
reasons for participation, if its touching on sexy topics talk about anonymity and
confidentiality, send reminders, could offer incentives but be aware of the costs
of this
Designing & Formatting Surveys

Designing and formatting a survey is like writing a paper

Organization and formatting has a direct effect on response rate

Objective of research will drive questions

The longer the survey the more likely people will not do it or will drop out part
way through

The overall layout: think about how you start
o Introduction covering communication about the survey, who’s doing it,
who are they associated with, how will results be used, who to contact for
more info, how long it will take etc.

Want lots of white space, feels more calming

Use headings if applicable and easy to read fonts

Emphasize important words

Number the questions

Group similar questions together

Have a thank you at the end, it helps to know that it’s been submitted (list who to
contact in case of questions)
Question Intent
Ensure questions are clear and direct. Use language the participants will understand.
Three main characteristics:

Relevant
o Know what you want and design questions accordingly to get the answers
you want
o Questions should be based on the objectives of the survey and
information to be obtained

Accurate
o Questions that collect data in a reliable and relevant way

Necessary
o If a question is not pertinent to the subject, do not ask it
Question Sequence

Good survey design motivates recall

Important just like interviewing

If you only want a certain type of person weed them out at the beginning

First questions should be easy and interesting (keep them involved)

Flow should be sensible and geared towards respondents abilities

Questions should appear in the section where they should be (i.e. filter questions
in the first half, end survey with demographic information)

Sensitive questions should be in the middle and not out of the blue
Constructing Good Questions
Use BOSS:
 Brief









Objective
Simple
Specific
They should be short, ask one at a time, don’t make them complex
Be objective: do not assume familiarity with something
o Do not their assume mother is alive
Avoid abbreviations, jargon and double barrel questions
Ask precise questions, do not be vague and ask the specific question
Be wary of being argumentative
Be careful about forcing a response
o Ask yes/no questions but be aware of uncertainty
o Give them an option to not answer to avoid bias
Question Types
Open-ended questions
 Provides Context
 Analysis is time consuming and difficult
Closed-ended questions
 One answer
 Filter
 Ranked
 Likert Scale
Pretest
Give the test to your friends (YOU NOW HAVE PLENTY!)
 This helps you to understand competency and length as well as the quality of the
responses you are receiving
Cleaning and Organizing the Data for Analysis
How are you going to organize it? What tools are you going to use?




Creating and documenting a database structure
Cataloguing data into the database
Checking the data for accuracy
Transforming the data for analytical purposes
Strengths of Survey Research






Large samples are feasible (because they are short and simple)
Useful in describing the characteristics of a large population.
Administered from remote locations using mail, email or telephone.
Many questions can be asked about a given topic giving considerable flexibility to
the analysis.
Standardized questions make measurement more precise
Standardization ensures that similar data collected from groups then interpreted
comparatively
Weaknesses of Survey Research



Forces researcher to develop questions general enough to be minimally
appropriate for all respondents.
Surveys are inflexible in that they require the questionnaire to remain unchanged
throughout the data collection.
Need to ensure a reasonable response rate


May be hard for participants to recall information or to tell the truth about a
controversial question.
Survey research (excluding some interview approaches) can seldom deal with
"context."