Survey Research
UAPP702: Research Design for Urban & Public Policy
Class Notes
Based on Earl Babbie, The Practice of Social Research
Danilo Yanich
Center for Community Research & Service
School of Public Policy & Administration
University of Delaware
1
Topics appropriate for survey research
 Used for exploratory, descriptive &
explanatory purposes
 Probably best method to collect original
data for describing a population too large
to observe directly
2
Guidelines for Asking Questions
 Choose appropriate question form: two options
 Open-ended--R is asked to provide own answer to
question
 Closed-ended--R is asked to select an answer from
among a list provided by researcher
 Closed-ended questions require the categories of
answers to be exhaustive and mutually exclusive
3
Guidelines for Asking Questions, p.2
 Make items clear
 Babbie uses examples of "proposed peace
plan"...Which peace plan?
 And, employment "last week"...Monday
through Friday or Sunday through Saturday as
the Census Bureau defines week?
4
Guidelines for Asking Questions, p.3
 Avoid double-barreled questions
 Example: "Did you walk to work or carry your
lunch?" …..Yes or No
 Regardless of the answer you get from the
respondent, you don’t know what information
you have.
5
Double-barreled to the max
 Question on 1986 Harris poll re: American
public opinion about Libya
 If Libya now increases terrorist acts against the U.S. and
we keep inflicting more damage on Libya, then inevitably
it will all end in the U.S. going to war and finally invading
that country which would be wrong.
 Respondents were given opportunity to answer
“Agree”, “Disagree” or “Not sure”
6
Double-barreled to the max, p.2
 Notice the elements contained in the complex statement





Will Libya increase its terrorist acts against the U.S.?
Will the U.S. inflict more damage on Libya?
Will the U.S. inevitably or otherwise go to war with Libya?
Would the U.S. invade Libya?
Would that be right or wrong?
 These elements offer the possibility of numerous points
of view—far more than the three alternatives offered to
the R’s
7
Guidelines for Asking Questions, p.4
 Respondents must be competent to answer

Babbie uses example of asking students to indicate the percentage of fees to be
used for a long list of activities, about which the respondents have little
knowledge
 Respondents must be willing to answer

There may be a risk to the respondent in answering the question

Reluctance may be due to nature of the information sought, but good science and
human subjects requirements dictate caution.

Example: asking a staff member of a non-profit agency her/his opinion about the
leadership qualities of the administrator

Using a technique that guarantees anonymity or confidentiality greatly increases
R’s willingness to answer questions
8
Guidelines for Asking Questions, p.5
 Questions should be relevant to the Respondent

Babbie uses example of fictitious person, Tom Sakumoto


9% of R's said they were familiar with him.
Shows R’s tendency to be helpful to researcher
 Short items are best
 Avoid negative items

Possibility of misinterpretation is great

Babbie uses example of asking R to say if they agree/disagree with
statement that "The U.S. should not recognize Cuba".

Often, R’s will “read over” the word “NOT”
9
Guidelines for Asking Questions, p.6
 Avoid biased items and
terms


Questions should not
encourage a particular
response or discourage
another.
Be wary of "social
desirability" of answers
Attitudes re: government spending for
programs in the General Social Survey, 1989
More support
Less support
Assistance to the poor
Welfare reform
Halting rising crime rate
Law enforcement
Dealing with drug
addiction
Drug rehabilitation
Solving problems of big
cities
Assistance to big cities
Improving conditions of
blacks
Assistance to blacks
Protecting social
security
Social security
10
Surveys “gone wild”
 National Park Survey
 NRA Survey 1
 NRA Survey 2
 NRA Survey 3
11
Questionnaire Construction
 General questionnaire format should be
uncluttered
 Squeezed-together questionnaires are
disastrous
 Formats for respondents should be clean
and clear
12
Questionnaire Construction, p.2
 Contingency question
 A question that only applies to the R’s who
answered a previous question in a particular
“qualifying” manner
 Make sure that the directions for answering the
subsequent questions are clear for the
respondent
13
Contingency question, example
 Have you ever been abducted by aliens?
 [ ] Yes
 [ ] No (go to ques. # 4)
If yes: Did they let you steer the ship?
[ ] Yes
[ ] No
If yes: How fast did you go?
[ ] Warp speed
[ ] Weenie speed
14
Questionnaire Construction, p.3
 Matrix questions
 Advantages
 Uses space efficiently, R's will probably complete it faster
 Format may increase comparability of responses to
different questions
 Disadvantages
 May encourage researcher to use matrix format when
another form would be better
 Can foster a response-set among R's in which they may
develop a pattern
15
Matrix question, example
 Contact with criminal justice system
Have you ever…
Yes
No
Ref/DK
Been a defendant in a criminal case
1
2
7
Been a witness in a criminal case
1
2
7
Reported a crime to the police
1
2
7
Been the victim of a crime
1
2
7
16
Matrix question, example 2
 Using the A, B, C, D, F grading system that we learned in school,
please rate the importance of the following services in this hotel.
Hotel service
A
B
C
D
F
Ref/DK
Dining services
5
4
3
2
1
7
Laundry service
5
4
3
2
1
7
Room service
5
4
3
2
1
7
17
Questionnaire Construction, p.4
 Ordering questions in a questionnaire
 The appearance of one question can affect
answers given to later ones
 As remedy, do not make the questions random,
that will look chaotic to the R and confuse the
issue
 Safest solution is sensitivity to the problem
18
Questionnaire Construction, p.5
 Questionnaire Instructions
 Must be clear, concise and COMPLETE
 Pre-testing the questionnaire
 ALWAYS PRE-TEST!!!!
19
Comparing different survey methods
Choosing among mail questionnaire, personal interview and telephone survey
Factors influencing coverage and information
gathered
Mailed
questionnaire
Personal
interview
Telephone
survey
Lowest relative cost
1
3
2
Highest % return
3
1
2
Highest accuracy of information
2
1
3
Largest sample coverage
3
1
3
Completeness, including sensitive material
3
1
2
Overall reliability & validity
2
1
3
Time required to gather information
3
2
1
Ease of gathering information
1
3
2
2,2,4
5,1,2
1,5,1
Total number of rankings—1,2,3
Note: 1=most favorable ranking, 2=intermediate ranking, 3=least favorable ranking
Source: Delbert Miller. Handbook of Social Research Design and Measurement, 5th Edition,1991, p. 168
20
Strengths of survey research
 Useful in describing large populations
 Are flexible in that you can ask many questions
regarding your topic
 Standardized questionnaires have strength
regarding measurement generally
 Survey researcher must ask the same question of all
subjects and…
 Impute the same intent to all respondents giving a
particular response
21
Weaknesses of survey research
 Standardized questionnaire items often
represent the least common denominator in
assessing people’s attitudes, etc.
 Does not deal well with the context of social life
 Can be inflexible…cannot change the survey
instrument if field conditions change
22
Weaknesses of survey research, p.2
 Subject to artificiality

A person giving a conservative answer to a questionnaire does
not necessarily mean the person is conservative

Artificiality has two aspects
 Topic of study may not be amenable to measurement through a
questionnaire
 Studying the topic may affect it…

Asking someone whether they think the governor should be impeached
when R may have given no thought to it until asked for the opinion
23
Ch 14: Quantifying data , Coding
 System to translate the responses to questions to
numeric expressions that the computer can process
 Two basic approaches to coding process
 Begin with developed coding scheme
 Babbie uses occupation as example...but must use scheme
that is appropriate for research question.
 Generate codes from data that you collected.
24
Using a developed coding scheme, example
 Occupation
 The question to the R would be: What is
your occupation?
 The R’s response is then placed in one of
several categories that you have already
identified
25
Using a developed coding scheme,
example, p.2
 What is your occupation?
 Let’s say that the answer was “nurse”
 Scheme 1: by type

Professional, managerial, clerical, semi-skilled, etc.

In this scheme, the R’s occupation would be placed in the
“professional” category
 Scheme 2: by sector of the economy

Manufacturing, health, education, commerce, etc.

In this scheme, the R’s occupation would be placed in the “health”
category
26
Using a developed coding scheme,
example, p.3
 Note: record the response of the R verbatim
 Place the response into one of your pre-determined
categories during the data manipulation phase
 Use “recode” facility in SPSS
 Remember, you can always “aggregate” data
 Cannot “disaggregate” data
27
Using a developed coding scheme,
example, p.4
 What is your occupation?

Verbatim responses
Type








R1: “nurse”
R2: “sell shoes”
R3: “build cars”
R4: “manager at Wendy’s”
R5: “physician”
R6: “computer programmer”
R7: “retired”
R8: “soldier”
Sector
professional
health
sales
commercial
semi-skilled labor
commercial
managerial
commercial
professional
health
professional
information tech
retired
retired
military
military
28
Generating codes from data collected,
example
Student responses to “biggest problem facing college today”
Responses can be coded as “academic” or “non-academic”
Academic
Non-academic
Tuition is too high
x
Not enough parking spaces
x
Faculty don’t know what they’re doing
x
Advisors are never available
x
Not enough classes offered
x
Cockroaches in the dorms
Too many requirements
x
x
Cafeteria food is infected
x
Books cost too much
x
Not enough financial aid
x
29
Codebook construction

End product of the coding process is the conversion of data
items into numerical codes

These codes represent attributes composing variables which....

Are assigned locations within a data file


Location means the specific column of the data file where, for
example, the responses for “gender” would occur
A codebook is a document that describes the locations and
lists the assignment of codes to the attributes composing
those variables

It is the fundamental document of the research process
30
Codebook serves two functions
 Primary guide for the coding process
 Guide to locating variables in the data
file during analysis
31
Example of Coding Instructions
Public Attitudes: Crime, Drugs & Public Services
Enterprise Community/Wilmington/Statewide Survey
Variable Name (column location)
Value label
1. ID# (1-3)
Continuous (001 through n)
2. Area (4)
1= Westside, 2= West Center, 3= Delaware Avenue
4= Southwest, 5= Eastside, 6= Northeast
7= Northwest, 8= New Castle, 9= Kent, 10= Sussex
3. Defendant in criminal case (5)
1=Yes, 2=No, 7=Ref/DK, 8=NA
4. Witness in a criminal case (6)
1=Yes, 2=No, 7=Ref/DK, 8=NA
5. Report crime to police (7)
1=Yes, 2=No, 7=Ref/DK, 8=NA
6. Victim of crime (8)
1=Yes, 2=No, 7=Ref/DK, 8=NA
32
Data Cleaning...a fundamental activity

The process of detecting and correcting coding errors

Two types


Possible-code cleaning--for any given variable there are only a
specified set of codes possible

Example, gender--computer program would "beep" when an
erroneous code is entered and refuse the code

Family Court experience with gender codes
Contingency Cleaning--process of checking only those cases
that should have data on a particular variable do in fact have
such data
33