Overview
We’ll cover





What is a survey?
Sampling
Questionnaire development and design
Analysis, interpretation, and reporting
Using the results
2
What is a Survey?
A scientific social research method that involves:
 Selecting a random sample of people to answer some
questions
 Designing & administering a standardized
questionnaire to get information about the research
questions
 Analyzing the results to provide descriptions about the
people in the sample and find relationships between
different responses
 Generalizing the results to the population from which
the sample was drawn
3
Basic survey formats
 Telephone interview
 Face-to-face interview
 Computer assisted interview
 Self-administered “paper & pencil”
survey (scannable or not)
 Self-administered Web survey
 Self-administered Email survey
4
Why do a survey?
Really bad answer:
Why not? Anyone can do a survey.
Slightly less bad answer:
My boss told me to.
Better answer:
I want to systematically evaluate the extent to which
my program is meeting its goals and objectives in
order to improve our programs through pro-active,
well-informed decision making.*
*Adapted from CUPR FAQ “What is Assessment?”
(http://www.ncsu.edu/provost/academic_programs/
FAQ/UAPRFAQwhatisassment.htm)
5
But why do a survey?
 To collect information not available from other
sources
 To have comparable information
 To be able to generalize results to the
population in which you’re interested
6
Before you plan to do a your
own survey…
Think about alternative sources


Information from non-survey sources
Information from existing survey data
7
Still think you need to do a
survey?
Who is going to









Identify the goals and objectives
Develop the research design
Design the questionnaire
Identify the population and select the sample
Pre-test the survey
Administer the survey
Analyze and interpret the results
Write up the results
Present the findings
8
Still think you need to do a
survey?
 How much TIME do you have ?
 Who is going to PAY for it all ?
 Who is going to USE the results ?
9
So you still want to do a
survey …
Helpful hints!
 Familiarize yourself with survey research ‘best
practices’ and code of ethics (see
www.aapor.org)
 (Get approval from your Institutional Review
Board)
 Form an advisory group w/ relevant
constituents
 Organize a survey team
10
So you still want to do a
survey …
More Helpful Hints!
Develop a detailed overall research
design





What?
Why?
When?
How?
Who?
11
What is the purpose of the
survey?
Explicitly state your
objectives
Example:
Enhance employee well-being
12
What is the purpose of the
survey?
Identify factors related to objectives
Examples of factors related to “well-being”:




Pay and benefits
Physical work environment
Work relationships
Etc.
13
What is the purpose of the
survey?
Make an exhaustive list of everything
you want to be able to measure
related to your objective.
Examples of dimensions of “work
relationships”:




Communication of necessary information
Feedback on performance
Opportunities to provide input in decision-making
Etc.
14
Who do you need to survey?
Identify your population(s)
Population = The theoretically specified
aggregation of all units about which
information is collected.
Examples for “work relationship”
populations?




Supervisors
Employees
Unit members
Other?
15
Who do you need to survey?
Survey population can influence








Data collection methods you use
Questions you ask
Analyses you do
Information you gather
Assumptions you can make
Recommendations you can make
Action you can take
Etc.
16
Who do you need to survey?
All possible populations?
(which technically would be a “census”)
If you
Must keep everyone happy
 Have unlimited time
 Have unlimited resources

17
Who do you need to survey?
Everyone in a specific population?
(which would still be a census)
If you
Have a relatively small population
 Have reason to expect a relatively low response
rate
 Want to have enough respondents to do subgroup analyses for small groups

18
Who do you need to survey?
A sample of the population?
If you
Understand sampling procedures
 Can convince others that it’s okay to sample
 Want/need to limit time in the field
 Want to enhance ability for quality control over
entire process
 Want to keep costs down

19
Select the sample
Create a “sampling frame”
A list of elements from which the sample will
be selected (i.e. members of the population)
Examples?
Company personnel data file
 Unit directory
 Etc.

20
Select the sample
Probability sampling methods
Simple random sample

List elements in the sampling frame and use random
number chart or computer program to select elements

Systematic sampling

List elements in the sampling frame and take every Nth
element, based on how many needed in sample


Warning: Be attentive to any inherent patterns in how the
sampling frame is ordered.
Stratified sampling


Draw appropriate proportion of elements from
homogenous groups interested in
21
Select the sample
Sample size? Depends on:

Cost


Anticipated response rate
Number of responses needed per
subgroup interested in

Level of precision want (“margin of error”)
See Appendix A for more information on sampling.
22
Select the sample
Sample Warnings!
BAD SAMPLE = BAD DATA


Survey results are only representative of the
sampling frame from which the sample was
selected.
Sample survey results are generalizable only if
respondents are randomly selected.
23
Select the sample
A Warning About Sampling Error!
It’s NOT the ONLY source of
survey error!!!
24
Select the sample
A Warning About Response Rates!
A high response rate
does NOT mean good data
if those who responded
are ‘biased’ in some way
25
Select the sample
A Warning about Web Surveys!
If you can not control who
accesses and responds to a web survey
the results are NOT
generalizable to the population.
26
Design the questionnaire
REMEMBER YOUR OBJECTIVES!
 What do you want to learn?
 How will you use the information?
27
Design the questionnaire
Don’t reinvent the wheel!!
Develop (or borrow*) questions/items to
measure your specific outcomes
*In general it is perfectly acceptable to borrow questions
from other surveys, but always get permission and/or
check for copyright restrictions. See Appendix B for
examples of places to find questions.
28
Design the questionnaire
To have confidence in your survey
results your questions must be

Reliable
Questions consistently convey the same meaning
to all people in the population being surveyed.

Valid
Questions measure what they are intended to
measure
29
Design the questionnaire
Warning!
Bad Question = Bad Data
The way a question is worded
and the response options offered
determine the nature of the data received.
30
Design the questionnaire
Open-end questions

Pros





Stimulate free thought
Solicit suggestions
Clarify positions
Richer information
Cons



Respondent burden
Incomplete, irrelevant, uninterpretable responses
Coding, analyzing, reporting
31
Design the questionnaire
Suggestion for
open-end questions!
Limit the number of open-end questions
BUT always include
at least one
for additional comments.
32
Design the questionnaire
“Forced-choice” (closed-ended)
questions

Pros





Less demanding on respondent
Determine levels of intensity, frequency of
participation, etc.
Use to form scales
Easy to analyze and make comparisons
Cons

Limits richness of data
33
Good Questions…
1)
2)
3)
4)
5)
6)
7)
8)
9)
10)
Are clear and use simple language
Are concise
Are specific
Are possible to answer
Don’t overly tax the respondent’s memory
Are not overly sensitive
Are relevant to the respondent
Do not use double negatives
Avoid biased terms
Have only 1 part (not “double-barrel”)
34
Good Questions…
Most importantly…
Can be connected to your objectives
and

Provide usable information

35
Good Questions…
Good Questions:
Use simple language...
My supervisor gives me clear feedback on my job
performance.
Is better than :
My supervisor provides lucid feedback when
transmitting her thoughts regarding the feats I
have accomplished in the preceding year.
36
Good Questions…
Good Questions:
Are concise...
How interested would you be in a flexible work
schedule in which you could work at home one
day a week?
Is better than:
How interested would you be in a flexible work
schedule that would still require you to work 40
hours per week, but on one weekday per week
you could not physically come in to the office but
work from home?
37
Good Questions…
Good Questions:
Are specific...
In what year did you first start working in a
permanent, full-time position at NC State
University?
Is better than:
When did you start working here?
38
Good Questions…
Good Questions:
Are possible for the respondent to
answer...
NO:
The overhead money I bring in on my
grants has a direct impact on the pay
raises provided by the state legislature.
39
Good Questions…
Good Questions:
Do not overly tax the respondent’s
memory...
NO:
How many of your colleagues said hello to
you within two weeks of you starting to
work at NC State?
40
Good Questions…
Good Questions:
Are not overly sensitive…
NO:
How often do you take time out to pray
during the course of a normal work day?
41
Good Questions…
Good Questions:
Are relevant...
NO:
My supervisor should get a better haircut.
42
Good Questions…
Good Questions:
Do not use double negatives...
The lack of flexibility in my work hours makes it
difficult for me to spend the amount of time I
would like with my family.
Is better than:
The inflexibility in my work hours does not make it
difficult for me to spend the amount of time I
would like with my family.
43
Good Questions…
Good Questions:
Do not use biased terms or suggestions...
To what extent do you agree or disagree that all
employees need to be in the office during
regular business hours?
Is better than:
Don’t you agree with the Chancellor when he says
that all employees need to be in the office during
regular business hours?
44
Good Questions…
Good Questions:
Do not ask two questions in one (“doublebarrel”)...
My supervisor is knowledgeable about performance
review policies.
My supervisor takes the time to make sure I
understand performance review policies.
Is better than:
My supervisor is knowledgeable about performance
review policies and takes the time to make sure I
understand them as well.
45
Response Options
Make sure your response options:
 Reflect the concepts you are trying to measure
 Fit with the question wording
 Get at level of precision/intensity needed
 Allow for comparisons between items
 Use “undecided/don’t know’ sparingly
 Are used consistently
 Use balanced scales
 Are mutually exclusive
 Are exhaustive
46
Response Options
Most importantly
 Relate back to your objectives
and
 Correspond to how want to use the results
47
Response Options
Must use
 Balanced scales
Example: NOT balanced...
Q. How would your rate your supervisor’s leadership
skills?
Outstanding
Excellent
Very good
Good
48
Response Options
 Balanced scales continued
Example: are balanced...
Q. How would your rate your supervisor’s
leadership skills?
Excellent
Good
Fair
Poor
49
Response Options
Must be:

Mutually exclusive
can select only one appropriate answer
Example: NOT mutually exclusive...
Q. On most days, what time do you leave your house
for work?
6:00 AM or earlier
7:30-8:00 AM
6:00-6:30 AM
8:00-8:30 AM
6:30-7:00 AM
8:30-9:00 AM
7:00-7:30 AM
9:00 AM or later
50
Response Options

Mutually exclusive continued
Example: IS mutually exclusive...
Q. On most days, what time do you leave your house
for work?
Before 6:00 AM
7:30-7:59 AM
6:00-6:29 AM
8:00-8:29 AM
6:30-6:59 AM
8:30-8:59 AM
7:00-7:29 AM
9:00 AM or later
51
Response Options
Must be:

Exhaustive
all possible answers are listed (including e.g., “other, “
“don’t know,” etc.)
Example: NOT exhaustive
Q. On most days, what time do you leave your house
for work?
6:00-6:29 AM
7:30-7:59 AM
6:30-6:59 AM
8:00-8:29 AM
7:00-7:29 AM
8:30-8:59 AM
52
Response Options

Exhaustive continued
Example: IS exhaustive
Q. On most days, what time do you leave your house
for work?
Before 6:00 AM
7:30-7:59 AM
6:00-6:29 AM
8:00-8:29 AM
6:30-6:59 AM
8:30-8:59 AM
7:00-7:29 AM
9:00 AM or later
53
Response Options
Some common scales:






Excellent, Good, Fair, Poor
Strongly agree, Agree, Disagree, Strongly disagree
Very useful, Somewhat useful, Not very useful, Not
at all useful
Definitely, Probably, Probably not, Definitely Not
All of the time, Most of the time, Some of the time,
Seldom, Never
Strongly agree, Agree, Neither agree nor disagree,
Disagree, Strongly disagree
54
Questionnaire format








As short as possible
Spread out & uncluttered
Attractive & professional looking
Broken into logical sections
Have a good flow
Uncomplicated & easy to follow
Include simple instructions for answering questions
Should have clear skip patterns for contingency
questions
55
Questionnaire format
Question order

Start with interesting, easy, non-threatening
questions
Remember –
‘Question order effects’
Earlier questions provide information and context that
influences responses to later questions
56
Questionnaire
Suggestion!
Before you start collecting data
use the questionnaire to
 Prepare an analysis plan
 Outline the final report
57
Pre-Test
A pre-test is
VERY VERY important!
Test for:




Question clarity
Questionnaire format
Variance in responses
Etc.
58
Data collection
Stages (in a perfect world…)
1.
2.
3.
4.
5.
Preliminary announcement
Cover letter, questionnaire w/ return
envelope and/or survey URL
Reminder (with copy of questionnaire and
return envelope and/or survey URL)
2nd reminder (with contact info to request
copy of questionnaire and/or URL)
Thank you card/page
59
Cover or “welcome” letter
Include with any self-administered survey



(On letterhead)
“Signed” by most important person possible
Briefly explain
 purpose of survey
 how results will be used
 why respondent was selected
 why participation is important
 (how to access questionnaire)
60
Cover or “welcome” letter
Continued





Emphasize confidentiality (if applicable)
(Describe/explain incentives)
(Follow IRB guidelines)
Provide contact information
THANK respondent for participating
61
Cover or “welcome” letter
Warnings about
Email Announcements


Be sure to follow any mass email policies
Think about spam filters:

Subject line text?

‘From’ field?
 Try to have identifiable name/title in ‘from’
field

Keep track of undelivered/unopened emails
62
A Word about Web
Surveys….
LOTS of software to choose from
Consider:







Extent of involvement/control over administration
Flexibility of survey design
Population/sampling options
Data collection capabilities non-response follow-ups
Data management and analysis options
Reporting capabilities
Cost
63
A Word about Web
Surveys….
Warning!
Even if using web survey software you
still need the skills to design and
administer a good, useful survey and
to analyze the results!!!
64
Clean, analyze, & interpret
the data




Get input from your advisory
committee
Remember your objectives
Remember your report outline
Remember your audience
 Keep it simple…
 You can always do more analyses…
65
Interpreting the results
Everything is relative!
Suggestions for what to look for:





Do responses/ratings meet pre-defined goals?

(e.g., % giving a positive rating)
How do responses/ratings for individual items compare to
each other?
What factors/characteristics are related to ratings?
How do responses/ratings from one group of respondents
compare to another?
(How have ratings changed over time?)
66
Prepare written report(s)
 Get input from your advisory
committee
 Think about the intended audience



What do they need to know?
How they are going to use the report?
How they like to get information…
67
Prepare written report(s)
Generally include




What you did (methods)
Why you did it
What you learned
How it can be used
68
Using the results
Take action: Some suggestions…




Spread the word!!
Institute policies/changes to address
problem areas identified by ratings that were
lower than predefined set goals.
Gear policies towards specific groups as
needed.
Do follow-up research
69
Want to learn more about
surveys?
American Association of Public Opinion Research, “Best Practices for Survey
and Public Opinion Research.” http://www.aapor.org/ethics/best.html
Babbie, Earl (2002), The Basics of Social Research 2nd ed. Wadsworth
Thomson Learning: CA.
Dillman, D.A. (2000), Mail and Internet Surveys 2nd ed. Wiley: NY.
Fink, Arlene and Jacqueline Kosecoff (1998), How to Conduct Surveys. Sage:
CA.
Fowler, Floyd J. (1993), Survey Research Methods. Sage: CA.
Fowler, Floyd J. (1995), Improving Survey Questions. Sage: CA.
National Council on Public Polls, “20 Questions a Journalist Should Ask About
Poll Results.” http://www.ncpp.org/qajsa.htm
Sudman, Seymour and Bradburn, Norman (1982), A Practical Guide to
Questionnaire Design. Jossey-Bass: San Francisco.
Tourangeau, R., and Smith, T.W. (1996), “Asking Senstive Questions: The
Impact of Data Collection Mode, Question Format, and Question Context,
“ Public Opinion Quarterly, 60:275-304.
70
Appendix A
Response Rate
# surveys completed and returned
net sample size*
*net sample size = sample size - undeliverable surveys
71
Appendix A
Margin of Error
Based on
# people in the sample (or population)
# people who respond
the response distribution




e.g., 75% said yes, 25% said no
how sure you want to be of you data (“confidence
interval”)


e.g., a 95% confidence interval means that you can be
sure that 95 out of 100 times the responses to a given
question would be within some calculable percentage
points of the actual number you got.
72
Appendix A
Margin of Error
Examples
(at 95% confidence interval)
Population Size # Respondents
100
75
40
30
10
7
Margin of Error
+ 2.8
+ 4.5
+ 11.1
E.G. If 50% of the 7 respondents (from your population of 10) said
“yes,” you can be sure that 95 out of 100 times between 38.9%
and 61.1% of those in your population would say “yes.”
73
Appendix B
Survey Question ‘Banks’

Eagleton Poll Archives
http://www.scc.rutgers.edu/eagleton/

The Roper Center (fee based)
http://www.ropercenter.uconn.edu/

The Odum Institute
http://www.irss.unc.edu/data_archive

The Gallup Poll
http://poll.gallup.com/

ICPSR
http://www.icpsr.umich.edu/
74