Survey Design and
Implementation
Elizabeth C Wells
Doctoral Candidate
Ag & Extension Education
http://www.msu.edu/~wellselz/
Why do people fill out surveys?
What do you want to ask/measure?
Accuracy/Survey Error
Random Sampling
Writing Questions
Questionnaire Design
Survey Implementation
Telephone, Interview and Internet Surveys
UCRIHS
Why People Fill Out Surveys
• Social Exchange Theory
Theory of human behavior in which
actions of individuals are motivated by the
return these actions are expected to bring.
Elements of reward, cost and trust
Social Exchange Theory
• Rewards
show positive regard
say thank you
ask for advice
support group values
give tangible rewards
make the questionnaire interesting
Social Exchange Theory
• Reduce Social Costs
avoid subordinating language
avoid inconvenience
avoid embarrassment
make it short and easy
minimize requests for personal information
keep requests consistent
Social Exchange Theory
• Establish Trust
provide token of appreciation in advance
sponsorship by legitimate authority
make task appear important
invoke other exchange
Constructs and Measurement
• What do you want to know?
• What are the variables?
• How will you measure it?
behavior, attitudes, feelings, demographics
Survey Error
• Sampling Error – Who are you sampling?
• Coverage Error – Does your list include
everyone?
• Measurement Error – Does everyone answer a
question the same way?
• Non-response Error – Why did respondent not
answer:
Instrument (whole questionnaire not returned)
Item (question not answered)
Non Response Error
• Only an issue if non responders would
answer differently and change results
compare early to late respondents
double dip
Random Sampling
• How much sampling error can you
tolerate?
• How large is the population?
• How varied is the population?
• How confident do you wish to be about
estimates made from the sample?
Random Sampling - Chart
Pop Size
100
200
400
600
800
1000
50/50 split
80
132
196
234
260
278
80/20 split
71
111
153
175
188
198
Sample size for a 95% confidence level
And a + 5% sampling error
Random Sampling
• http://www.randomizer.org
A www site that will generate random
numbers which you can use for random
selection of participants on a mailing list.
Writing Questions
• Everyone will interpret the same way
• Respondents are able to respond
accurately
• Respondents are willing to answer
Writing Questions
•
•
•
•
•
•
•
Use simple words
Keep it short
Be specific
Do not talk down
Avoid bias
Avoid hypothetical questions
Use complete sentences
Question Criteria
• Does the question require an answer?
If you fixed dinner last night, did you eat meat as part of
that meal?
• To what extent does the respondent have
a ready made and accurate answer?
Are tall people more likely to be elected President?
• Can respondent accurately recall and
report past behavior?
How much television did you watch last month?
Question Principles
• Use simple words and phrases
tired……………………………exhausted
honest…………………………candid
work……………………………employment
most important………………..top priority
free time……………………….leisure
doctor………………………….physician
your answers………………….your responses to this
questionnaire
Question Principles
• Avoid vague quantifiers
How often did you attend religious services last year?
 Never
 Rarely
 Occasionally
 Regularly
Question Principles
How often did you attend religious services
during the past year?
 Not at all
 A few times
 About once a month
 Two or three times a month
 About once a week
 More than once a week
Question Principles
• Avoid bias by using both positive and
negative sides in the question stem
“To what extent do you agree….”
vs
“To what extent do you agree or disagree with
this statement…. “
Question Principles
• Avoid double-barreled questions
16. Should the new school have a swimming
pool that includes lanes for swimming laps
and is not enclosed for winter use?
Question Principles
• Avoid asking respondent to say yes in
order to mean no.
5. Do you favor or oppose not allowing the
state to approve a casino without approval
from 60% of the voters?
 Favor
 Oppose
5. Do you favor or oppose requiring 60%
approval by voters in order for a casino to
be built.
 Favor requiring 60% approval
 Oppose requiring 60% approval
Question Principles
• Use equal numbers of positive and
negative categories for scalar questions
AND distinguish undecided from neutral by
placing at the end of the scale.
To what extent do you agree or disagree with
this statement: “Listening to classical music
is good for a persons emotional health.”
 Strongly agree
 Somewhat agree
 Neither agree nor disagree
 Somewhat disagree
 Strongly disagree
 No opinion
Question Principles
• Soften impact of potentially
objectionable questions:
Have you ever shoplifted something from
a store?
vs
Have you ever taken something from a
store without paying for it?
22. What was your total income from all sources in 2003?
______________ Total income for 2003
22. Which category best describes your total income from
all sources in 2003?
 $10,000. or less
 $10,001. to $20,000.
 $20,001. to $35,000.
 $35,001. to $50,000.
 $50,001. or above
Question Principles
• Don’t combine numbers and words
Doing so may impact respondents
perception of the scale.
1= strongly agree, 2 = agree,
3 = disagree, 4 = strongly disagree
Question Principles
• Eliminate check-all-that-apply questions to
reduce primacy and sufficiency effect.
People tend to check the first ones more
than the last ones and sometimes check a
few and feel that is enough to satisfy the
researcher.
Question Principles
 Extension Agent
 Other Cattlemen
 Fertilizer and other
salesmen
59% selected
Extension Agent
 Other Cattlemen
 Fertilizer and other
salesmen
 Extension Agent
41% selected
Extension Agent
Questionnaire Design
• Paper and layout choice
booklet format preferred
one column for legal size
two columns for full size
avoid upper left staple design
avoid unusual size, shape or fold
Order of Questions
• A questionnaire is like a conversation. It
evolves according to social norms.
Constantly switching topics makes it
appear as though the researcher is not
listening to the respondents answers.
• Doing otherwise results in answers that
are less thought out when you want them
to focus!
Order of Questions
•
•
•
•
•
•
•
•
•
What was your family income in 2003?
Do you like to play golf?
What is your opinion on Global Warming?
Are you married?
How often have you gone fishing this year?
How adequate is your current health care?
How old are you?
What is your political party preference?
What is your occupation?
Question Order
• Group alike questions together
• Begin asking questions in the upper left
corner
• Ask one question at a time
• Use item-in-a-series format
The First Question
•
•
•
•
Should clearly apply to everyone
Question should be easy
Question needs to be interesting
Connectedness between the question and
the purpose of the survey.
Unacceptable First Questions
1. Please think about the things that make
MSU Extension a pleasant place to work
and write down the five most important
aspects and rate them one to five in
importance to you.
.
Unacceptable First Questions
1. Please describe in your own words what
you consider good about working for
MSU Extension.
1. What year were you born?
_________year born
Good First Question
1. Thinking about MSU Extension, how
would you rate it as a place to work?




Excellent
Good
Fair
Poor
Graphic Design
• Stimuli on page consists of two languages:
written words
graphic symbols and arrangements
Graphic Design
• Desired navigational path
• Create visual navigational guides
• Develop additional guides for redirecting
during skip patterns
• Be consistent through out your instrument
Questionnaire Design
• Instructions need to be exactly where
needed not at the beginning of the whole
questionnaire.
• Use italics or bolding or underline to
emphasize instructions and directions
Questionnaire Design
• Pre-attentive processing is 210 degrees
• Focused reading is 2 degrees or about 8
to 9 characters
• Use font, size, contrast, underline, spacing
to lead the respondent through the
questionnaire
Start here:
1. Which of the following is your main work activity?
 Research
 Teaching
 Administration
 Other (Please Specify) _________________
START HERE:
 Which of the following is your main work activity?
 Research
 Teaching
 Administration
 Other (Please specify) ____________________
Questionnaire Format
• For paper questionnaires the standard font
is 12 pt Times New Roman – also consider a
larger font for senior audiences
• For on-line questionnaires the standard
font is 10 pt Ariel
Questionnaire Design
• Use a square for your answer and ask for
an “X” in the square.
[ ]

1
Questionnaire Design
• Place instructions exactly where needed
within the question
• Differentiate with font, bolding, underlining
etc
5. How many months have you worked at your current job?
_________Number of months
Please be as specific as possible in answering the next
question, including any area of specialization. Example:
Extension Agent – Agriculture. If you had more than one job,
answer for the job for which you worked the most hours.
6. What kind of work do you do in your current job?
_________________________________ Kind of work
5. How many months have you worked in your current
job?
_____________________ Number of months
6. What kind of work do you do in your current job?
Please be as specific as possible in answering. Include
any are of specialization, for example: “Extension Agent
– Agriculture”. If you had more than one job, answer for
the job which you worked the most hours.
______________________________Kind of work
Questionnaire Design
• Vertical alignment eases response task
• Don’t double or triple bank answers
1. Which one of the following best describes the
4-H project with the largest enrollment within
your club?
 market beef
 dogs
 goats
 lambs
 cats
 poultry
 hogs
 rabbits
 dairy
1. Which one of the following best describes the 4-H
project with the largest enrollment within your club?
 beef
 hogs
 lambs
 dogs
 cats
 rabbits
 goats
 poultry
 dairy
Questionnaire Design
• Major changes are needed for skip patterns
• Skip patterns are difficult to follow
• Skip patterns often lead to item non response
12. Normally do you eat breakfast every day?
 Yes (Go to 13)
 No (Go to 18)
13. How many times a week do you drink orange
juice with your breakfast?
__________ Times a week
12. Normally do you eat breakfast every day?
 No
(Skip to 18)
 Yes
13. (If Yes) How often do you drink orange juice
with your breakfast?
____________ Times a week
Questionnaire Design
• Demographic data questions
Place at end of questionnaire but not on
back page
Only ask what is necessary
Questionnaire Design
• Front cover needs to stand out
Consider simple art work
Include name of survey
Include name and address for return
Questionnaire Design
• Back cover needs to be simple with white
space
Try not to ask specific questions
May place open ended question here
Give opportunity to add comments
Conclude with a thank you
Implementation
•
•
•
•
•
Respondent friendly questionnaire
Multiple contacts
Token pre-paid incentive
SASA for return
Personalized correspondence
Implementation
• Four contacts plus one “special” = 5 contacts
Brief pre notice letter
Questionnaire mailing with SASE
Thank you postcard
Replacement questionnaire
Final contact with “special” mailing
Implementation – Contact 1
• Pre-notice letter:
Date at top
Personalized letter
What will happen
What it is about
Usefulness of survey
Thank you
Real signature
P.S. Token incentive
Mail out a few
days to one week
before mailing
instrument
4-6% higher
response rate
Implementation – Contact 2
• Mail the questionnaire
Include personalized cover letter
Questionnaire
Self addressed stamped envelope with
real stamp
Enclose incentive
Use first class postage – plain envelope
Implementation – Cover Letter
•
•
•
•
•
•
•
•
•
Inside address
Request
Why you were selected
Usefulness of survey
Confidentiality
Token of appreciation
Willingness to answer questions
Thank you
Real signature in contrasting ink
Implementation – Token Incentive
• Cash in advance $1. or $2. = 12% to 19%
increase in response rate
• Less of an increase for larger amount later
• Increasing incentive not cost effective
• Non dollar incentive not as effective and may
be effective because of packaging
Implementation – Contact 3
• Postcard to all who
were mailed a
questionnaire thanking
them
• Serves to jog memory
• Gives opportunity to
call for replacement if
needed
Mail one week after
questionnaire
Increase up to 13%
Implementation – Contact 4
• Replacement
questionnaire to those
who have not yet
responded.
• Letter reminding them
of the survey and
asking again for their
assistance.
Sent only to
non-respondents
Two to four
weeks after 3rd
contact
Implementation – Contact 5
• Special mailing to those
who have not yet
responded
• Use unique method of
mailing – certified,
priority etc
• May also telephone and
fill out over the phone
Final contact
is a week or
more after 4th
contact
Tracking Responses
• Assign each questionnaire a number
• Assign each name and address a number
• Send respondents their individually numbered
questionnaire
• Have assistant mark off numbers as they are
returned
• Note surveys are confidential not anonymous
• Keep completed questionnaires in secure
location
Mixed Mode Surveys
• Responses differ by mode:
“Excellent Health” = 44% by personal interview
= 37% by telephone
= 30% by mail
social desirability
acquiescence
primacy vs recency effect
Telephone Surveys
• Difficult to obtain phone numbers
• Caller id and answering machines may
screen calls
• Need well trained interviewers
• Limited number of questions and answers
Internet Surveys
• Consider coverage error. Does your entire
audience have and use the internet on a
regular basis?
• E-mail addresses change often
• Difficult to include incentive
• Shorter less detailed questionnaire
Survey Monkey
Research Ethics
Anonymous – No way to know who returned
the survey
Confidential – Only researcher can connect
the respondent with their answers. All
answers remain confidential.
Research Ethics
• If you wish to publish results you MUST
get approval from UCRIHS before you
send out surveys.
University Committee on Research
Involving Human Subject
References
Mail and Internet Surveys – The Tailored
Design Method by Don Dillman 2000