Collecting it Right the First Time:
Survey and Questionnaire
Design
Cynthia Wilson Garvan
PhD Statistics, MA Mathematics
College of Nursing
cgarvan@ufl.edu
Outline
• Horror stories
• Backward Design
• Using established instruments
• Maximizing information
• Bad questions
• Dillman’s principles
• Open ended questions
• Respondent burden
• Little things
2
Horror stories (some just funny)
• Back pain cancer study
• Student behavior intervention study
• Dental study
• McDonald’s survey
• Coding missing values as zeroes
3
What does “collecting it right” mean to you?
4
Sources of Bias
• Sample is not representative of population
• Non-coverage (some persons of interest have no chance of being interviewed)
• Non-response (segment of sample does not participate due to no contact or
refusal)
• Poorly worded questions
• Social desirability bias
• Interviewer bias
• Item non-response
• Poor analysis
“Learn from the
mistakes of others.
You can’t live long
enough to make
them all yourself."
~ Eleanor Roosevelt
6
Backward Design
What data do you need to collect?
http://www.consort-statement.org/
7
8
Table design for manuscript
9
Using established
instruments
There is an instrument to
measure everything!
Example of Google
search:
“how can i measure the
personality of my pet”
10
How to choose?
11
Maximizing information
http://biostat.mc.vanderbilt.edu/wiki/Main/ManuscriptChecklist
Statistical Problems to Document and to Avoid
12
13
Bad questions
HANDOUT: Find the bad questions!
14
Dillman’s principles
Dr. Dillman is recognized internationally as a major
contributor to the development of modern mail, telephone
and Internet survey methods. In 1970, he was founding
coordinator of the SESRC’s Public Opinion Laboratory
(1970-1973), one of the first university-based telephone
survey laboratories in the United States. His book, Mail
and Telephone Surveys: The Total Design Method (1978),
was the first to provide detailed procedures for conducting
surveys by these methods, and was recognized in 1990 by
the Institute for Scientific Information as a "Citation
Classic." It has been cited in more than 3,600 scientific
publications.
15
Dillman’s Principles to Create
Survey Items
Question writing principle 1
Choose simple over specialized words.
Poor example: Please indicate how many occupants
of this household operate a vehicle on a daily basis:
Improvement: From the people who live in this
house, how many drive a car every day?
Question writing principle 2
Choose as few words as possible to pose the question.
Poor example: Do you strongly agree, agree,
disagree or strongly disagree with standardized
testing being required for all elementary school
grades in the state of Florida?
Improvement: To what extent do you agree or
disagree with requiring standardized testing for all
elementary school grades?
Question writing principle 3
Use complete sentences to ask questions.
Poor example:
Years teaching science : ___________
Improvement:
For how many years have you taught science?
Number of years ___________
Question writing principle 4
Avoid vague quantifiers when more precise estimates can be obtained.
How often did you attend religious services during the past year?
Poor example:
Improvement:
 Never
 Not at all
 Rarely
 A few times
 Occasionally
 About once a month
 Regularly
 Two to three times a month
 About once a week
 More than once a week
Question writing principle 5
Avoid specificity that exceeds the respondent’s potential for
having an accurate, ready-made answer
Poor example:
Please indicate the number of times you visited a park during the
spring semester:
________ Number of visits to a park
Question writing principle 6
Use equal numbers of positive and negative categories for
scalar questions.
Poor example:
How satisfied are you with your graduate program?
Completely satisfied
Mostly satisfied
Somewhat satisfied
Neither satisfied nor dissatisfied
Dissatisfied
Question writing principle 7
Distinguish undecided from neutral by placement of
the Undecided or No opinion at the end of the scale
Example of adequate placement:
Strongly agree
Somewhat agree
Neither agree nor disagree
Somewhat disagree
Strongly disagree
Undecided
Question writing principle 8
Avoid bias from unequal comparison
• Poor example:
Which group plays a major role on student success on
standardized tests?
School
Family
Community
Well-trained teachers
Question writing principle 9
State both sides of attitude scales in the question items.
Poor example:
• To what extent do you agree with requiring Ph.D. students
to learn statistics?
Improvement:
• To what extent do you agree or disagree with requiring
Ph.D. students to learn statistics?
Question writing principle 10
Eliminate check-all-that apply question formats to reduce
primacy effects.
Poor example:
Please indicate which type of music you prefer to listen at
work (mark all that apply):
Blues
Rock
Jazz
Salsa
Hip hop
Country
Question writing principle 11
Develop response categories that are mutually exclusive.
Poor example:
From which one of these sources did you first learn about
the wild fires?
Radio
Television
Someone at work
While at home
While traveling to work
Question writing principle 12
Use cognitive design techniques to improve recall
Good example:
When was the last time you had an accident?
How serious was it?
How long did the emergency response system take to
send a person to assist you?
The researcher is most interested in the last question,
but uses the others to improve recall.
Question writing principle 13
Provide appropriate time referents. Can people accurately
recall and report past behaviors? Problems happen with
events that are either too far in the past or too regular.
Poor example:
How many times have you visited the library
during the Spring semester?
_____ Number of visits to the library
Question writing principle 14
Be sure questions are technically accurate.
Poor example: Please indicate whether you agree or disagree with the
UF requirement that students own cats.
 strongly agree
 agree
 disagree
 strongly disagree
Question writing principle 15
Choose question words that allows comparison with
previously collected data.
Poor example:
Time 1:
• How often do you read to your child?
Time 2:
• How often do you read stories to your child?
Question writing principle 16
Avoid asking respondents to say yes in order to mean no.
Poor example:
Do you favor or oppose not allowing the state to raise taxes
without approval of 60% of the voters?
 Favor
 Oppose
Question writing principle 17
Avoid double barreled questions.
Poor example:
Do you agree or disagree with creating an Office of
Statistical Support at UF and charging a fee to use it?
 Agree
 Disagree
Question writing principle 18
Soften the impact of potentially objectionable questions. Individuals may not
be willing to reveal the requested information.
Poor example:
Please indicate how frequently you have thoughts about leaving your
current graduate program?
 Daily
 At least once a week
 At least once a month
 At least once a semester
 Never
Question writing principle 19
Avoid asking respondents to make unnecessary calculations.
Poor example:
During the last academic year, what percentage of your income was
spent with mortgage payments?
Percentage: ________ %
Question writing principle 20
Make sure the question requires an answer.
Poor example:
When feeding your cat, do you also pet them?
 Yes
 No
Additional Resources
http://tobaccoeval.ucdavis.edu/documents/DillmanPrinciples.pdf
https://en.wikiversity.org/wiki/Survey_Design_for_Evaluation
http://www.sesrc.wsu.edu/dillman/coursedocument/Soc%20525%20%
202006%20writing%20questions.pdf
37
Open ended questions
A GOOD IDEA!
38
Respondent burden
A BIG PROBLEM!
39
Little things
Practice good ID hygiene:
•
•
•
•
Labels
ID numbers each page
Use same ID different time points
Don’t let participants make up their own ID
Check with others on your survey/questionnaire!
40