Module 7
Erika Trapl, PhD
Two-Part Survey Series
 Module 7
 Matching survey items to research questions
 Question Selection/Development
 Response categories
 Survey design and formatting
 Module 8
 Taking your survey “live”
 Data collection protocols
Five Stages of Survey Development
and Completion
 Initial survey design and preliminary planning
 Pre-testing
 Final survey design and planning
 Data collection
 Data coding, data-file construction, analysis, final
report
Five Stages of Survey Development
and Completion
 Initial survey design and preliminary planning
 Pre-testing
 Final survey design and planning
 Data collection
 Data coding, data-file construction, analysis, final
report
Survey Design and Preliminary
Planning
What are the goals of the survey?
 To test a hypothesis (or a hypothetical model of
interrelated hypotheses)
 eg, Females are more likely than males to believe that a
woman should be permitted to have an abortion for any
reason.
 To estimate the proportion of people who hold a
certain belief or engage in a specific behavior
 eg, What proportion of our population believe that our
criminal justice system is working well?
Initial Design of the Questionnaire
 What types of information do you need to collect from
participants?
 How can you best elicit that information?
 Open-ended questions
 eg, “What is the most difficult problem facing families
today?”
 Best for face to face rather than mailed surveys
 Can provide lists of choices, visual aids, participant can
consult records
 Closed-ended questions
 eg, “Do you approve or disapprove of abortion?”
 Respondents choose from a list of provided responses
Final Survey Design and Planning
• Develop a Study Timeline
Activity
No. Weeks
Review literature and draft questionnaire
8
Assemble think alouds with 5 respondents
1-2
Revise questionnaire
2
Conduct pretest (n=25-30)
2
Debrief interviewers and revise questionnaire
3
Pretest #2 (n=20-25)
2
Debrief interviewers, revise questionnaire, develop
training materials
4
Select samples
4
Conduct main data collection
8
Code data & prepare data files
10
Analyze data and write report
16
Drafting the Questionnaire
 Borrowing questions from other research studies is
acceptable
 Enables comparison of your study results to previous
studies
 Note how researchers in past studies defined concepts


Note the questions they used to measure these concepts
Free to use the same wording for items unless copyrighted
 Caveat: don’t assume that the questions are problem-
free just because someone else developed and used
them
Drafting the Questionnaire
 Must pre-test questionnaires even if borrowed most
items from prior studies
 Expect to do many drafts of the questionnaire to work
out all of the problems
Guiding Questions
…or, What do you want to know?
 The first step: write a series of guiding questions or
objectives that clearly indicate what you are trying to
find out through the survey
 Examples as questions:
 How do occupational therapists incorporate play into their
practice?
 Do occupational therapists assess play behaviors and what
methods do they use?
 Are there differences between school-based and non-schoolbased settings that influence the role of play within pediatric
occupational therapy?
Guiding Questions
 Hypotheses may be used to examine relationships
 Examples as hypotheses:


Men will rate themselves higher in career success compared
with women
Men will report greater importance of salary and position
compared with women
Developing Survey Questions
 Operationalize the Research Questions into survey
questions/items
 Be concrete
 Specify exactly what information you want from
respondents
 Consider all possibilities of interpretation and response
How is a question answered?
 For each question, the respondent must:
 Comprehend the question (interpret the subject and the
task)
 Recall information, form judgment
 Decide on answer
 Record answer on a form or speak the answer to an
interviewer
 Anticipate potential problems at each step for each
question
Developing Survey Questions
 “How many college mathematics courses have you taken?”
Possible interpretations:
1. Courses being taken now but not yet completed
2. College-level math courses taken in high school
3. High school level remedial math courses taken in college
4. Math courses taken by transfer students at their previous
college
5. A math course taken but failed last semester
6. A course that primarily involves math but is given by another
department or is not listed as a math course
7. An audited math course not taken for credit
8. A math course taken on a pass-fail option
9. College level math courses taken and completed
Developing Survey Questions
 If the choice of what to count is left to respondents,
different people will include different things,
introducing measurement error into the data
 For the math question, it may be more useful to break
up the question into a number of questions that ask
about the different types of courses (ie, regular, audit,
pass-fail)
 But if and how you do this depends on your Research
Question(s)!
Developing Survey Questions
 Not every possible question can be included in a
survey
 But you should include all the questions you need to
answer your Research Question
 Each question has a ‘cost’ and must be justified
So, how do you decide what’s in and what’s out?
Choosing Survey Questions
Decision Guide
A. Does the question measure some aspect of the Research Questions?
B. Does the question provide information needed in conjunction with
some other variable? (If no to both A & B, drop the question. If yes to
one or both, proceed).
C. Will most respondents understand the question in the same way? (If
no, revise or drop. If yes, proceed).
D. Will most respondents have the information to answer it? (If no,
drop. If yes, proceed)
E. Will most respondents be willing to answer it? (If no, drop. If yes,
proceed)
F. Is other information needed to analyze this question? (If no proceed.
If yes, proceed if the other information is readily available or can be
obtained from the survey)
G. Should this question be asked of all respondents or of a subset? (If
all, proceed. If a subset, proceed only if the subset is identifiable
beforehand or through questions in the questionnaire/interview)
Choosing Good Survey Questions
 The questions should pose tasks that the respondent can




do well (ie, recall information, make judgments)
In wording the question, do not influence the respondent’s
answer in one direction or another
Questions should be uncomplicated, explicit, and naturalsounding
Avoid ambiguity
Avoid double-barreled questions
 eg, Do you think the police and the courts are doing an
excellent, good, fair, or poor job?
 Avoid ‘agree/disagree’ response format
 Acquiescence response set
Structure of Survey Questions
 Two main categories of survey questions: open
questions & closed questions
 Responses to open questions are narratives that must
be interpreted and coded by the investigators
 Recommend using closed questions as much as
possible
Structure of Survey Questions
 Closed questions have 2 parts
 Statement of the question (aka ‘item’)
 Response categories (aka ‘options’)
 The question should be stated as directly as possible
 The answer must be given in terms of only one of the
answer choices provided (typically)
 Qualifiers and conditions are commonly used in
questions to limit the question to match the research
objectives
Example Qualifiers
 How many times have you shopped at the Book Center
since January?
 In the past year, did anything happen to you or your
property that you thought was a crime or an attempt at
a crime?
 In the past year, would you say that the violent crime
situation in your neighborhood has gotten better,
gotten worse, or stayed about the same?
Closed Ended Response Formats
 Responses should be exhaustive; including all possible
responses
 Protections to assure this include ‘NA,’ ‘Don’t know,’ or
‘Other (please specify) _________’
 If interested in more than one answer to a question,
offer the ‘mark all that apply’ option
 Each response should be mutually exclusive; each
choice should clearly represent a unique answer
 Order responses according to an inherent hierarchy if
one exists (eg, increasing or decreasing intensity)
Response options: NOIR
 NOMINAL: Mutually exclusive, but not ordered, categories (gender)
 ORDINAL: Order matters but not the difference between values.
 For example, you might ask patients to express the amount of pain they
are feeling on a scale of 1 to 10. A score of 7 means more pain that a
score of 5, and that is more than a score of 3. But the difference between
the 7 and the 5 may not be the same as that between 5 and 3. The values
simply express an order.
 INTERVAL: Measurement where the difference between two values is
meaningful.
 The difference between a temperature of 100 degrees and 90 degrees is
the same difference as between 90 degrees and 80 degrees.
 RATIO: has all the properties of an interval variable, and also has a
clear definition of 0.0. When the variable equals 0.0, there is none of
that variable.
 Variables like height and weight are ratio variables.
Response options have implications for data analysis and interpretation.
Categorical Response Formats
What do you like most about your job?
___ Interaction with clients
___ Interaction with other staff
___ Intellectual challenge
___ Opportunities for professional growth
 Disadvantages of this type of response set





Does not allow respondents to express their personal viewpoints
May over look some important responses
May bias answers by presenting a particular attitude
Categorical - Nominal scale has implications for analysis
Almost like four different yes/no items
Categorical Response Formats
What race or ethnicity do you most identify with?
___ Asian
___ American Indian or Alaskan Native
___ Black or African American
___ White or Caucasian
___ Native Hawaiian or Pacific Islander
___ I identify with more than one
___ I don’t identify with any of these
In this example, we want respondents to ‘categorize’ themselves.
Continuous Response Formats
 Characteristics on a continuum, such as attitudes or quality
of performance
 A range of responses allows the respondent to select the
choice representing the intensity of their response
 Example:
 Very important
 Important
 Somewhat important
 Not important
 Unsure
 Frequently used
 Continuous scale (ordinal, interval or ratio)
Common Response Categories
Opinions
Knowledge
Frequency of
Events
Ratings
Satisfied /
Somewhat satisfied
/ Somewhat
dissatisfied / Very
dissatisfied
Very Familiar /
Somewhat
familiar/ Not
too familiar/
Not at all
familiar
Per day/Per
week/Per
month/Per
year/Never
Excellent/ Good / Fair/
Poor
Important…
True/False
Always/Frequently
/ seldom/never
High / Medium /Low
Oppose/Support
A lot/Some/ A
little/ None
Always/Sometimes Gotten better / Gotten
/ Never
worse/ Stayed about the
same
Strongly oppose/
Oppose/ Support/
Strongly support
All/Most/Some / A Very fair/ Fair / Unfair /
few / None
Very unfair
Often/
Sometimes/ Rarely
/ Never
Measurement Scales
 A scale is an ordered system of questions (a set of
items) that provides an overall rating representing an
attitude, value, or characteristic
 Measures one dimension or characteristic
 Each item on the scale carries the same weight or point
value
 Results in a calculated summary score
 Examples: Stress, Depression, Parental monitoring
Types of Scales
 Likert
 Semantic differential
 Visual analogue
 Guttman
Likert Scale
 Used to assess an attitude or a value
 Originally a 5-point response category format
 No consensus on the number of response categories that
should be used

Should there be a neutral category?
 Example
 Strongly agree
 Agree
 Neutral
 Disagree
 Strongly disagree
Likert Scale Development
 A large pool of items should be developed that reflect
an equal number of favorable and unfavorable
attitudes
 Do not include items that are intended to elicit neutral
responses
 Scale should be validated through item analysis
 Those who score the highest should agree with positively
worded statements, those who score the lowest should
disagree
Semantic Differential
 Measures attitudes or feelings about a concept on a
continuum that extends between the two extremes
 Typically a 7-point scale
 Only extremes are labeled
 Tends to measure evaluation, potency, and/or activity
Good ____ ____ ____ ____ ____ ____ ____Bad
Visual Analogue Scale
No pain
Pain as bad
_______________________________ as it can be
 Measures intensity of a subjective experience
 A 100 mm line is typically drawn
 Word anchors on either end represent extremes of a
characteristic
 Respondent places a mark along the line to indicate
their perceived level for that characteristic
 Scored by measuring in millimeters
 Often treated as ratio level data
Guttman Scale
 Addresses problematic issue with Likert scale
 different individuals with the same summary score on
Likert scale may have marked different responses
 Guttman cumulative scale ensures that there is only
one unique combination of responses that can achieve
a particular score
Guttman Scale
Example – Bogardus Social Distance Scale
Are you willing to permit immigrants to live in your
country? Yes No
2. Are you willing to permit immigrants to live in your
community? Yes No
3. Are you willing to permit immigrants to live in your
neighborhood? Yes No
4. Are you willing to permit immigrants to live next door to
you? Yes No
5. Would you permit your child to marry an immigrant?
Yes No
1.
Putting it together…
 You have all of your questions and responses
selected—what’s next?
Organization of the Questionnaire
 Group according to categories or topics
 Questions should proceed from general to specific
 Initial questions should pique the respondents interest
or at least be neutral
 Sensitive questions should come later
 Some put demographic questions at the beginning,
others at the end because they are less
interesting/engaging for the respondent
Organization of the Questionnaire
 Make it easy for the respondent to “navigate” the
survey
 Use headers and simple instructions
 Page should be uncluttered in appearance
 Laser quality print
 Clear alignment of questions/responses
 Font of 11 or 12 point
 White or light-colored paper
Sources
Czaja R, Blair J. 1996. Designing Surveys: a guide to
decisions and procedures. Thousand Oaks, CA: Sage.
Portney LG, Watkins MP. 2000. Foundations of Clinical
Research, 2nd Ed. Upper Saddle River, NJ: Prentice Hall.