EMR 6500:
Survey Research
Dr. Chris L. S. Coryn
Lyssa N. Wilson
Spring 2015
Agenda
• Constructing open- and closed-ended
items
• Case Study #3
Open-Ended Questions
Open-Ended Requests for
Numerical Responses
• General guidelines
1. Ask for the specific unit desired in the
question stem
2. Provide answer spaces that are sized
appropriately for the response task
3. Provide units labels with the answer
spaces
Open-Ended Numeric
A poorly constructed open-ended number box question
In an average week, how often do you cook dinner at home?
A revision that includes the unit desired in the question stem
In an average week, how many days do you cook dinner at home?
A revision that provides an appropriate sized answer space for the response task
In an average week, how often do you cook dinner at home?
A revision that includes unit labels with the answer space
In an average week, how often do you cook dinner at home?
Days per week (0 – 7)
Open-Ended Requests for a
List of Items
• General guidelines
1. Specify the number and type of
responses desired in the question stem
2. Design the answer spaces to support
the number and type of responses
desired
3. Provide labels with answer spaces to
reinforce the type of response
requested
Open-Ended Lists
A list-style question as commonly designed
What businesses would you most like to see in the Kalamazoo area that are currently not available?
A list-style question with multiple answer boxes
What businesses would you most like to see in the Kalamazoo area that are currently not available?
A list-style question with labeled answer boxes
What businesses would you most like to see in the Kalamazoo area that are currently not available?
Business #1
Business #2
Business #3
Open-Ended Requests for
Description and Elaboration
• General guidelines
1. Provide extra motivation to respond
2. Provide adequate space for
respondents to completely answer the
question
3. Use scrollable boxes on internet
surveys
4. Consider programming probes to openended responses in internet surveys
Open-Ended Description and
Elaboration
A poorly constructed descriptive open-ended question
Why did you move to Kalamazoo?
A revision that provides extra motivation to respond
Your answer to this question is very important for understanding what brings people to Kalamazoo. Why
did you choose to move to Kalamazoo?
Closed-Ended Questions
General Guidelines
1. State both positive and negative sides
in the question stem when asking
either/or types of questions
2. Develop lists of answer categories that
include all reasonable possible answers
3. Develop lists of answer categories that
are mutually excusive
4. Maintain spacing between answer
categories that is consistent with
measurement intent
Positive and Negative Sides in
Question Stem
Exhaustive and Mutually
Exclusive Questions
A question that is not exhaustive or mutually exclusive
From which one of these sources did you first learn about the tornado in Kalamazoo?
Radio
Television
Someone at work
While at home
While traveling to work
A revision that is exhaustive and mutually exclusive
From which one of these sources did you first learn about the tornado in Kalamazoo?
Radio
Television
Internet
Newspaper
Another person
Where did you first hear about it?
At work
At home
Traveling to work
Somewhere else
Spacing Response Options
Evenly
Closed-Ended Questions:
Nominal Scales
• General guidelines
1. Ask respondents to rank only a few items
at once rather than a long list
2. Avoid bias from unequal comparisons
3. Randomize response options if there is
concern about order effects
4. Use forced-choice questions rather than
check-all-that-apply questions
5. Consider using differently shaped answer
spaces (circles and squares) to help
respondents distinguish between singleand multiple-answer questions
Closed-Ended Unordered
Most simple form: Two categories
While growing up did you live mostly on a farm or mostly elsewhere?
Farm
Elsewhere
A more difficult form: Multiple complex categories
If the highway bypass were to built on one of these routes, which
would you most prefer?
A north route that starts west of the city at Exit 21 (Johnson Road) off Highway
30, crosses Division at North 59th Street, and reconnects to Highway 30 three
miles east of the city at River Road.
A modified north route that starts further west of the city at Exit 19, crosses
Division at 70th Street, and reconnects to Highway 30 three miles east of the
city at River Road.
A south route that begins west of the city at Exit 19, crosses Division at South
24th Street, and reconnects to Highway 30 east of the city at River Road.
Another difficult form: The ranking question
Which of these do you believe are the largest and smallest problems
facing residents of the Kalamazoo area? Use “1” for the largest
problem, “2” for the second largest problem, and so forth until you
have completed all eight.
Lack of community involvement
Taxes are too high
Lack of affordable health care
Lack of money for local schools
Lack of affordable housing
Lack of good jobs
Too much crime overall
Too much drug use
Ranking: Pairwise Comparisons
• In ranking questions, it is typically
better to present respondents with
sets of paired comparisons so that
they are only comparing two
concepts at a time until all options
have been compared to one another
Bias from Unequal
Comparisons
Unequal comparison
Which of the following do you feel is most responsible for recent
outbreaks of violence in America’s schools?
Irresponsible parents
School policies
Television programs
A revision with more neutral response options
Which of the following do you feel is most responsible for recent
outbreaks of violence in America’s schools?
The way children a raised by parents
School policies
Television programs
A simplified revision
Which of the following do you feel is most responsible for recent
outbreaks of violence in America’s schools?
Parents
Schools
Television
A revision retaining more complex descriptions
To what extent do you feel that the way children are raised by parents
is responsible for recent outbreaks of violence in America’s schools?
Completely responsible
Mostly responsible
Somewhat responsible
Not at all responsible
Check-all-that-Apply versus
Forced-Choice
Check-all-that-apply formatted question
Which of the following varsity sport would you consider yourself to be
a fan of? Check all that apply.
Men’s baseball
Men’s basketball
Women’s basketball
Women’s cross-country
Men’s gold
Women’s golf
A revision converting to the forced-choice format
Do you consider yourself to be a fan of each of the following varsity
sports?
Yes No
Men’s baseball
Men’s basketball
Women’s basketball
Women’s cross-country
Men’s gold
Women’s golf
Distinguishing Between Singleand Multiple-Answer Questions
Little distinction between single- and multiple-answer questions
Of the following considerations, which is
the most important to you when purchasing
a new car?
Of the following considerations, which three
are most important to you when purchasing
a new car?
Price
Price
Color
Color
Exterior condition
Exterior condition
Interior condition
Interior condition
Amenities
Amenities
Answer space designed to help differentiate between single- and multiple-answer questions
Of the following considerations, which is
the most important to you when purchasing
a new car?
Of the following considerations, which three
are most important to you when purchasing
a new car?
Price
Price
Color
Color
Exterior condition
Exterior condition
Interior condition
Interior condition
Amenities
Amenities
Closed-Ended Questions:
Ordinal Scales
• General guidelines
1.
2.
3.
4.
5.
6.
7.
8.
Choose an appropriate scale length—in general, limit scales
to four or five categories
Choose direct or construct-specific labels to improve
cognition
Provide scales that approximate the actual distribution of the
characteristic in the population
Provide balances scales where categories are relatively equal
distances apart conceptually
Consider how verbally labeling and visually displaying all
response categories may influence answers
Carefully evaluate the use of numeric labels and their impact
on measurement
Align response options vertically in one column or
horizontally in one row and strive for equal distance between
categories
Place nonsubstantive options at the end of the scale and
separate them from substantive options
Polarity in Ordinal Scales
• Unipolar ordinal scales measure
gradation along one dimension
where the zero point falls at one end
of the scale
• Bipolar ordinal scales measure
gradation along two opposite
dimensions with the zero point falling
in the middle of the scale
– Level (e.g., satisfaction, importance)
– Direction (i.e., positive, negative)
Scalar Questions
A 5-point bipolar scale
A 7-point bipolar scale
How likely or unlikely are you to make a
monetary donation to the Red Cross this
year?
How likely or unlikely are you to make a
monetary donation to the Red Cross this
year?
Very likely
Very likely
Somewhat likely
Somewhat likely
Neither likely nor unlikely
Slightly likely
Somewhat unlikely
Neither likely nor unlikely
Very unlikely
Slightly unlikely
Somewhat unlikely
Very unlikely
A 4-point unipolar scale
A 5-point unipolar scale
How important do you feel it is to volunteer
your time with organizations like the Red
Cross?
How successful do you feel the Red Cross
has been at getting assistance to natural
disaster victim?
Very important
Completely successful
Somewhat important
Very successful
Slightly important
Somewhat successful
Not at all important
Slightly successful
Not at all successful
Construct-Specific Scales
Question with multiple constructs
Construct-specific questions
Construct of interest: Accessibility of instructors
To what extent do you agree or disagree
that your instructors are available outside
of class?
How accessible or inaccessible are your
instructors outside of class
Strongly agree
Very accessible
Agree
Somewhat accessible
Neutral
Neutral
Disagree
Somewhat inaccessible
Strongly disagree
Very inaccessible
Construct of interest: Frequency of attending religious services
How often did you attend religious services
during the past year?
How often did you attend religious services
during the past year?
Never
Not at all
Rarely
A few times during the year
Occasionally
About once a month
Regularly
Two to three times a month
About once a week
More than once a week
Balanced Scales with Even
Distance Between Categories
Unbalanced scale with uneven distance between
categories
To what extent do you favor or oppose the
death penalty?
Balanced scale with more even distance between
categories
To what extent do you favor or oppose the
death penalty?
Strongly favor
Strongly favor
Slightly favor
Somewhat favor
Neither favor nor oppose
Neither favor nor oppose
Slightly oppose
Somewhat oppose
Mostly oppose
Strongly oppose
Entirely oppose
Fully Labeled and Point-Polar
Labeled Scales
A fully labeled scale with verbal labels for all of the categories
How satisfied are you with your decision to attend WMU?
Very satisfied
Somewhat satisfied
Neutral
Somewhat dissatisfied
Very dissatisfied
A polar-point labeled scale with verbal labels only for the endpoints
How satisfied are you with your decision to attend WMU?
5 Very satisfied
4
3
2
1 Very dissatisfied
A polar-point question with no visual display of the scale (as it might be read over the telephone
On a 5-point scale, where 5 means very satisfied and 1
very dissatisfied, how satisfied are you with your decision
to attend WMU? You may use any of the numbers 5, 4, 3,
2, or 1 for your response.
Alignment of Response Options
Primacy and Recency
• Primacy occurs when respondents
are more likely to select the first
option when a scale is presented
visually
• Recency effects occur when
respondents are more likely to select
the last option when a scale is
presented aurally
Aligning Conceptual and Visual
Midpoints
Conceptual and visual midpoints are properly aligned
To what extent do you approve or disapprove of the
way Democrats in Congress are handling their jobs?
Very much approve
Somewhat approve
Visual
midpoint
Neither approve nor disapprove
Conceptual midpoint
Somewhat disapprove
Very much disapprove
Adding a “Don’t know” category misaligns the conceptual and visual midpoints
To what extent do you approve or disapprove of the
way Democrats in Congress are handling their jobs?
Very much approve
Somewhat approve
Neither approve nor disapprove
Visual
midpoint
Conceptual midpoint
Somewhat disapprove
Very much disapprove
Don’t know
No opinion
Visually separating the “Don’t know” category realigns the conceptual and visual midpoints
To what extent do you approve or disapprove of the
way Democrats in Congress are handling their jobs?
Very much approve
Somewhat approve
Visual
midpoint
Neither approve nor disapprove
Somewhat disapprove
Very much disapprove
Don’t know
No opinion
Conceptual midpoint
Other Types of Closed-Ended
Scales
Method of Equal-Appearing
Intervals
• One of Thurstone’s methods of scaling
1.
2.
3.
4.
5.
6.
7.
8.
Determine attitude object to be measured
Construct a set of statements about the attitude
toward the object that captures an entire range of
opinions, from extremely favorable, to neutral, to
extremely unfavorable
Typically requires 40-50 statements
Judges rate favorability values for statements using an
11-point scale
Statements are sorted into 11 physical piles (Q-sort)
Measures of central tendency are used to construct
values for each statement
The final scale is constructed by randomly organizing
final statements into a checklist
Scale scores are calculated by taking the mean of
endorsed statements
Thurstone Scale
Check (ü) the statements with which you agree.
1. Abortion brings happiness to some, unhappiness to others (5.1)
2. Abortion should be encouraged for all unwanted pregnancies (10.4)
3. Abortion prevents an unwanted child growing up in an orphanage (8.0)
4. Abortion is allowable if there is no financial cost to the public (6.0)
5. Abortion weakens the moral fiber of our society (1.6)
6. We must allow abortions for certain situations (8.7)
7. Abortion is never justified (1.1)
8. It does not make any difference to me whether we allow abortions or not (5.6)
9. Most people are too lax about allowing abortions (3.2)
10. People who get abortions deserve our understanding, but not special treatment (7.6)
Semantic Differential
• Osgood’s method of scaling based on
connotative meaning of words, which refers to
the associations of meaning that are attached
to a word that are not part of its formal
definition (denotative meaning)
• Opposing adjective pairs are used to assess
three underlying dimensions of connotative
meaning
1. Evaluation, which reflects the good-bad
continuum of meaning underlying words (e.g.,
good-bad, valuable-worthless)
2. Potency, which reflects the strong-weak
continuum (e.g., large-small, strong-weak)
3. Activity, which reflects the active-passive
continuum (e.g., fast-slow, hot-cold)
Semantic Differential Response
Formats
Check (ü) the position between each adjective pair that best describes the
meaning of abortion to you.
Bad
Good
Bad
1
2
3
4
5
6
7
Good
Bad
-3
-2
-1
0
+1
+2
+3
Good
Bad
Good
Case Study #3
Case Study Activity
1. Construct three questions of any type
that reflect the concept/ construct of
‘course quality’
2. Present your questions and response
options to the class and discuss how
are your questions ‘better’ than the
standardized questions
–
The group with the ‘best’ questions will
receive 5 extra credit points, the second
best will receive 4 extra credit points, and
so forth