Response Biases
in Survey Research
Hans Baumgartner
Smeal Professor of Marketing
Smeal College of Business, Penn State University
Response biases in survey research
Response biases
 when a researcher conducts a survey, the expectation is that
the information collected will be a faithful representation of
reality;
 unfortunately, this is often not the case, and survey
researchers have identified many different sources of error in
surveys;
 these errors may contaminate the research results and limit
the managerial usefulness of the findings;
 if the response provided by a respondent does not fully reflect
the “true” response, a response (or measurement) error
occurs (random or systematic);
 response biases (systematic response errors) can happen at
any of the four stages of the response process, are elicited by
different aspects of the survey, and are due to a variety of
psychological mechanisms;
Response biases in survey research
The relationship between observed
measurements and constructs of interest
 The total variability of observed
scores consists of trait
(substantive), random error, and
systematic error (method)
variance.
T
M
E
T1
T2
E2
E1
M1
M2
 Random and systematic errors
are likely to confound relationships between measures and
constructs and between
different constructs.
 They also complicate the
comparison of means.
Response biases in survey research
Outline of the talk
 Misresponse to reversed and negated items
□
□
□
Item reversal and negation, types of misresponse,
and mechanisms
Reversed item bias: An integrative model
Eye tracking of survey responding to reversed and
negated items
 Item grouping and discriminant validity
 Extreme and midpoint responding as satisficing
strategies in online surveys
 Stylistic response tendencies over the course of
a survey
Response biases in survey research
The issue of item reversal
 Should reverse-keyed items (also called oppositelykeyed, reversed-polarity, reverse-worded, negatively
worded, negatively-keyed, keyed-false, or simply
reversed items) be included in multi-item summative
scales?
 If reversed items are to be used, does it matter
whether the reversal is achieved through negation or
through other means?
 What’s the link between reversal and negation, what
types of MR result, what psychological mechanisms
are involved, and how can MR be avoided?
Response biases in survey research
Item reversal vs. item negation
 authors often fail to draw a clear distinction between
reversals and negations and use ambiguous terms such
as ‘negatively worded items’, which makes it unclear
whether they refer to reversed or negated items, or both;
 examples from the Material Values scale (Richins and
Dawson 1992):
□
□
□
It sometimes bothers me quite a bit that I can't afford to buy all
the things I’d like.
I have all the things I really need to enjoy life.
I wouldn't be any happier if I owned nicer things.
Response biases in survey research
Item negation
 items can be stated either as an assertion (affirmation) or
as a denial (disaffirmation) of something (Horn 1989);
 negation is a grammatical issue;
 classification of negations in terms of two dimensions:
□
what part of speech is negated (how a word is used in a
sentence: as a verb, noun/pronoun, adjective, adverb or
preposition/conjunction);
□
how the negation is achieved (by means of particle negation,
the addition of no, the use of negative affixes, negative adjectives
and adverbs, negative pronouns, or negative prepositions);
Negated by means of
Not, n’t
No
negative affixes
negative
adjectives
and adverbs
negative
pronouns
negative
prepositions
[This salesperson
does not make
false claims.]
n.a.
dislike, disagree,
etc.
[I dislike food
shopping very
much.]
reluctant,
hesitant, never,
rarely, seldom,
hardly (ever),
less, little, etc.
[I seldom
daydream.]
n.a.
without, instead
of, rather than,
etc.
[This supplier
sometimes
promises to do
things without
actually doing
them later.]
5 (1.7%)
26 (8.6%)
discomfort,
disagreement,
etc.
[There is
considerable
disagreement as
to the future
directions that
this hospital
should take.]
little, few, a lack
of, none of the,
not much, neither
of, etc.
[Many times I
feel that I have
little influence
over things that
happen to me.]
Part of speech
Verb
135 (44.7%)
Noun/
pronoun
Adjective
not everyone, not
no object, no
(only), etc.
reason, no
[I and my family
purpose, etc.
will consume
[Clipping,
only certain
organizing, and
brands, not
using coupons is
others.]
no fun.]
5 (1.7%)
Total
171 (56.6%)
no-one, nobody,
except for,
none, nothing, without, with the
etc.
exception of,
[Energy is really instead of, rather
not my problem
than, etc.
because there is [American people
simply nothing I
should always
can do about it.]
buy Americanmade products
instead of
imports.]
4 (1.3%)
17 (5.6%)
5 (1.7%)
10 (3.3%)
4 (1.3%)
14 (4.6%)
n.a.
n.a.
uninterested,
dishonest, etc.
[Most charitable
organizations are
dishonest.]
rarely, less, etc.
[I would be less
loyal to this rep
firm, if my
salesperson
moved to a new
firm.]
n.a.
n.a.
55 (18.2%)
4 (1.3%)
54 (17.9%)
59 (19.5%)
Negated by means of
Not, n’t
No
negative affixes
Part of speech
Adverb
Preposition/
Conjunction
negative
adjectives
and adverbs
not much, etc.
no longer, etc.
[I often dress
rarely, less, etc.
[After I meet
[Hard work is no unconventionally [I feel I have to
someone for the longer essential
even when it's
do things hastily
first time, I can
for the welllikely to offend
and maybe less
usually
being of society.]
others.]
carefully in order
remember what
to get everything
they look like,
done.]
but not much
about them.
2 (.7%)
1 (.3%)
3 (1.0%)
1 (.3%)
not for, not (just)
in, not (only) as,
not until, not if
(incl. unless), not
because, etc.
[I enjoyed this
shopping trip for
its own sake, not
just for the items
I may have
purchased.]
n.a.
n.a.
n.a.
10 (3.4%)
Total
151 (50.0%)
negative
pronouns
negative
prepositions
n.a.
n.a.
7 (2.3%)
for nothing, etc.
[I don't believe
in giving
anything away
for nothing.]
n.a.
11 (3.7%)
1 (.3%)
18 (6.0%)
68 (22.5%)
41 (13.5%)
Total
5 (1.7%)
19 (6.3%)
302
Response biases in survey research
Item reversal
 an item is reversed if its meaning is opposite to a
relevant standard of comparison (semantic issue);
 three senses of reversal:
□
□
reversal relative to the polarity of the construct being
measured;
reversed relative to other items measuring the same
construct:


□
reversal relative to the first item
reversal relative to the majority of the items
reversal relative to a respondent’s true position on the
issue under consideration (Swain et al. 2008);
Response biases in survey research
Integrating item negation and item reversal
Item reversal
Non-reversed
Reversed
Regular (RG)
items
Polar opposite
(PO) items
Talkative, enjoying
talking to people
Quiet, preferring to do
things by oneself
Negated polar
opposite (nPO) items
Negated regular
(nRG) items
Not quiet, preferring
not to be by oneself
Not talkative, not
getting much pleasure
chatting with people
Non-negated
Item negation
Negated
Response biases in survey research
Misresponse to negated and reversed items
MR → within-participant inconsistency in response to multiple
items intended to measure the same construct;
Consistent
responding
Misresponse to
negated items
(NMR)
Misresponse to
reversed items
(RMR)
Misresponse to
polar opposites
(POMR)
Talkative (RG)
A
A
A
A
Not talkative (nRG)
D
A
A
D
Quiet (PO)
D
D
A
A
Not quiet (nPO)
A
D
A
D
Item
Response biases in survey research
Using reversed and negated items in
surveys: Some recommendations
 although responding to reversed items is error
prone, wording all questions in one direction does
not solve the problem;
 negations should be employed sparingly, esp. if they
do not result in an item reversal (note: negations
come in many guises);
 polar opposite reversals can be beneficial (esp. at
the retrieval stage), but they have to be used with
care;
Response biases in survey research
An integrative model of reversed item bias:
Weijters, Baumgartner, and Schillewaert (2012)
 two important method effects:
□
response inconsistency between regular and
reversed items;
□ difference in mean response depending on whether
the first item measuring the focal construct is a
regular or reversed item;
 three sources of reversed item method bias:
□
□
□
acquiescence
careless responding
confirmation bias
Response biases in survey research
The survey response process
(Tourangeau et al. 2000)
Comprehension
Attending to and interpreting survey
questions (careless responding)
Retrieval
Generating a retrieval strategy and
retrieving relevant beliefs from
memory (confirmation bias)
Judgment
Integrating the information into a
judgment
Response
Mapping the judgment onto the
response scale and answering the
question (acquiescence)
Response biases in survey research
Empirical studies
 (net) acquiescence and carelessness explicitly measured;
 confirmation bias modeled via a manipulation of two item
orders in the questionnaire, depending on the keying
direction of the first item measuring the target construct;
 three item arrangements:
□
grouped-alternated condition (related items are grouped
together and regular and reverse-keyed items are alternated);
□ grouped-massed condition (items are grouped together, but
the reverse-keyed items follow a block of regular items, or vice
versa;
□ dispersed condition (items are spread throughout the
questionnaire, with unrelated buffer items spaced between the
target items);
Response biases in survey research
Results for Study 2
 both NARS (gNARS = .33, p < .001) and IMC (gIMC = .31, p
< .001) were highly significant determinants of
inconsistency bias;
 the effect of NARS on inconsistency bias was invariant
across item arrangement conditions, as expected;
 the effect of IMC did not differ by item arrangement
condition;
 the manipulation of whether or not the first target item
was reversed (FIR) did not affect responses (although in
the first study the effect was significantly negative);
 the effect of FIR did not differ by item arrangement
condition;
Response biases in survey research
Eye tracking of survey responding
(with Weijters and Pieters)
 eye-tracking data may provide more detailed
insights into how respondents process survey
questions and arrive at an answer;
 eye movements can be recorded unobtrusively, and
eye fixations show what respondents attend to while
completing a survey;
Response biases in survey research
Eye tracking study
 101 respondents completed a Qualtrics survey and
their eye movements were tracked; effective sample
size is N=90;
 Design:
□
□
each participant completed 16 four-item scales shown
in a random sequence;
the fourth (target) item on each screen was an RG,
nRG, PO, or nPO item (4 scales each);
Response biases in survey research
Areas of interest
AOI1a to AOI1e
AOI3a
AOI2a
AOI3b
AOI2b
AOI2c
AOI3c
AOI2d
AOI3d
AOI5b
AOI5a
AOI4a
AOI4b
Response biases in survey research
Fixation durations for various AOI’s
Mean fixation duration
aoi1
aoi1a
aoi1b
aoi1c
aoi1d
aoi1e
Aoi2
aoi2a
aoi2b
aoi2c
aoi2d
aoi5a
aoi5b (for nRG and nPO)
aoi3
aoi3a
aoi3b
aoi3c
aoi3d
aoi4
other areas
Total
1.05
0.14
0.24
0.38
0.21
0.08
15.45
3.75
3.39
4.11
4.20
0.61
0.39
4.88
1.40
1.12
1.15
1.20
0.43
2.20
24.01
Percentage of total
fixation duration
0.050
0.006
0.011
0.016
0.010
0.004
0.624
0.153
0.140
0.162
0.169
0.025
0.016
0.212
0.061
0.048
0.049
0.053
0.020
0.099
1.000
Response biases in survey research
Determinants of total fixation duration
for fourth item (logaoi23dplus1)
Solutions for Fixed Effects
Effect
Estimate
S. E.
t Value
Pr > |t|
Intercept
0.6203
0.1601
3.87
0.0015
Survey completion time
0.0013
0.0002
8.66
<.0001
Screen sequence
-0.0052
0.0020
-2.65
0.0082
Number of words
0.0387
0.0112
3.46
0.0006
Reversal of item
0.0411
0.0185
2.23
0.0261
Negation of item
0.1249
0.0202
6.20
<.0001
Use of a PO in the item
0.0605
0.0188
3.22
0.0013
Note: These results are based on a mixed model with respondent and construct as random
effects.
Response biases in survey research
Determinants of misresponse
Solutions for Fixed Effects
Effect
Estimate
S. E.
t Value
Pr > |t|
0.6418
0.0850
7.55
<.0001
0.1394
0.0417
3.34
0..0009
Reversal of item
0.0383
0.0184
2.08
0.0380
Negation of item
0.0159
0.0185
0.86
0.3912
Use of a PO in the item
0.1197
0.0187
6.40
<.0001
Intercept
Fixation duration for item
(logaoi23plus1)
Note: These results are based on a mixed model with dist=gamma and construct as a random
effect.
Response biases in survey research
Item grouping and discriminant validity
(Weijters, de Beuckelaer, and Baumgartner,
forthcoming)
 question whether items belonging to the same scale
should be grouped or randomized:
□
grouped format is less cognitively demanding and often
improves convergent validity;
□ random format may reduce demand effects, respondent
satisfacing, and carryover effects, as well as faking;
 effect of item grouping on discriminant validity:
□
grouping of items enhances discriminant validity (Harrison
and McLaughlin 1996);
□ item grouping may lead to discriminant validity even when
there should be none;
Response biases in survey research
Method
 523 respondents from an online U.S. panel
 questionnaire contained the 8-item frugality scale of
Lastovicka et al. (1999) and 32 filler items;
 frugality scale presented in two random blocks of 4
items each, with the 32 filler items in between
□
□
Condition 1: 1-2-3-4 vs. 5-6-7-8
Condition 2: 1-2-7-8 vs. 3-4-5-6
 within blocks item order was randomized across
respondents;
Response biases in survey research
Estimated models
Response biases in survey research
Results
COND
Cond 1
Cond 2
²
DF
CFI
TLI
Hypothesized model
53.82
19
.964
.946
.037
.083
Best of alternative
permutations
198.13
19
.810
.720
.090
.188
Average of alternative
221.84
permutations
19
.788
.687
.096
.200
One factor
One-factor model
234.04
20
.776
.686
.095
.200
Two factors
Hypothesized model
43.67
19
.961
.943
.040
.071
Best of alternative
permutations
88.24
19
.880
.823
.069
.120
Average of alternative
129.93
permutations
19
.825
.743
.076
.151
One-factor model
20
.810
.734
.076
.154
Model
Two factors
One factor
MODEL
140.83
SRMR RMSEA
Response biases in survey research
Results (cont’d)
12
11
10
9
frequency
8
7
6
condition 1
condition 2
5
4
3
2
1
0
40
50
60
70
80
90 100 110 120 130 140 150 160 170 180 190 200 210 220 230 240
² value
Response biases in survey research
Results (cont’d)
Factor 1
Factor 2
Condition 1
Condition 2
Average Variance Extracted (AVE)
.54
.53
Composite Reliability (CR)
.82
.81
Average Variance Extracted (AVE)
.61
.40
Composite Reliability (CR)
.86
.71
.61
.61
.38
.38
Factors 1 and 2 Correlation
Shared Variance (SV)
Response biases in survey research
Extreme and midpoint responding as
satisficing strategies in online surveys
(Weijters and Baumgartner)
 when respondents minimize the amount of effort they
invest in formulating responses to questionnaire items by
selecting the first response that is deemed good enough,
they are said to be satisficing; when respondents put in
the cognitive resources required to arrive at an optimal
response, they are optimizing (Krosnick 1991);
 the effectiveness of procedural remedies to prevent or at
least reduce satisficing (MacKenzie & Podsakoff 2012) is
limited;
 post hoc indices designed to identify satisficers often
exhibit limited convergent validity and unambiguous
cutoff values are often unavailable;
Response biases in survey research
Satisficing in online surveys (cont’d)
 online surveys are likely to contain data from
respondents who are satisficing, but what will be the
consequences?
 we review satisficing and related measures that
have been proposed in the literature and propose a
new measure called OPTIM;
 we investigate the effect of satisficing on two stylistic
response tendencies (ERS and MRS) and we
demonstrate that the direction of the relationship
varies across individuals;
Response biases in survey research
The concept of satisficing
 the notion of satisficing is consistent with the view of
people as cognitive misers (Fiske and Taylor 1991);
 satisficing is conceptually similar to carelessness,
inattentiveness, insufficient effort responding, and
content-nonresponsive, content-independent,
noncontingent, inconsistent, variable or random
responding;
 Krosnick (1991) argues that in weak forms of satisficing
each of the four steps of the response process
(comprehension, retrieval, judgment, response) might be
compromised to some extent, whereas in strong forms of
satisficing the second and third steps might be skipped
entirely;
Response biases in survey research
Measures of satisficing
DIRECT MEASUREMENT
Satisficing is assessed directly by
measuring respondents’ tendency
to minimize time and effort when
responding to a survey
INDIRECT MEASUREMENT
Satisficing is assessed indirectly
based on the presumed
consequences of respondents’
attempts to minimize time and
effort on the quality of responses
DEDICATED MEASURES
NO DEDICATED MEASURES
Special items or scales are
included in the questionnaire to
measure satisficing
Satisficing is inferred from
respondents’ answers to
substantive questions
CATEGORY 1
CATEGORY 2
Self-reported effort (e.g., I
carefully read every survey item).
Response time
CATEGORY 3
CATEGORY 4
Quality of responses to special
items or scales (e.g. bogus items,
instructed response items)
Quality of responses to
substantive questions (e.g.,
outlier analysis, lack of
consistency of responses,
excessive consistency of
responses)
Response biases in survey research
Measures of satisficing (cont’d)
 a single-category measure is unlikely to assess
satisficing adequately;
 direct measures of satisficing are desirable (esp.
response time measures);
 bogus items and IMC’s have limitations;
 response differentiation for unrelated items might be
a good outcome-based measure;
Response biases in survey research
A new measure of satisficing
 optimizing as the time-intensive differentiation of responses
to items that are homogeneous in form but heterogeneous in
content:
OPTIM=log(TIME*DIFF)
 survey duration:
□
□
input side of effort (indicator of the cognitive resources invested by a
respondent);
time taken to complete the survey (in minutes), rescaled to a range of
0 to 10;
 response differentiation:
□
□
output side of effort (indicator of optimizing for heterogeneous items);
DIFF = (f1+1)*(f2+1)*(f3+1)*(f4+1)*(f5+1), rescaled to a range of 0 to
10;
Response biases in survey research
ERS and MRS as satisficing strategies
 previous research suggests that both ERS and MRS
may be used as satisficing strategies (even though
ERS and MRS tend to be negatively correlated),
although the empirical findings have not been very
consistent;
 different respondents may use different satisficing
strategies:
□
□
some respondents may simplify the rating task by
only using the extreme scale positions (resulting in
increased ERS);
others may refrain from thinking things through and
taking sides (resulting in increased MRS);
Response biases in survey research
Method
 two online studies with Belgian (n=320) and Dutch
(n=401) respondents;
 in dataset A 10 heterogeneous attitudinal items and
in dataset B Greenleaf’s (1992) ERS scale;
 these items were used to construct the ERS
(number of extreme responses), MRS (number of
midpoint responses) and DIFF measures; survey
duration was measured unobtrusively;
 use of a multivariate Poisson regression mixture
model of ERS and MRS on OPTIM;
Response biases in survey research
Model
Response biases in survey research
Regression estimates by class
DATASET A
DATASET B
DV
Intercept
B
SE
90% CI
Class 1
(46.3%)
Extreme
responders
Class 2
(47.1%)
ERS
2.64
-.54
.10
[-.70, -.38]
MRS
-3.08
1.39
.20
[1.06, 1.72]
ERS
-1.04
.90
.11
[.72, 1.08]
Yeah-sayers
MRS
-.53
.53
.11
[.35, .71]
ERS
-7.25
3.04
.83
[1.67, 4.41]
MRS
2.39
-.49
.14
[-.72, -.26]
ERS
-1.73
1.13
.12
[.93, 1.33]
MRS
2.59
-.53
.08
[-.66, -.40]
ERS
-.41
.88
.39
[.24, 1.52]
MRS
.24
.44
.25
[.03, .85]
ERS
2.91
-.48
.25
[-.89, -.07]
MRS
-.40
.57
.15
[.32, .82]
Class 3
(6.7%)
Midpoint
responders
Class 1
(43.7%)
Midpoint
responders
Class 2
(35.7%)
Yeah-sayers
Class 3
(20.6%)
Extreme
responders
Response biases in survey research
Dataset A
Dataset B
Response biases in survey research
Discussion
 OPTIM as an unobtrusive measure that integrates
several aspects of optimizing/satisficing;
 across two distinct samples, three satisficing
segments emerged:
□
□
□
extreme responders
midpoint responders
acquiescent responders
 OPTIM is useful if a continuous measure of
satisficing is required, but it may be less useful as a
screening device for careless responders;
Response biases in survey research
Stylistic response tendencies over the course
of a survey (Baumgartner and Weijters)
 three perspectives on stylistic responding:
□
□
□
nonexistence of response styles (complete lack of
consistency);
instability of response styles (local consistency);
stability of response styles (global consistency);
 Weijters et al. (2010) showed that
□
□
□
the nonexistence of response styles was strongly
contradicted by the empirical evidence for both extreme
responding and acquiescent responding;
there was a strong stable component in the ratings; and
there as a weaker local component (as indicated by a
small time-invariant autoregressive effect);
Response biases in survey research
Unresolved questions
 how do stylistic response tendencies evolve over the
course of a questionnaire?
 prior research has only considered the effect of stylistic
responding on the covariance structure of items or sets
of items and has ignored the mean structure;
 are there individual differences in both the extent to
which stylistic response tendencies occur across
respondents and the manner in which stylistic response
tendencies evolve over the course of a survey?
 prior research has not emphasized heterogeneity in
stylistic response tendencies across people;
Response biases in survey research
ALT model
Response biases in survey research
Integrated ALT model for NARS and ERS
Response biases in survey research
Method
 data from 523 online respondents;
 each participant responded to a random selection of
eight out of 16 possible four-item scales shown on eight
consecutive screens in random order;
 eight separate response style indices were computed for
both (net) acquiescence response style or NARS (i.e.,
respondents’ tendency to express more agreement than
disagreement) and extreme response style or ERS (i.e.,
respondents’ disproportionate use of more extreme
response options);
 the design guarantees that there is no systematic
similarity in substantive content over the sequence of
eight scales across respondents;
Response biases in survey research
Results
Estimate
Means
Variances
iERS
sERS
iNARS
sNARS
iERS
sERS
iNARS
sNARS
SE
T
p
.918
-.009
3.298
-.019
.105
.001
.155
.001
.019 49.03 < .001
.003 -2.77
.006
.027 123.59 < .001
.005 -3.77 < .001
.012
9.00 < .001
.000
2.49
.013
.024
6.38 < .001
.001
1.30
.195
-.312
.120
iERS with iNARS
.491
iERS with sNARS
Correlations iERS with sERS
-2.60
95% confidence
interval
[ .881, .955]
[ -.015, -.003]
[3.246, 3.350]
[ -.029, -.009]
[ .082, .128]
[ .000, .002]
[ .108, .203]
[-.001, .003]
.009
[-.547, -.077]
.082
5.98 < .001
[ .330, .651]
-.189
.204
-.93
.352
[-.588, .210]
iNARS with sNARS
-.583
.132
-4.42 < .001
[-.842, -.325]
sERS with iNARS
-.074
.172
-.43
.669
[-.411, .264]
sERS with sNARS
.129
.388
.33
.740
[-.632, .889]
Response biases in survey research
NARS and ERS trajectories
1.20
1.00
5
0.92
0.86
4
3
0.40
2
3.20
3.18
3.17
3.22
3.26
3.24
3.28
0.20
1
2
3
4
5
6
Questionnaire page (screen) number
7
8
0.00
1
NARS
ERS
ERS - 1 SD slope
ERS + 1 SD slope
NARS
0.60
3.30
ERS
0.80
Response biases in survey research
Distribution of the slope factor for ERS
Response biases in survey research
Response distributions on the first and
the last screen of the questionnaire
Response biases in survey research
Backup slides
Response biases in survey research
A comprehensive model of measurement error
yijt = ijt + ijt jt + ijt + ijt
systematic
error
yijt 
jt

ijt
ijt
ijt
ijt




random
error
a person’s observed score on the ith measure
of construct j at time t
a person’s unobserved score for construct j at
time t
systematic error score
random error score
coefficient (factor loading) relating yijt to jt
intercept term (additive bias)
Response biases in survey research
Empirical data
 we analyzed items from volumes 1 through 36 of
JCR (1974 till the end of 2009) and volumes 1
through 46 of JMR (1964 to 2009);
 we included all Likert-type scales for which the items
making up the scale were reproduced in the article
and factor loadings or item-total correlations were
reported;
 total of 66 articles in which information about 1330
items measuring 314 factors was provided;
 of the 1330 distinct items in the data set, 608 came
from JCR and 722 from JMR;
Response biases in survey research
Item reversal (cont’d)
 in our data set of 1330 items, between 83 and 86
percent of items were nonreversed (depending on
the definition of reversal);
 the proportion of factors (or subfactors in the case of
multi-factor constructs) that do not contain reversed
items was 70 percent;
 only 8 percent of factors (out of 314 factors) were
composed of an equal number of reversed and
nonreversed items (i.e., the scale was balanced);
Response biases in survey research
Cross-classification of negation and reversal
Reversal relative to
Polarity of construct
Polarity of first item
Dominant keying direction
Total
Nonreversed
Reversed
Nonreversed
Reversed
Nonreversed
Balanced
Reversed
No
negation(s)
71.1%
8.7%
71.0%
8.7%
70.3%
5.2%
4.2%
79.7%
Negation(s)
11.7%
8.7%
15.4%
4.9%
13.7%
2.9%
3.7%
20.3%
Total
82.8%
17.4%
86.4%
13.6%
84.0%
8.1%
7.9%
100.0%
Response biases in survey research
Theoretical explanations of MR:
Reversal ambiguity and comprehension
 Rs may not view antonyms as polar opposites [POMR];
 contradictories vs. contraries:
□
□
Antonym reversals can be contradictories or contraries, depending
on whether they are bounded or unbounded (Paradis and Willners
2006);
Negation reversals are contradictories if the core concept is the
same; the situation is more complicated for the negation of
bounded and unbounded antonyms;
 simultaneous disagreement with contraries is more likely
when items are worded extremely (McPherson and Mohr
2005);
 “Buddhism’s ontology and epistemology appear to make
East Asians relatively comfortable with apparent
contradictions” (Wong et al. 2003, p. 86) [RMR];
Response biases in survey research
Theoretical explanations of MR:
Comprehension
 Careless responding (Schmitt and Stults 1985):
□
□
□
respondents fail to pay careful attention to individual
items and respond based on their overall position on
an issue [RMR];
more likely when a reversed item is preceded by a
block of nonreversed items;
Remedies:


make Rs more attentive and/or explicitly alert them to
the presence of reversed items;
use balanced scales, alternate the keying direction, and
disperse the items;
Response biases in survey research
Theoretical explanations of MR:
Comprehension (cont’d)
 Reversal ambiguity:
 Rs
may not view antonyms as polar opposites [POMR];
 “Buddhism’s ontology and epistemology appear to make
East Asians relatively comfortable with apparent
contradictions” (Wong et al. 2003, p. 86) [RMR];
 contradictories vs. contraries:


Negated statements are contradictories;
Antonyms can be contradictories or contraries, depending on
whether they are bounded or unbounded (Paradis and Willners
2006);
 simultaneous
disagreement is more likely when items
are worded extremely (McPherson and Mohr 2005);
Response biases in survey research
Theoretical explanations of MR:
Comprehension (cont’d)
□
Remedies:




use more sophisticated procedures to identify
appropriate antonyms (formulate linguistic contrasts in
two stages; see Dickson and Albaum 1977);
may be particularly useful in cross-cultural research;
bounded antonyms have to be pretested and
unbounded antonyms have to be used with care;
extreme statements should be avoided;
Response biases in survey research
Theoretical explanations of MR:
Retrieval
 Item-wording effects:
□
□
□
□
Confirmation bias (Davies 2003; Kunda et al. 1993);
Directly applicable to antonymic reversals;
For negation reversals, confirmation bias can lead to
MR if a non-negated polar opposite schema is readily
available (Mayo et al. 2004);
Remedies:


Use polar opposite reversals to get richer belief samples,
even though they may increase apparent MR;
Negation reversals have few retrieval benefits;
Response biases in survey research
Theoretical explanations of MR:
Retrieval
 Positioning effects:
□
□
□
Dispersed PO items reduce carryover effects and can
increase coverage, but the task is more taxing for Rs
and internal consistency may suffer;
Item similarity may determine whether Rs engage in
additional retrieval when items are grouped together;
Remedies:


The use of dispersed antonyms should encourage the
generation of distinct belief samples;
Avoid very similar (negated) statements when items are
grouped;
Response biases in survey research
Theoretical explanations of MR:
Judgment
 Item verification difficulty (Carpenter and Just 1975;
Swain et al. 2008):
□
□
MR is a function of the complexity of verifying the
truth or falsity of an item relative to one’s true beliefs,
which depends on whether the item is stated as an
affirmation or negation [NMR];
Remedies:



Negations are problematic because they increase the
likelihood of making mistakes (remember there are
many types of negations);
Negated polar opposites are most error-prone;
Mix of regular and PO reversals should be best;
Response biases in survey research
Item verification difficulty
MR
Negation
Affirmation
Truth value
True
False
Response biases in survey research
Theoretical explanations of MR:
Response
 Acquiescence: Rs initially accept a statement and
subsequently re-consider it based on extant evidence; the
first stage is automatic, the second requires effort
(Knowles and Condon 1999) [RMR];
 Remedies:
□
□
Although response styles are largely individual
difference variables, situational factors may be under
the control of the researcher (e.g., reduce the
cognitive load for Rs);
Problems with online surveys;
Response biases in survey research
Theoretical explanations of MR:
Response (cont’d)
 Asymmetric scale interpretation: the midpoint of the
rating scale may not be the boundary between agreement
and disagreement for Rs (esp. if the response categories
are not labeled; cf. Gannon and Ostrom 1996) [RMR];
 Remedies:
□
Use fully labeled 5- or 7-point response scales;
Response biases in survey research
Careless responding
 respondents do not always pay attention to the instructions,
the wording of the question, or the response options before
answering survey questions because of satisficing;
 respondents may form expectations about what is being measured
and respond to individual items based on their overall position
concerning the focal issue, rather than specific item content;
 this can result in inconsistent responding to reverse-keyed items;
 esp. likely when constructs are labeled and when grouped-massed
item positioning is used;
 measurement:
 instructional manipulation checks (IMC), bogus items, and selfreport measures of response quality
 response times
 post hoc response consistency indices (too much or too little)
Response biases in survey research
Examples of IMC’s
Between 14% and 46% of respondents failed this test in Oppenheimer et
al. (2009)
Response biases in survey research
Examples of IMC’s
About 7% of respondents (out of over 1000) failed this test (see
Oppenheimer et al. 2009)
Response biases in survey research
(Dis)Acquiescence
 tendency to agree (ARS) or disagree (DARS) with items
regardless of content (agreement or yea-saying vs.
disagreement or nay-saying bias);
 leads to response inconsistency for reversed items;
 Measurement:

simultaneous (dis)agreement with contradictory statements;

(dis)agreement with many heterogeneous items;

net acquiescence as the relative bias away from the midpoint;
 different arrangements of the items in the questionnaire
should have no differential effect on acquiescent
responding;
Response biases in survey research
Confirmation bias
 when respondents answer a question, they tend to
activate beliefs that are consistent with the way in
which the item is stated (positive test strategy,
inhibition of disconfirming evidence);
 this leads to a bias in the direction in which the item
is worded (e.g., Are you introverted? vs. Are you
extraverted?) and differences in mean response;
 the effect of the keying direction of the first item on
confirmation bias should be strongest in the
grouped-massed condition and weakest in the
dispersed condition;
Response biases in survey research
Example item with 4 negations
Top management in my company has let it be known in
no uncertain terms that unethical behaviors will not be
tolerated.
Response biases in survey research
Revised Life Orientation Test (LOT)
 In uncertain times, I usually expect the best.
 I’m always optimistic about my future .
 Overall, I expect more good things to happen to me
than bad.
 If something can go wrong for me, it will.
 I hardly ever expect things to go my way.
 I rarely count on good things happening to me.
Response biases in survey research
Weijters, Baumgartner, and Schillewaert
(forthcoming)
 models in which method effects are included generally
yield a much better fit to the data than models in which
only substantive factors are included;
 it is often difficult to clearly distinguish between
different method effect specifications on the basis of
statistical criteria;
 the psychological processes causing method effects
are frequently left unspecified;
 although method factors have been related to a variety
of other psychological constructs, the choice of these
other constructs often seems ad hoc;
Response biases in survey research
Response biases in survey research
Response biases in survey research
Response biases in survey research
Weijters, Baumgartner, and Schillewaert
(forthcoming)
 models in which method effects are included generally
yield a much better fit to the data than models in which
only substantive factors are included;
 it is often difficult to clearly distinguish between
different method effect specifications on the basis of
statistical criteria;
 the psychological processes causing method effects
are frequently left unspecified;
 although method factors have been related to a variety
of other psychological constructs, the choice of these
other constructs often seems ad hoc;
Response biases in survey research
Responses to bogus items
(Meade and Craig, 2012)
% strongly disagree or
disagree responses
(1,2)
% other responses
(3-7)
I sleep less than one hour per night.
90
10
I do not understand a word of English.
90
10
I have never brushed my teeth.
92
8
I am paid biweekly by leprechauns.
80
20
All my friends are aliens.
82
18
All my friends say I would make a great poodle.
73
27
Response biases in survey research
Strongly
Disagree
(1)
Disagree
(2)
Neither Agree
nor Disagree
(3)
Agree
(4)
Strongly
Agree
(5)
I feel satisfied
with the way my
body looks right
now.





I am satisfied
with my weight.





I am pleased
with my
appearance
right now.





I feel attractive.





I don’t feel
attractive.





I feel ugly.





I don’t feel ugly.





Response biases in survey research
Strongly Disagree
(1)
Disagree (2)
Neither Agree nor
Disagree (3)
Agree (4)
Strongly Agree
(5)
A product is more
valuable to me if
it has some snob
appeal.





The government
should exercise
more
responsibility for
regulating the
advertising, sales
and marketing
activities of
manufacturers.





Most retailers
provide adequate
service.





I feel attractive.





I don’t feel
attractive.





I feel ugly.





I don’t feel ugly.





Response biases in survey research
OPTIM