The measurement
effect in PC
smartphone and tablet
surveys
Valerija Kolbas
University of Essex
ISER
Ipsos-MORI
The European Survey Research Association Conference
13-17 July, 2015 Reykjavik
Background and motivation
Limiting surveys to the PC mode affects
the size and representativeness of the
sample
 PC, smartphones, tablets differ in:






size of the screen
input method
speed of connection
processing power
Different effect on the measurement
error
Background and motivation

Smartphone

more break-offs
longer completion time
More straightlining


 Tablet
 Less primacy effect
 Less straightlining
 Less or comparable
completion time
Other indicators




question order effect
primacy effects
open-ended answers
response distribution
Mixed results
… but non-optimized mobile design affects completion rates, satisfaction with
the survey
Background and motivation



PC and mobile response distributions are
equally affected by response formats
Drop-boxes –preference for first options
Grids – preference for visible options,
straightlining
- no conclusive evidence
which format is better for mobiles
Research Question
How mode of administration and response format
affect survey responses

Indicators of measurement error:




overall satisfaction rates
straightlining
response distribution
length of open-end questions
Survey and Questionnaire design






National Satisfaction Survey 2014
Administered to final year higher education
students in UK
22 core questions using a 5-point Likert scale
2 open-end questions
Mixed-mode: self-selected mail, phone, web
5 response formats – randomly allocated
Sample composition
Radio-button
PC
N=5529
Web
Survey
Sample
N=9276
Radio-button
Smartphone
N=3196
Drop-box (+)
Drop-box (-)
Tablet
N=551
Drop-box (0)
Drop-box ()
Screenshots: response design
• Once clicked a list with options appears on a separate screen.
• Responses always in the same order
Screenshots: response design
• Requires vertical
scrolling.
• All questions are visible on the
screen.
• Requires horizontal scrolling .
• Portrait or landscape viewing.
Measurement Effect
between PC smartphone
tablet responses
Comparisons made
across all three modes, but within one
radio-button response format
10
Measurement Effect between PC
smartphone tablet
MANOVA to test for differences F=2.3, p<.05
F
p
PC
smartphone
tablet
Straightlining
3.9
<.05
6.9%
10.3%
6.4%
Modal
responses
<1
>.05
Mean
values
2.7
.07
4.1
4.1
4
Positive
feedback
1.3
>.05
195
180
191
Negative
feedback
1
>.05
257
240
243
‘Mostly
Agree’
‘Definitely Agree’
&
‘Mostly Agree’
‘Mostly
Agree’
ME between PC, smartphone,
tablet summary
 Smartphone
straightlining significantly
higher
 Tablet straightlining rate the lowest
 Signs of the visibility effect for smartphone
 Other quality indicators comparable
between all three modes
Measurement Error between
different response designs
presented on a smartphone and a
tablet
Comparisons made across five survey
response formats within smartphone and
tablet modes
13

MANOVA test of differences
F=2.7, p<.05
F
p
device
2.7
<.05
format
1.7
<.05
interaction
1.3
.18
 Both device and format affected data quality
Responses in Drop-box with a positive initial
option
70.00%
Smartphones
60.00%
 higher selection of initially suggested
response.
 less moderately positive responses
 11.1% straightliners
 Mean 4.3
50.00%
40.00%
30.00%
Tablets
20.00%
10.00%
0.00%
Def.Agree
Mostly
Agree
Neither
smartphone
Mostly
Disagree
tablet
Def.
Disagree
 Similar selection of extreme and
moderately positive responses
 More negative responses
 5.6% straightliners
 Mean 4.1
Responses in Drop-box with a negative initial
option
50.00%
Smartphones
45.00%




40.00%
35.00%
30.00%
More extreme negative responses
Less positive responses
8.4% straightliners
Mean 4
25.00%
20.00%
Tablets
15.00%
10.00%
5.00%
0.00%
Def.Agree
Mostly
Agree
Neither
smartphone
Mostly
Disagree
tablet
Def.
Disagree
 More positive responses
 Extremely low ‘Definitely Disagree’
frequency
 2.5% straightliners
 Mean 4.1
Responses in Drop-box with a middle initial option
45.00%
Smartphone and Tablet
40.00%
Comparable results
35.00%
 Weak evidence of selecting the
middle option
 Most frequent ‘Mostly
Agree/Disagree’ selection across
formats
 6.5% and 6.1% of straightliners
 Mean 4 an 3.9
30.00%
25.00%
20.00%
15.00%
10.00%
5.00%
0.00%
Def.Agree
Mostly
Agree
Neither
smartphone
Mostly
Disagree
tablet
Def.
Disagree
ME indicators summary

Smartphone – answers affected by
response formats.

Initially suggested response is selected more
often
Tablet – no strong effect of response
formats.
 No significant differences in the length of
open answers between formats.

Potential Limitations






No reverse-coding
Similar question wording
Not counterbalanced question or response
order
Instructions universal for each response
format
Self selected device condition
Survey sample: highly educated, IT literate,
similar age, highly motivated
Thank You
vkolba@essex.ac.uk