1
Third International Conference
on Establishment surveys.
Montréal 2007
Web and Paper Questionnaires seen
from the Business Respondent’s
Perspective
Gustav Haraldsen, Statistics Norway
Jacqui Jones, UK Office of National Statistics
1
Outline
• A quality driven approach to response burden
• The effect of survey design, external conditions and
personal characteristics on perceived response burden
• The impact of perceived response burden on data
collection and data quality
• (The perceived response burden of web versus paper
questionnaires)
2
Origins and Effects of Perceived Response Burden in
Establishment Surveys
Survey Requirements
Survey Design
Sample
Stakeholder (s)
Specification
Frame
Sampling procedure
Size
Response environment
Response
quality
Context:
Encoding/record formation
Respondent selection/identification
Priorities
Questionnaire
Survey
organisation
Length
Structure
Content
Layout
Procedure
Contact mode(s)
Persuasion strategies
Management
Confidentiality
Respondent(s)
Release
• Availability
• Motivation
• Competence
Revision
3
The quality driven approach
• Response burden is mainly a problem as long as it affects
the response quality
• It is the respondents’ response burdens that matters
• Perceived response burden is a more relevant variable than
completion time
• The response burden may be affected by external
conditions or personal characteristics of the respondents as
well as by the survey design.
• Perceived response burden questions should identify why
questionnaires are burdensome
4
Core PRB Questions
Do you think it was quick or time consuming to collect the
necessary information to complete the questionnaire?
+1
+0.5
0
-0.5
-1
Very quick
Quite quick
Neither quick nor time consuming
Quite time consuming
Very time consuming
These were asked to pick the
main reason(s) from a list or
specify other reason(s)
Did you find it easy or burdensome to complete the questionnaire?
+1
+0.5
0
-0.5
-1
Very easy
Quite easy
Neither easy nor burdensome
Quite burdensome
Very burdensome
These were asked to pick the
main reason(s) from a list or
specify other reason(s)
5
Sources of perceived response burden
•
Survey Design Factors
– Characteristics of 6 web questionnaires directed to different industries
– (The time it took to fill in the questionnaire)
•
External conditions
– Size: The number of employees
– Information sources: Documentation problems that made the information
collection time consuming
– Division of labour: The number of persons who took part in the response
process
•
Personal Characteristics
– Availability: How easy or difficult it was to set aside time for the
questionnaire
– Motivation: How useful for the business and society the respondent felt that
statistics produced from the data were
– Competence: If he/she was a first time or experienced respondent
6
2006 Web Response and PRB Response for
Annual Business Surveys
Web response
PRB response
Hotels and restaurants
31 %
43 %
Transport &
communication
31 %
40 %
Sea transport
41 %
30 %
Service industries
33 %
45 %
Domestic trade
33 %
44 %
Construction
31 %
43 %
7
Perceptions of the Information Collection and
Completion
of the
2006
Questionnaires
Collecting Information
and
Completing
Business
Questionnaires
Index
Minutes
Collecting
0,01
98
Completing
0,10
55
0%
20 %
-1
40 %
-0,5
60 %
0
80 %
0,5
100 %
1
8
Sources of Time Consuming Preparations
Other reasons
21,1 %
Waiting for information
21,1 %
Many questionnaires
21,3 %
Need for help
32,7 %
72,3 %
Different sources
0%
20 %
40 %
60 %
80 %
100 %
9
Sources of Response Burdens
E-technical
5,8 %
Other
7,9 %
E-usability
Terms
8,8 %
15,9 %
22,3 %
Layout
No. of questions
31,3 %
34,0 %
Response categories
46,2 %
Mismatch
48,8 %
Calculations
0%
20 %
40 %
60 %
80 %
100 %
10
Information collection and Perceived response
burden index: Industries and completion time
IC-index
PRB - index
Construction
-0.22
-0.17
Sea transport
0.04
(0.02)
Transport and communication
0.03
0.09
Hotels & restaurants
-0.05
0.10
Domestic trade
0.01
0.10
Service industry
0.12
0.32
Completion time
0 – 25 minutes
0,32
26 – 50 minutes
0,11
51 minutes +
-0,16
11
PRB-index: External conditions
PRB- index
Size of
0 – 9 employees
establishment
10 – 19 employees
(0,13)
20 – 49 employees
0,06
50 or more employees
0,10
Documentation
None
0,15
problems
1
(-0,14)
2
((-0,23))
3
-0,45
No of
1
0,15
Respondents
2 or more
0,11
((-0,01))
12
PRB-index: Personal characteristics
PRB- index
Available time
Very easy to find
0,68
Quite easy to find
0,41
Neither easy nor difficult
0,06
Quite difficult
-0,09
Very difficult
-0,37
Perception of
Very useful
((0,22))
usefulness
Quite useful
0,30
Neither useful nor useless
0,18
Quite useless
0,09
Very useless
-0,11
No
0,04
Yes
0,11
Experience
13
The effect of survey design, external conditions and
personal characteristics on PRB
• Calculations and mismatch between questions and available
•
•
•
•
•
information are general problems in business surveys
Sufficient time available for the questionnaire is essential for
the perceived response burden.
The number of questions was probably the most important
difference between questionnaires
Poor layout and usability problems were often mentioned by
those who felt that the questionnaire was very burdensome
The quality of documentation systems is probably the most
important difference between industries
The perception of uselessness goes together with perceived
response burden
14
Web Response Patterns for Construction and
Service Industry Questionnaires
100 %
Service
80 %
Construction
60 %
40 %
20 %
de
ad
l
O ine
ct
ob
er
Fi
na
l
in
de
r
2.
re
m
Au
gu
st
de
ad
li
June
1.
ne
re
m
in
de
r
2.
de J
ad uly
lin
e
1.
M
ay
0%
15
Questionnaires corrected
Construction
80,9 %
Sea transport
53,3 %
Transport and
communication
63,3 %
Hotels & restaurants
79,7 %
Domestic trade
70,1 %
Service industry
33,4 %
16
The impact of perceived response burden on data
collection and data quality
• Generally in business surveys..
– the data collection time is very long and
– the number of questionnaires that are corrected is very high
• When there is a pronounced difference in perceived
response burden between questionnaires, there also seems
to be a difference in…
– How quickly the data are received
– The number of questionnaires that need to be corrected
• Problems with collecting the information asked for may also
have an effect on these quality indicators
17
Perceived response burden of the paper and web
versions of the Annual Manufacturing Industry
Web versus
surveys (2004 and
2005)paper
PRBindex
Multi business web
0,01
Multi business paper
-0,08
One business web
0,39
One business paper
0,32
0%
10 % 20 % 30 % 40 % 50 % 60 % 70 % 80 % 90 % 100 %
-1
-0,5
0
0,5
1
18
Sources of response burden in paper and web
versions of the Manufacturing Industry Surveys
One business
Multi business
Paper
Web
Diff
Paper
Web
Diff
Mismatch
53
53
0
72
67
5
Calculations
18
39
-21
36
56
-20
Terms
24
29
-5
22
78
-56
No of questions
5
10
-5
21
44
-23
Response categories
14
18
-4
21
22
-1
Layout
1
4
-3
3
22
-20
E-technical/usability
0
6
-6
0
11
-11
Other
6
14
-8
5
6
-1
SUM
121
173
-52
180
306
-126
19
(The perceived response burden of web versus
paper questionnaires)
• When the Manufacturing Industry Survey changed from
paper to web…
– The perceived response burden apparently went down
– While the number of problems reported by those who felt the
questionnaire burdensome seemed to go up
A quote for reflection: “It takes 100 men to do as much
harm as a computer can make within 1/100 of a second”
20