Measuring Respondent Burden to
Statistical Surveys
John-Mark Frost, Sarah Green, Jacqui Jones
& Denise Williams
UK Office for National Statistics
Aim of Presentation
• To provide an overview of the:
- Drivers for the UK pilot of using the Standard Cost
Model to measure respondent burden from
statistical surveys
- Issues highlighted by this work
Overview of Presentation
• Measuring respondent burden from statistical surveys
- Drivers
- Measurement
• Standard Cost Model for measuring respondent
burden from statistical surveys
-
Drivers
Measurement
Pilot
Issues
• The way forward
Drivers for Measuring Respondent
Burden from Statistical Surveys
PMIs
Prime Ministers’ Instructions on the
Control of Statistical Surveys
Code
of
Practice
UK Code of Practice for Official Statistics
Report annually the estimated costs (for example, on
businesses, service providers, or the public) of
responding to statistical surveys …
Measurement of Respondent Burden
from Statistical Surveys
Measurement
PMIs
Code
of
Practice
Simple method:
- used during survey control reviews
- 20% of survey sample
- postal self-completion questionnaire
Calculated by:
(No. of questionnaires (100% response)) *
(estimated completion time) *
(appropriate hourly rate)
+
Up-rating for respondent re-contact
Recognised weakness in method:
“questionnaire did not adequately capture all
activities involved in complying with the
regulation”
SCM to Measure Total Respondent
Burden in Europe
Measurement
PMIs
Code
of
Practice
Statistics Netherlands Standard Cost Model
- European approach to measure administrative
burdens for all government information requests
- Including survey participation
Standard Cost Model approach:
SCM
- Highlights impact of international legislation
- Transparent measurement
- Measures administrative burden
- Data collected in face-to-face interviews
SCM to Measure Total Respondent
Burden in the UK
Measurement
PM’s
Code
Administrative Burden Reduction Project
of
- Total UK administrative burden
- Includes survey participation
- Burden associated with statistical surveys
forms only a small part of the total
- Use SCM
Practice
SCM
ABRP
Drivers for Measuring Respondent
Burden from Statistical Surveys
Measurement
PM’s
Need to change method
Code
of
!
Practice
Need consistency
= Use SCM
SCM
ABRP
Introduction of the SCM to Measure
Respondent Burden from Statistical Surveys
• ABRP used face-to-face interviews with very small
sample sizes
- Not feasible or robust enough for measuring compliance on
a regular basis
• A paper-based version of the SCM developed to
measure costs of complying with statistical surveys
- Implement quantitative rather than qualitative approach
• The paper-based version was piloted in ONS
SCM to Measure Respondent Burden
from Statistical Surveys
• Questionnaire developed
-
10 questions
Breakdown of responding activity
Information available from ‘business-as-usual’
External costs
Survey irritants
- For example: unclear definitions or available information
did not match information requested
• Standard Cost Model formula:
mean weighted cost per questionnaire + re-contact uplift *
number of questionnaires in survey sample *
survey frequency
Pilot for Measuring Respondent Burden
from Statistical Surveys Using the SCM
• Pilot highlighted large differences in estimates
when compared to:
- Face-to-face interview methodology
- Previous ONS methodology
• Differences to previous ONS methodology
are driven by:
-
Different information in models
Change in pay rates
Different uplifts for overheads
Adjustment to internal costs for business-as-usual
Issues with the SCM for Measuring
Respondent Burden from Statistical Surveys
• Figures surrounded by confidence intervals of the
same magnitude as previous methodology
• Method is hugely resource intensive
- Respondents
- Data producers
• Difficulties in estimating accurately
- Breakdown of activities
- Associating costs with respondents
Questions Raised from Pilot Results
• Statistical surveys form only small fraction of
total administrative burden
- Does measurement need to be consistent?
• Are such resource costs beneficial?
- The figures are purely estimates
• Would a simpler method be:
- Fit-for-purpose?
- As robust?
Way Forward for Measuring Respondent
Burden from Statistical Surveys
• Recommendation to use a simpler model:
- More proportionate
- Is concerned with consistently measuring change
over time rather than accurate levels
• ONS has worked with producers of UK official
statistics to develop such a model
- Currently proposals are being finalised to take to
the National Statistician for approval
Thank you
John-Mark.Frost@ons.gov.uk