Task force on improvement of
response rates and minimisation of
respondent load
Assessing and minimising the impact
of non-response on survey estimates
Richard McKenzie OECD
Nov 14-15 2005
Joint OECD / European Commission workshop on international
development in business and consumer tendency surveys
Terms of reference
– How to asses the impact (bias) that non-response can
have on survey estimates. Work on this issue should
take into account both the impact of the overall level of
non-response (e.g. 10% vs 50%) and non-response by
type of business or consumer (e.g. large businesses vs
small) and type of variable / question.
– Develop methods to minimise the impact (bias) of nonresponse (e.g. imputation methods, estimation
methodologies).
Nov 14-15 2005
Joint OECD / European Commission workshop on international
development in business and consumer tendency surveys
Task force work to address the issue
3 main activities:
• Internet research to find relevant documents on the
topic
• Provision of documents by task force members
(from their work or work they were aware of)
• Specific questionnaire to a sample of institutes to
collect information relevant to the terms of
reference
Nov 14-15 2005
Joint OECD / European Commission workshop on international
development in business and consumer tendency surveys
Main issues explored
• Defining and observing non-response rates
• Weighting methodologies used and their
relationship to the treatment of non-response
• Literature review to assess the likely impact on nonresponse on survey estimates
• Conclusions and recommendations for future work
Nov 14-15 2005
Joint OECD / European Commission workshop on international
development in business and consumer tendency surveys
Defining non-response rates
OECD (2003)
'
1
 * wi
i 1 f i
NR3  n
*100
1
 * wi
11 f i
n
Nov 14-15 2005
Joint OECD / European Commission workshop on international
development in business and consumer tendency surveys
Institute
Survey & Response rate
Confederation of British Industry
Industry 45%, Retail 22% , Services 20%
South Africa: Bureau for Econ
Research
40-45% across retail / wholesale, manufacturing,
motor trade and contractors (building related).
Canada: Statistics Bureau
Manufacturing 55%
Finland: Statistics Bureau
Consumer Survey 74%
Austria: WIFO
Manufacturing 30%, Construction 30%, Services
34%.
France: INSEE
Industry 81% , Retail 82% (5,000), Services 76%,
Construction 84%
Netherlands: Statistics Office
Industry 90%
Slovenia: Statistics Office
Consumer Survey 68%, Industry 91%, Construction
92%, Services 92% , Retail 82%
Japan: Central Bank
Whole economy: TANKAN survey 95%.
Germany: IFO
Manufacturing 90%, Construction 70%
Slovak Republic: Statistics Office
Industry 75%, Construction 85%, Retail 60%,
Services 63% (500)
Statistics Norway
Industry 85%
China
Nov 14-15 2005
Joint OECD / European Commission workshop on international
Whole economy 95%
development in business and consumer tendency surveys
Issues affecting response rates
• Authority data collected under
• Survey resources available
• Timeliness requirements (limits follow ups)
– Regardless of the reasons, representativeness of the
responding sample is the key issue
Nov 14-15 2005
Joint OECD / European Commission workshop on international
development in business and consumer tendency surveys
Weighting methodology
• Weighting (and estimation) methodology and its
relationship to sampling error and non-response
bias is a major issue for quantitative business
surveys
– Doesn’t appear to have been given the same attention in
qualitative business surveys
• Key role of qualitative business surveys is to give
leading indicators of likely movements in macroeconomic (qualitative) variables
Nov 14-15 2005
Joint OECD / European Commission workshop on international
development in business and consumer tendency surveys
Weighting methodology: OECD guidelines
Business weight should be composed of:
• Sampling probability / fraction
• Business size (usually employment or
turnover)
Above weights are used to form survey
estimates at the Branch / group level, which
should then be weighted together to form
aggregates (e.g. total manufacturing)
Nov 14-15 2005
Joint OECD / European Commission workshop on international
development in business and consumer tendency surveys
Importance of the sampling fraction
Sampling fraction may be difficult to use as:
• Poor quality (or non existent) survey frame
• Infrequent sample updates
• Business selection methods (e.g. quota
sampling, recruitment to panel)
Therefore it may be considered unobtainable or
irrelevant?
Nov 14-15 2005
Joint OECD / European Commission workshop on international
development in business and consumer tendency surveys
Importance of the sampling fraction
If small and large business are included in the same
group / cell estimate, weighted only by business
size then the resulting estimates are biased and not
representative of the population.
– Must estimate a weight for each (group) of business in
the sample which accounts for other similar size business
in the population that they are representing
– Otherwise you are better off excluding small businesses
from the scope of the survey (cutoff sampling) and saving
resources.
Nov 14-15 2005
Joint OECD / European Commission workshop on international
development in business and consumer tendency surveys
Treatment of non-response: OECD (2003)
If non–response can be expected to be systematic,
in the sense that units which have had or are
expecting an especially good or bad development
are also an unduly large part of non–response, then
special measures need to be taken in order to avoid
bias. One possible approach is to construct a
separate “non–response stratum”, and take a
repeat sub–sample from this stratum for which
further strong efforts are made to collect data. This
information can then be used to make separate
estimates for this “non–response stratum”.
Nov 14-15 2005
Joint OECD / European Commission workshop on international
development in business and consumer tendency surveys
The ‘missing at random’ assumption
European Commission – harmonised guidelines
• Non responding businesses are ignored in applying
survey weights and forming survey estimates
• Assumes the average (weighted) distribution of
answers from responding businesses is
representative of non-responding businesses
– Is this a reasonable assumption? It was used by all
institutes surveyed for BTS
• Consumer surveys can use population estimates
from census’s for post stratification.
Nov 14-15 2005
Joint OECD / European Commission workshop on international
development in business and consumer tendency surveys
Is the ‘missing at random’ assumption
reasonable?
• Appears to have been very little work done to
explore this issue for business tendency surveys
– Wang (2004) attempted this for Statistics Norway, but the
study was based on only one cycle of the survey and
involved a number of simplifying assumptions
– Nearest neighbour imputation based on employment size
showed some evidence of difference in response pattern
between size of unit – although impact on estimates was
small.
Nov 14-15 2005
Joint OECD / European Commission workshop on international
development in business and consumer tendency surveys
Is the ‘missing at random’ assumption
reasonable?
• Pellisier (2005) looked at the impact of response
patterns on the Retail Business Confidence
Indicator in the South African Retail Trade survey
– 3 groups of respondents: active (respond more than 75%
of the time); less active (30 – 75%); occasional (< 30%)
– Some evidence of differing response patterns, more
details on this study in the next presentation!
Nov 14-15 2005
Joint OECD / European Commission workshop on international
development in business and consumer tendency surveys
Is the ‘missing at random’ assumption
reasonable?
• INSEE: ‘constant sample’ imputation method
“As enterprises do not systematically answer each
monthly survey, or may submit responses after the cutoff
for publication, only taking into account answers from
businesses responding to a particularly monthly cycle of
a survey can lead to a false diagnosis of changes in the
business climate if this is due only to a change in
structure of the respondents”
– Constant sample method explicitly imputes for both
complete and item non-response based on the historical
response pattern of the business
Nov 14-15 2005
Joint OECD / European Commission workshop on international
development in business and consumer tendency surveys
Non-response bias for consumer surveys
An extensive study by Curtin et al.(2000) found:
– Some small difference in Index of Consumer Sentiment
(ICS) between easier and harder to interview
respondents, but there was a negligible impact on the
overall index if harder to interview respondents are
excluded
– Difference in the ICS between easier and harder to
contact respondents was constant over time – implying
there is no non-response bias for time series (i.e.
estimates of change)
• Conclusion was that ICSs ability to predict future
changes in economic conditions is unlikely to be
affected by non-response bias
Nov 14-15 2005
Joint OECD / European Commission workshop on international
development in business and consumer tendency surveys
Conclusions and recommendations
Non-response bias appears to be well contained in
consumer surveys, provided institutes make full use
of census data to post-stratify / calibrate survey
weights to ensure the responding sample is
representative of the population.
– The methodologies of Statistics Finland and Statistics
Slovenia reviewed in this study are cited as good practice
references
Nov 14-15 2005
Joint OECD / European Commission workshop on international
development in business and consumer tendency surveys
Conclusions and recommendations
If aggregation to the branch or cell level combines
businesses chosen with different probabilities or in
different ways (e.g. large and small businesses),
then an estimate of sampling fraction must be a
factor in the weighting process – otherwise survey
estimates will be biased.
– If institutes are not doing this, then small businesses are
most likely unrepresented in the survey estimates to such
an extent that their inclusion in the survey is irrelevant
Nov 14-15 2005
Joint OECD / European Commission workshop on international
development in business and consumer tendency surveys
Conclusions and recommendations
The limited research performed so far suggests that
the missing at random assumption for treating nonresponse in business tendency surveys may not
hold, therefore:
– Institutes are encouraged to undertake more research on
this issue, particularly through making estimates for
businesses with different response behaviours (e.g.
regular and irregular respondents)
– Institutes are encouraged to experiment in applying the
constant sample imputation methodology developed by
INSEE, which may be effective in reducing the impact of
non-response bias.
Nov 14-15 2005
Joint OECD / European Commission workshop on international
development in business and consumer tendency surveys