JOINT EUROPEAN COMMISSION – OECD WORKSHOP ON INTERNATIONAL
DEVELOPMENT OF BUSINESS AND CONSUMER TENDENCY SURVEY
Task Force on Improvement of Response Rates and
Minimisation of Respondent Load
Part 1 – Relationship between response rates
and data collection methods
Bianca Maria Martelli,
Institute for Studies and Economic Analysis -ISAE – Rome, Italy
BRUSSELS, 14-15 November 2005
TERMS OF REFERENCE
1. DATA COLLECTION METHODS
which is the most effective for:
•
•
•
•
•
Improving response rates
Reducing burden
Impact on costs
Sectoral effects
Size effects
2. COMMUNICATION METHODS
most effective and cost efficient for:
•
•
2
Compulsory surveys
Voluntary participation
Joint OECD/European Commission Workshop on International
Development in Business and Consumer Surveys 14-15 November 2005
ACKNOLEDGEMENTS
The present research has been realised with
the substantial contributions of the
members of the EC/OECD task force on RR
who supported me with extremely useful
material , references, information, personal
experience and suggestions.
3
Joint OECD/European Commission Workshop on International
Development in Business and Consumer Surveys 14-15 November 2005
WHY IS THE RESPONSE RATE IMPORTANT?
• Response Rate (RR) one of the most important
factors to ensure a high quality of the BTS results
(Etter, 2002)
• Non response rate (NR) is one of the components
of non-sampling error that may affect the survey
and the most efficient strategy for reducing
measurement errors is to eliminate possible causes of
such errors during the whole survey design stage
(OECD 2003)
4
Joint OECD/European Commission Workshop on International
Development in Business and Consumer Surveys 14-15 November 2005
WHY FIRMS/CONSUMER RESPOND?
Conceptual framework for COS/BTS Survey
participation (proposed by Willimack 2002, reported by Petroni
2004):
• Businesses weight response burden against
business goals when considering the decision to
participate in a survey. Both burden and goals may
be impacted by:
– factors relating to the external environment, the business
and the respondent, which are not under control of the
survey organisation
– survey design features, which are under the
survey organisation control
5
Joint OECD/European Commission Workshop on International
Development in Business and Consumer Surveys 14-15 November 2005
FACTORS INFLUENCING RESPONSE RATES
Kind of Survey
Subject
Target Universe
Reporting,
Sampling,
Response
Unit
RR
Questionnaire
Frame
Sampling design
Data Collection Mode
6
Joint OECD/European Commission Workshop on International
Development in Business and Consumer Surveys 14-15 November 2005
WHAT IS (NON) RESPONSE RATE?
• CONFUSION ON THE TOPIC :
people deals with RR/NR, but not explain what really
apply
• NEED OF CLEAR RR/NR DEFINITION
The response rate is the number of
complete/incomplete interviews with reporting
units divided by the number of eligible
reporting units in the sample (AAPOR 2004)
• SEVERAL KIND OF RR/NR
• UNIT / ITEM NR
7
Joint OECD/European Commission Workshop on International
Development in Business and Consumer Surveys 14-15 November 2005
KINDS OF NON RESPONSE RATES
OECD DEFINITIONS (2003) (1/3):
 n  *100
NR1  n
'
where n’ = number of enterprises which did not submit useable
information
n = number of enterprises in the survey
8
Joint OECD/European Commission Workshop on International
Development in Business and Consumer Surveys 14-15 November 2005
KINDS OF NON RESPONSE RATES
OECD DEFINITIONS (2003)(2/3):
In case of unequal sampling fraction but uniform weights a proper measure
is the following:
n'
1

fi
i 1
NR 2  n
1

fi
11
where
*100
ni
fi 
n
is the sampling fraction of the ith unit
9
Joint OECD/European Commission Workshop on International
Development in Business and Consumer Surveys 14-15 November 2005
KINDS OF NON RESPONSE RATES
OECD DEFINITIONS (2003) (3/3):
In the most general case, of unequal sampling fraction and reporting units
with different weights:
n'
1
* wi

i 1 f i
NR3  n
*100
1
* wi

11 f i
where wi is the size weight of the ith unit
10
Joint OECD/European Commission Workshop on International
Development in Business and Consumer Surveys 14-15 November 2005
KINDS OF (NON) RESPONSE RATES
OTHER LITERATURE ON RR/NR DEFINITIONS (AAPOR 2004)
• COOP= Co operation rate
Proportion of all cases interviewed of all eligible units ever contacted
• REF = Refusal Rate
Proportion of all cases in which a respondent refuses to do an
interview, or breaks-off an interview of all potentially eligible cases
• CON = Contact Rate
Proportion of all cases in which a respondent refuses to do an
interview, or breaks-off an interview of all potentially eligible cases
11
Joint OECD/European Commission Workshop on International
Development in Business and Consumer Surveys 14-15 November 2005
KINDS OF (NON) RESPONSE RATES
OTHER LITERATURE ON RR/NR DEFINITIONS
• OVERALL COMPLETION RATE (Lozar Manfreda,
2003)
% of responses among all sent invitations or exposed to
invitation/eligible, including partial and complete respondents
The author assumes that most often researchers when dealing with
RR actually refer to it (similar to OECD NR1)
• FULL COMPLETION RATE
Considers only the % of complete (unit) respondents
• DROP-OUT RATE
Referring only to item non response
• CLICK-OUT RATE
For Web surveys (% of accessing the Web questionnaire among all
invited)
12
Joint OECD/European Commission Workshop on International
Development in Business and Consumer Surveys 14-15 November 2005
RECOMMENDATION ON (NON) RESPONSE
RATES
• Clearly specify which kind of response rate
is applied
According to the specific work purposes of this paper:
• OECD NR1 (or OVERALL COMPLETION
RATE)
is useful in the first steps of the survey design to check
the robustness of the data collection process.
• FULL COMPLETION RATE , DROP OUT RATE,
CLICK TROUGH are also useful for analysing the DCM
13
Joint OECD/European Commission Workshop on International
Development in Business and Consumer Surveys 14-15 November 2005
DATA COLLECTION MODES TECHNIQUES
• FACE TO FACE
–Historically one of the first applied (since 1912)
–Mainly for COS
–High quality, High costs
• CAPI/CASI
Computer Assisted Personal Intervieving/Computer
Asssisted Self- Interviewing
–“Computerised” version of face-to face
• MAIL
–The first historically applied (1788 as reported by De Leeuw,
2005)
– Implies the self-administration of questionnaire
– Visual approach
– “Low” costs, relatively low RR
•
14
Joint OECD/European Commission Workshop on International
Development in Business and Consumer Surveys 14-15 November 2005
DATA COLLECTION MODES TECHNIQUES
• FAX
– Similar to MAIL
– Widely used for BTS
• TELEPHONE
–
–
–
–
Useful as supporting measure
“Aural” (instead of “visual”)contact: risk of “loss of context”
Primacy/recency effects (for COS)
Primary role of interviewer
• CATI (Computer Aided Telephone Interview)
–
–
–
–
15
Widely used for COS, less for BTS (Italy, CB)
Allows to quickly collect information
Allows immediate control of inconsistencies
Expensive
Joint OECD/European Commission Workshop on International
Development in Business and Consumer Surveys 14-15 November 2005
Focus: CATI AT ISAE for BTS
• Since 1988 for Business (since 2002 completely)
• Since the beginning (1992) for Services
plus:
• Rapidity
• Reliability
• High response rate
• Less sample biases
• Easy substitution
16
minus:
• Costs (in comparison
with postal tecnique)
• Possible subjective
biases
Joint OECD/European Commission Workshop on International
Development in Business and Consumer Surveys 14-15 November 2005
Focus: CATI AT ISAE for BTS
ISAE Manufacturing and Services CATI Interviews structure
Outcome
Number of phone calls
MANUFACTURING (sample size
4100 units)
1- Accepted interview (explanatory material received)
SERVICES
(sample size 2100
units)
4032
1846
2 - Accepted interview (expl. material non received)
31
93
3 - Accepted first interview
46
36
4 - Refusal
69
64
5 - Unreachable
129
730
6 -Out of target
4
2
Failure
7 -No response
75
8 -Phone busy
37
9 - Automatic replayer
10 - Wrong phone number
51
72
11 - Already contacted
1
12 - No more active
7
13 – Exceeding quotas
Total phone calls
105
4359
Monthly average January – June 2005
17
9
Joint OECD/European Commission Workshop on International
Development in Business and Consumer Surveys 14-15 November 2005
3076
DATA COLLECTION MODES TECHNIQUES
• E - MAIL
–
–
–
–
“Visual”approach, similar to MAIL and FAX
Decreasing trend
Low cost
Confidentiality problems (not allowed at INSEE )
• TOUCH-TONE Data Entry/VOICE RECOGNITION
Entry
– For COS
– Low RR
– Low cost
18
Joint OECD/European Commission Workshop on International
Development in Business and Consumer Surveys 14-15 November 2005
DATA COLLECTION MODES TECHNIQUES
• WEB/CAWI
–
–
–
–
–
Increasing relevance for BTS (less for COS)
Low cost
Coverage errors
Risk of loss of context in the questionnaire
Higher Unit NR/Lower Item NR (vs MAIL, Annaud, 2005,
Etter)
– Self selection bias
– Only a part of participants (30-50%) freely choose this mode
(Etter, Stangl, Kershoff)
19
Joint OECD/European Commission Workshop on International
Development in Business and Consumer Surveys 14-15 November 2005
DATA COLLECTION MODES PROCESS
DCM often is a MULTISTAGE PROCESS and for
any step more than one mode can be applied
STEPS
(timing of interaction)
– CO OPERATION / PRE
SURVEY NOTIFICATION
– OVERALL RESPONSE
STAGE
– FULL COMPLETION
– REMINDINGS
– FOLLOW UPs
20
MODES
the different modes can be
applied :
- CONCURRENT
and/or in a:
- SEQUENTIAL way
Joint OECD/European Commission Workshop on International
Development in Business and Consumer Surveys 14-15 November 2005
DATA COLLECTION MODE: MIXED MODE
(De Leeuw, 2005)
• MULTIMODE SYSTEM
• OPTIMIZE data collection procedures
• REDUCE total survey ERRORS within the
available time and costs
21
Joint OECD/European Commission Workshop on International
Development in Business and Consumer Surveys 14-15 November 2005
MIXED MODE DATA COLLECTION
EFFECTS ON BURDEN, COSTS, RR
FIRM
INSTITUTION
REDUCES BURDEN
INCREASES RR
REDUCES PANEL
ATTRITION (TIME IN
SAMPLE)
INCREASES BURDEN
INCREASES VOLUNTARY
PARTICIPATION
ERRORS DUE TO MODE
EFFECTS
22
INCREASES COSTS
Joint OECD/European Commission Workshop on International
Development in Business and Consumer Surveys 14-15 November 2005
SECTOR AND SIZE EFFECTS
SECTOR
• Firms specific effect is
present
– Differences (for both Mail
and Internet) (Etter,
2002)
– Differences (for both
CATI and FAX), Italy
• …..but limited
• Retail, Services lower
RR (but also smaller
size…)
23
SIZE
• Generally inverse
correlation between RR
and Size
– medium/large firms respond
more regularly (Etter, 2002)
– Burden is greater for small
firms (Petroni, 2003)
• Some exception
– South Africa (Kershoff, 2005),
Latvia do not notice
remarkable influence
Joint OECD/European Commission Workshop on International
Development in Business and Consumer Surveys 14-15 November 2005
COMMUNICATION METHODS (CO OPERATION STAGE)
• Action useful in getting co operation (and
reducing NR):
– Tailoring contacts to the characteristics of the
establishment
– Contacting the “right” person (mid level management)
– Overcoming the lack of awareness (of the Survey /
Institution)
– Building the initial relationship (personal contact
advisable)
– Knowledge of survey characteristics
– Survey pre notice
24
Joint OECD/European Commission Workshop on International
Development in Business and Consumer Surveys 14-15 November 2005
COMMUNICATION METHODS: INTERVIEWERS
(CO OPERATION STAGE)
It is strongly advisable:
• Use of interviewers
– in the initial step, even if the ongoing DCM is different,
increases RR (BLS Survey)
• Training of interviewers
– Mainly for BTS, less for COS
– Overcome gate keepers: difficult first contact does not
necessarily mean unwillingness to participate
25
Joint OECD/European Commission Workshop on International
Development in Business and Consumer Surveys 14-15 November 2005
COMMUNICATION METHODS
(CO OPERATION STAGE)
all the above mentioned actions are essential also for
effectively communicating:
• COMPULSION (OF RESPONSE)
as it may increase the (perceived) burden of
respondent
(Mixed ) Effects of compulsions :
– Increases the Overall Completion Rate (lowers the Unit
NR)
– May increase the Drop Outs (Item NR)
– Positive effects in France, mainly for large firms
– Positive effects in Italy (Retail)
26
Joint OECD/European Commission Workshop on International
Development in Business and Consumer Surveys 14-15 November 2005
COMMUNICATION METHODS
(CO OPERATION STAGE)
• Voluntary participation
–
–
–
–
27
most effective approaches :
Sending the business survey to named individuals and
use telephone follow-up methods to encourage responses
(Paxon, 1995)
Lowering the response burden (Linn 2003)
Increasing the perceived relevance of the survey to
business goals (Linn, 2003)
Adopting a Mixed DCM
Joint OECD/European Commission Workshop on International
Development in Business and Consumer Surveys 14-15 November 2005
OVERALL RESPONSE STAGE : PANEL ATTRITION
• Maintain contacts with participant
• Renewing interest
• Using simple and plain questionnaires
– Avoiding open ended and “difficult to answer questions”
• Question wording
– accuracy, mainly when using concurrent DCMs
• Remindings
• Follow – ups
28
Joint OECD/European Commission Workshop on International
Development in Business and Consumer Surveys 14-15 November 2005
CONCLUSIONS AND RECOMMENDATIONS
•
Efforts should be made in improving
preliminary contacts and remindings / follow
up actions
•
•
All the modes have plusses and drawbacks
The starting point of view should be the firm
(to a lesser extent the consumer): adopting
a company-centric point of view
– Efforts should be made to allow surveys’
participants to choose the mode they prefer
29
Joint OECD/European Commission Workshop on International
Development in Business and Consumer Surveys 14-15 November 2005
CONCLUSIONS AND RECOMMENDATIONS
•
Although Internet (online) seems to be the
mode of the future, a large part of firms still
prefer the most traditional mail
questionnaire and a significant part of
respondents prefer fax (habit persistence)
• The future appears to be even more bound
to a mixed data collection mode
– Mixed mode data collection should be carried
out both concurrently and in a multi stage
way (as far as possible bound to BTS and COS
timeliness)
30
Joint OECD/European Commission Workshop on International
Development in Business and Consumer Surveys 14-15 November 2005