1 Third International Conference on Establishment surveys. Montréal 2007 Web and Paper Questionnaires seen from the Business Respondent’s Perspective Gustav Haraldsen, Statistics Norway Jacqui Jones, UK Office of National Statistics 1 Outline • A quality driven approach to response burden • The effect of survey design, external conditions and personal characteristics on perceived response burden • The impact of perceived response burden on data collection and data quality • (The perceived response burden of web versus paper questionnaires) 2 Origins and Effects of Perceived Response Burden in Establishment Surveys Survey Requirements Survey Design Sample Stakeholder (s) Specification Frame Sampling procedure Size Response environment Response quality Context: Encoding/record formation Respondent selection/identification Priorities Questionnaire Survey organisation Length Structure Content Layout Procedure Contact mode(s) Persuasion strategies Management Confidentiality Respondent(s) Release • Availability • Motivation • Competence Revision 3 The quality driven approach • Response burden is mainly a problem as long as it affects the response quality • It is the respondents’ response burdens that matters • Perceived response burden is a more relevant variable than completion time • The response burden may be affected by external conditions or personal characteristics of the respondents as well as by the survey design. • Perceived response burden questions should identify why questionnaires are burdensome 4 Core PRB Questions Do you think it was quick or time consuming to collect the necessary information to complete the questionnaire? +1 +0.5 0 -0.5 -1 Very quick Quite quick Neither quick nor time consuming Quite time consuming Very time consuming These were asked to pick the main reason(s) from a list or specify other reason(s) Did you find it easy or burdensome to complete the questionnaire? +1 +0.5 0 -0.5 -1 Very easy Quite easy Neither easy nor burdensome Quite burdensome Very burdensome These were asked to pick the main reason(s) from a list or specify other reason(s) 5 Sources of perceived response burden • Survey Design Factors – Characteristics of 6 web questionnaires directed to different industries – (The time it took to fill in the questionnaire) • External conditions – Size: The number of employees – Information sources: Documentation problems that made the information collection time consuming – Division of labour: The number of persons who took part in the response process • Personal Characteristics – Availability: How easy or difficult it was to set aside time for the questionnaire – Motivation: How useful for the business and society the respondent felt that statistics produced from the data were – Competence: If he/she was a first time or experienced respondent 6 2006 Web Response and PRB Response for Annual Business Surveys Web response PRB response Hotels and restaurants 31 % 43 % Transport & communication 31 % 40 % Sea transport 41 % 30 % Service industries 33 % 45 % Domestic trade 33 % 44 % Construction 31 % 43 % 7 Perceptions of the Information Collection and Completion of the 2006 Questionnaires Collecting Information and Completing Business Questionnaires Index Minutes Collecting 0,01 98 Completing 0,10 55 0% 20 % -1 40 % -0,5 60 % 0 80 % 0,5 100 % 1 8 Sources of Time Consuming Preparations Other reasons 21,1 % Waiting for information 21,1 % Many questionnaires 21,3 % Need for help 32,7 % 72,3 % Different sources 0% 20 % 40 % 60 % 80 % 100 % 9 Sources of Response Burdens E-technical 5,8 % Other 7,9 % E-usability Terms 8,8 % 15,9 % 22,3 % Layout No. of questions 31,3 % 34,0 % Response categories 46,2 % Mismatch 48,8 % Calculations 0% 20 % 40 % 60 % 80 % 100 % 10 Information collection and Perceived response burden index: Industries and completion time IC-index PRB - index Construction -0.22 -0.17 Sea transport 0.04 (0.02) Transport and communication 0.03 0.09 Hotels & restaurants -0.05 0.10 Domestic trade 0.01 0.10 Service industry 0.12 0.32 Completion time 0 – 25 minutes 0,32 26 – 50 minutes 0,11 51 minutes + -0,16 11 PRB-index: External conditions PRB- index Size of 0 – 9 employees establishment 10 – 19 employees (0,13) 20 – 49 employees 0,06 50 or more employees 0,10 Documentation None 0,15 problems 1 (-0,14) 2 ((-0,23)) 3 -0,45 No of 1 0,15 Respondents 2 or more 0,11 ((-0,01)) 12 PRB-index: Personal characteristics PRB- index Available time Very easy to find 0,68 Quite easy to find 0,41 Neither easy nor difficult 0,06 Quite difficult -0,09 Very difficult -0,37 Perception of Very useful ((0,22)) usefulness Quite useful 0,30 Neither useful nor useless 0,18 Quite useless 0,09 Very useless -0,11 No 0,04 Yes 0,11 Experience 13 The effect of survey design, external conditions and personal characteristics on PRB • Calculations and mismatch between questions and available • • • • • information are general problems in business surveys Sufficient time available for the questionnaire is essential for the perceived response burden. The number of questions was probably the most important difference between questionnaires Poor layout and usability problems were often mentioned by those who felt that the questionnaire was very burdensome The quality of documentation systems is probably the most important difference between industries The perception of uselessness goes together with perceived response burden 14 Web Response Patterns for Construction and Service Industry Questionnaires 100 % Service 80 % Construction 60 % 40 % 20 % de ad l O ine ct ob er Fi na l in de r 2. re m Au gu st de ad li June 1. ne re m in de r 2. de J ad uly lin e 1. M ay 0% 15 Questionnaires corrected Construction 80,9 % Sea transport 53,3 % Transport and communication 63,3 % Hotels & restaurants 79,7 % Domestic trade 70,1 % Service industry 33,4 % 16 The impact of perceived response burden on data collection and data quality • Generally in business surveys.. – the data collection time is very long and – the number of questionnaires that are corrected is very high • When there is a pronounced difference in perceived response burden between questionnaires, there also seems to be a difference in… – How quickly the data are received – The number of questionnaires that need to be corrected • Problems with collecting the information asked for may also have an effect on these quality indicators 17 Perceived response burden of the paper and web versions of the Annual Manufacturing Industry Web versus surveys (2004 and 2005)paper PRBindex Multi business web 0,01 Multi business paper -0,08 One business web 0,39 One business paper 0,32 0% 10 % 20 % 30 % 40 % 50 % 60 % 70 % 80 % 90 % 100 % -1 -0,5 0 0,5 1 18 Sources of response burden in paper and web versions of the Manufacturing Industry Surveys One business Multi business Paper Web Diff Paper Web Diff Mismatch 53 53 0 72 67 5 Calculations 18 39 -21 36 56 -20 Terms 24 29 -5 22 78 -56 No of questions 5 10 -5 21 44 -23 Response categories 14 18 -4 21 22 -1 Layout 1 4 -3 3 22 -20 E-technical/usability 0 6 -6 0 11 -11 Other 6 14 -8 5 6 -1 SUM 121 173 -52 180 306 -126 19 (The perceived response burden of web versus paper questionnaires) • When the Manufacturing Industry Survey changed from paper to web… – The perceived response burden apparently went down – While the number of problems reported by those who felt the questionnaire burdensome seemed to go up A quote for reflection: “It takes 100 men to do as much harm as a computer can make within 1/100 of a second” 20