Collecting online:
How Statistics Netherlands moved its
business Surveys onto the internet
Ger Snijkers
thanks to
Deirdre Giesen, Hank Hermans, Myra Wieling
Overview
1. Overview of developments at Statistics Netherlands:
a. Electronic data collection
b. Organising data collection and data sharing
2. Case study:
Moving the Structural Business Survey to the web
a. Developing and testing the questionnaire
b. Communication strategy to launch the survey and
improving web-take up rates
c. Implementation of the survey
Lessons learned and challenges
3. Looking into the future: future challenges
15 October 2013
OSS Seminar, Statistics New Zealand, Wellington
2
Brief history of electronic data collection at SN
Before 2000:
• 1984
First PCs
• Mid 1980s Blaise 1.0
• 1993
E-reporting for International Trade: IRIS
-
disk-by-mail (floppy disks, CDs),
now offline using downloadable software
• Mid 1990s Electronic Data Interchange: EDIsent
-
• 1998
Simple establishment surveys
-
15 October 2013
software installed on business’ systems
e-mail
OSS Seminar, Statistics New Zealand, Wellington
3
E-mail questionnaire Business Tendency Survey 1999
15 October 2013
OSS Seminar, Statistics New Zealand, Wellington
4
Brief history of electronic data collection at SN (cont)
After 2000:
• 2000
Short Term Statistics (STS)
-
• 2003
• 2006
• 2006
Transportation Survey
STS – New HTML forms
Structural Business Surveys
-
2007
first electronic forms (HTML)
Pilot offline
Structural Business Survey in the field
-
• 2009
• 2014
offline downloadable software
Business Surveys online: Transportation Survey
• Next
- New integrated system for online surveys
- Blaise 5
EDI using XBRL integrated in questionnaires
• 2005
• 2008
Experiments for household surveys (on-line)
Household surveys on the internet (on-line)
15 October 2013
OSS Seminar, Statistics New Zealand, Wellington
5
Transportation Survey 2003
15 October 2013
OSS Seminar, Statistics New Zealand, Wellington
6
Offline SBS Survey 2007
15 October 2013
OSS Seminar, Statistics New Zealand, Wellington
more to come
7
Online Transportation Survey 2009
15 October 2013
OSS Seminar, Statistics New Zealand, Wellington
8
XBRL-based system (future)
15 October 2013
OSS Seminar, Statistics New Zealand, Wellington
9
Health Survey 2010
15 October 2013
OSS Seminar, Statistics New Zealand, Wellington
10
Business Data Collection at the moment
A few numbers:
•
•
•
•
•
 450 different questionnaires
 400 mixed-mode: electronic & paper
> 500,000 business units every year
> 750,000 questionnaires on average each year
 700,000 electronic questionnaires available
15 October 2013
OSS Seminar, Statistics New Zealand, Wellington
11
Drivers for using Web for Business Surveys
• From the data collection agency perspective:
-
data quality:
cleaner (fewer edits) and more complete
cost-efficiency: cheaper and less editing
timeliness:
faster
• From the business perspective:
-
internet penetration for businesses is high
much of the data held by businesses is already in digital form
many of the regulatory and administrative tasks they perform
are already online
can be easier to do than paper (if designed well)
 This only holds is surveys are designed well!
- the whole communication process: from end to end
15 October 2013
OSS Seminar, Statistics New Zealand, Wellington
12
Conclusions so far
Until 2007:
• Each survey for its own: stove-pipe approach
• No coordinated/harmonised design
• Have it in the field quickly
• Focus on technology, not on methodology/communication
No efficient questionnaire design and communication:
• Internally: every survey starts all over, no learning curve
• Externally: no professional questionnaire and survey
communication:
- various designs
- respondents have to learn every questionnaire again
15 October 2013
OSS Seminar, Statistics New Zealand, Wellington
13
Lessons learned
• Analyse contextual factors:
-
Internal constraints and processes
Response processes within businesses
• Focus on usability = success factor:
-
internally AND externally
technology AND methodology perspective
“We suspect that many of the survey organisations that
introduce web questionnaires forget that it is not the
technology in itself, but how it is utilised that determines
the result” ... both internally and within businesses
From: Haraldsen & Couper (2013)
• Apply Deming cycle – PDCA:
15 October 2013
also C & A: Evaluate and Improve
OSS Seminar, Statistics New Zealand, Wellington
14
Lessons learned (cont’d)
• Good design is to create benefits for both sides:
-
15 October 2013
Create internal conditions to be successful (under control)
> One Data Collection Division
Tailor to business conditions (not under control; try to influence)
> Moving SBS to the Web
OSS Seminar, Statistics New Zealand, Wellington
15
Overview
1. Overview of developments at Statistics Netherlands:
a. Electronic data collection
b. Organising data collection and data sharing
2. Case study:
Moving the Structural Business Survey to the web
a. Developing and testing the questionnaire
b. Communication strategy to launch the survey and
improving web-take up rates
c. Implementation of the survey
Lessons learned and challenges
3. Looking into the future: future challenges
15 October 2013
OSS Seminar, Statistics New Zealand, Wellington
16
Data Collection over the years
< 1994
1994 2000
Stove pipes
Data Collection 2000
2000 2007
Present
Partial Centralisation
15 October 2013
Data Collection Now
OSS Seminar, Statistics New Zealand, Wellington
17
Organizing Data Collection in one Division
Process-and-knowledge Driven approach
Clustering of practices and knowledge
• Focus on how knowledge is being used, for social and business
surveys
• Pilot 2007; effectuated 2011
Front desk: 12 fte
• Client relations
• Survey design
Design:
35 fte
• Questionnaire
• Sampling
• Training
15 October 2013
Survey deployment:
240 fte
• CAPI / CATI Interviewing
92
26
118 fte
• Planning & control
• Fieldwork logistics
• Support
• Special services
122 fte
OSS Seminar, Statistics New Zealand, Wellington
• Fte’s:
- 375 (2008)
- 277 (2012)
• Budget:
€ 18.5M
• 145 surveys
18
Drivers for centralizing data collection
Internal drivers
• Achieving efficiency by:
∘ Abolishing redundant processes, workflows, and
activities for social and business surveys
- Monitoring processes, workflows, and activities
∘ Abolishing redundant systems and tools
-
One data collection management system
Maintaining as less systems and tools as needed
• Implementation of Data Collection Strategy (2005)
External drivers:
• Professional external focus
• Reducing response burden
15 October 2013
OSS Seminar, Statistics New Zealand, Wellington
19
The 2011 Data Collection Strategy:
Retrieving and returning
(first version in 2005)
3 steps:
1. Re-use of available data
-
2.
Use of new registers and other
secondary sources
-
3.
Multi-source
designs
Traditional government-based registers
Big/organic data
Data on the internet (web-crawlers)
Primary data collection:
1.
2.
3.
•
•
15 October 2013
Data sharing & data warehousing
EDI technologies, like XBRL
Web surveys
traditional modes: paper, CATI, CAPI
Using new technologies
Reciprocity: report back to respondents
OSS Seminar, Statistics New Zealand, Wellington
Mixed-mode
designs
20
Data collection: Developments over the years
Summary and conclusions
• From stove-pipes to a coordinated system of data collection
• From uni-mode surveys to
multi-source/mixed-mode data collection designs
• From single-survey managers to managers of integrated
sets of statistics
• From local decision making to corporate decision making
• From instrumental data collection to a communication
perspective
 We are still in the middle of this process!
 A cultural shift needs years to change!
But … are we moving too slow?
15 October 2013
OSS Seminar, Statistics New Zealand, Wellington
21
Overview
1. Overview of developments at Statistics Netherlands:
a. Electronic data collection
b. Organising data collection and data sharing
2. Case study:
Moving the Structural Business Survey to the web
a. Developing and testing the questionnaire
b. Communication strategy to launch the survey and
improving web-take up rates
c. Implementation of the survey
Lessons learned and challenges
3. Looking into the future: future challenges
15 October 2013
OSS Seminar, Statistics New Zealand, Wellington
22
Moving to the Web: two basic approaches
• Copy all paper questionnaires to the web in a short period
of time, in one or various ways:
-
PDF, offline electronic questionnaire, online questionnaire
 Quick results, but how is the questionnaire utilised?
 High risks of: - web Qs not being used
- negative effects on public image
 Pushing problems and costs downstream
 Evaluate and improve, if you still get the chance ...
• Develop one approach and apply to all surveys:
-
investing in the organisation, technology, and methodology
investing in the ‘respondent experience’: survey communication
investing in PR: harmonised approach
 Takes time and costs money
 Higher chances of good results: web Qs actually used
15 October 2013
OSS Seminar, Statistics New Zealand, Wellington
23
Moving the SBS questionnaire to the web: approach
Approach by Stats Neth.: 2nd approach
• Develop one approach:
-
Good results for SBS …
 No systematic evaluations and improvements
• that is applied to all surveys:
Some follow-up, but not consistent, due to:
- organisation not yet ready (one of the main factors for project failure)
- still existing stove-pipes
 Needs a long-term vision that is followed up
 Assurance that investments are not lost
 Do it right: create benefits for both sides (or don’t do it all)!
 It is not going to be easy!
15 October 2013
OSS Seminar, Statistics New Zealand, Wellington
24
Moving the SBS questionnaire to the web: Project
Project goals and constraints:
• Develop a web questionnaire
-
same contents
mixed-mode design: coherent paper and web
support completion process
motivate respondents to use this mode
into the field: March 2006
• Start of project: June 2004
-
15 October 2013
we had one year, until Sept. 2005
OSS Seminar, Statistics New Zealand, Wellington
25
Dutch Annual Structural Business Survey
Some characteristics:
• Annual survey of economic activity, three parts:
-
revenues and costs
summary of business accounts: profits and losses
industry specific specifications
• Mandatory
• 75.000 business each year:
-
Sample of small businesses, bigger businesses each year
• Until 2006: paper questionnaire (A4 booklet):
-
15 October 2013
Length may differ, 20 pages is typical
Items are grouped in sections
Completion process: complicated and difficult:
lots of details asked for, mismatch of definitions,
various business departments involved, design issues
OSS Seminar, Statistics New Zealand, Wellington
26
Vernieuwde huisstijl:
2. Onderzoek naar de PS
–Q oud
15 October 2013
OSS Seminar, Statistics New Zealand, Wellington
27
Vernieuwde huisstijl:
2. Onderzoek naar de PS
–Q oud
15 October 2013
OSS Seminar, Statistics New Zealand, Wellington
28
3. 4 projects
3. Improving paper questionnaire
–Q nieuw
15 October 2013
OSS Seminar, Statistics New Zealand, Wellington
29
Developing the electronic questionnaire
Developing and testing in five stages:
1. Develop and test the prototype
-
pre-tests to test usability: 3 waves
field agents, business respondents
2. Revision of questionnaire
-
(1-1-’06)
additional usability tests
4. Implementation of field pilot
5. Implementation of survey
15 October 2013
(1-9-’05)
expert reviews
3. Testing of revised questionnaire
-
(31-1-’05)
OSS Seminar, Statistics New Zealand, Wellington
(1-3-’06)
(1-3-’07)
30
The electronic questionnaire: the prototype
Developing paper Q
and testing this
prototype helped in
thinking about a good
design for an eQ
15 October 2013
OSS Seminar, Statistics New Zealand, Wellington
31
Testing the prototype
Research issues:
1. How does the e-Q work in practice?
-
Completing the questionnaire
Question-and-answer process
Usability
2. What features should be included to make it easy to use?
-
Respondent friendly: ‘Computer-assisted’ tools
User demands
3. How should the eQ be designed in relation to the paper Q?
-
15 October 2013
The same or a different design
‘look-and-feel’ of paper and e-form
OSS Seminar, Statistics New Zealand, Wellington
32
Testing the prototype: Research issues
1. How does the Q work?
Laborious and complex process
• Long, complex questionnaire (≥ 25 items)
• Complex completion process:
-
several sessions, several informants
kick-and-rush behaviour
• Imagine ...
a respondent sitting behind his/her computer ...
 Respondent got lost in the questionnaire
15 October 2013
OSS Seminar, Statistics New Zealand, Wellington
33
Illustration of usability
“Medieval tech
support”
A new system
is introduced:
The ‘book’ (“beek”),
in stead of
rolls of paper.
The monk Ansgard
has called the
helpdesk and is
waiting for someone
to come …
15 October 2013
OSS Seminar, Statistics New Zealand, Wellington
34
Illustration of usability
“Medieval tech
support”
It illustrates what
we found in the
pre-tests for the
electronic
questionnaire …
15 October 2013
OSS Seminar, Statistics New Zealand, Wellington
35
Testing the prototype: Research issues
2. Features to make it easy to use?
• What am I supposed to do (next)?
-
•
How is the questionnaire built up?
-
•
Easy to download, install, complete, send data back
It is one process: going to the internet – submitting data,
Clear instructions and explanations (but not read)
Show how the questionnaire is structured: overview
Help to find the way in the questionnaire
No hidden rules, no unexpected functionalities
Where am I? What did I do so far?
-
Provide overview of the completion process
Clear navigation, no scrolling
Printing function
 Usability, navigation and overview: design of Tax forms
 Choose a design people know: don’t be original!
15 October 2013
OSS Seminar, Statistics New Zealand, Wellington
36
Testing the prototype: Research issues
3. Design of paper and web Q?
The computer is different than paper
• The web Q reacts to the respondent
• Reading from the screen is different
• Navigating and getting an overview works differently
• Kick-and-rush behaviour, even stronger than on paper
• The use of computer-assisted functionalities;
• the respondent expects the computer to react
 Start with paper, then add functionalities
15 October 2013
OSS Seminar, Statistics New Zealand, Wellington
37
Testing the prototype:
Conclusions of pre-test waves
•
Visual design
 Clear and logical: “Don’t make me think!”
 Simple, transparent, consistent
 No hidden and unexpected functionalities
•
Support the completion process, end to end
 Other mode, other features, other visual design
 Use computer functionalities: automatic calculations
 Use layout and usability principles that are known
to R (country specific)
•
Tailor to kick-and-rush behaviour
 Split Q into small sections, small tasks
 Short and clear explanations
15 October 2013
OSS Seminar, Statistics New Zealand, Wellington
38
Lessons learned: Web Design Guidelines
Web questionnaire design is
communication design
Questionnaire designer
=
Communication designer
Questionnaire communication design
15 October 2013
OSS Seminar, Statistics New Zealand, Wellington
39
The revised questionnaire
Based on:
• Pre-test results
• Expert reviews
Iterative process with
- Professional designer
- Questionnaire designers
- Methodologists
• A user-friendly design was put first, not the IT tool
• New prototypes designed in Power Point
15 October 2013
OSS Seminar, Statistics New Zealand, Wellington
40
15 October 2013
OSS Seminar, Statistics New Zealand, Wellington
41
The revised questionnaire: additional pretests
• 10 concurrent in-depth interviews
• Usability and user friendliness had been improved
 respondents enjoyed working with the questionnaire
 they could handle the task:
they recognised the design and usability
 even though … the task had not changed
15 October 2013
OSS Seminar, Statistics New Zealand, Wellington
42
Field pilot
Goals:
• Implementation of web questionnaire
-
-
test usability and completion process in the field:
downloading – completing – sending-in data
test communication strategy to get high web take-up rates
• Test the whole process
-
logistics for paper and web flows
• Response rates and web take-up rates?
• Data quality
-
15 October 2013
data editing, mode effects?
OSS Seminar, Statistics New Zealand, Wellington
43
Field pilot
Set-up:
• March-July 2006
• 7200 businesses, 5 industries
• Communication strategy:
-
internet portal: www.cbs.nl/productiestatistiek
advance letter with user name and password
leaflet to introduce web questionnaire
and explain why the survey is conducted
- paper questionnaire not mentioned
- only 2e or 3e reminding letters include paper questionnaire
In the Netherlands businesses have to report electronically to the Tax
Office.
15 October 2013
OSS Seminar, Statistics New Zealand, Wellington
44
Leaflet
15 October 2013
OSS Seminar, Statistics New Zealand, Wellington
45
Field pilot
Results:
• Web questionnaire worked well
• Web take-up rate: 80%,
total response rate: comparable
• No mode effects, but more research is needed
• Internal processes worked well,
but we missed checking the link to data editing.
 SBS questionnaire and communication design was
implemented by the end of 2006, and run in 2007
15 October 2013
OSS Seminar, Statistics New Zealand, Wellington
46
Communication strategy: guidelines
How to increase web take-up rate:
• Easy access to the web version combined with
restricted access to the paper version
• Offer real advantages
• Making the respondents aware of the web option
• Offer practical guidance
• Computer competence and motivation play a minor part
(if designed well)
15 October 2013
OSS Seminar, Statistics New Zealand, Wellington
47
For more guidelines …
15 October 2013
OSS Seminar, Statistics New Zealand, Wellington
48
Implementation of SBS Survey
• Running from 2007 onward
• About 75.000 business receive this questionnaire every
year
Web take-up rates:
• Dutch Annual SBS, 2007: 84 %
Total response rate did not change (80%)
• Dutch average 2008: 51%
3 largest Dutch business surveys (2008): 77 %
(STS: 63%, SBS: 80%, Intrastat: 100%)
15 October 2013
OSS Seminar, Statistics New Zealand, Wellington
49
Implementation of SBS Survey
https://formulieren.cbs.nl/downloads
per survey, per period, per business unit the log on codes
15 October 2013
OSS Seminar, Statistics New Zealand, Wellington
50
The portal http://formulieren.cbs.nl
15 October 2013
OSS Seminar, Statistics New Zealand, Wellington
51
Welcome page
15 October 2013
OSS Seminar, Statistics New Zealand, Wellington
52
In the index items that are complete and correct
according to the person who is filling in the form are
ticked off
15 October 2013
OSS Seminar, Statistics New Zealand, Wellington
53
Click on the ? and then a more extensive explanation
appears
15 October 2013
OSS Seminar, Statistics New Zealand, Wellington
54
Print of the SBS questionnaire
15 October 2013
OSS Seminar, Statistics New Zealand, Wellington
55
The confirmation of receipt
15 October 2013
OSS Seminar, Statistics New Zealand, Wellington
56
Paper version SBS questionnaire
15 October 2013
OSS Seminar, Statistics New Zealand, Wellington
57
SBS design in production
Using SBS design as basis for other surveys
• Visual design principles of new paper and web
questionnaires starting point for new style guide
-
for all 450 business questionnaires
for various modes: paper, offline, online
• Development of QUAT: Questionnaire Application Tool
-
15 October 2013
project 2005-2006
OSS Seminar, Statistics New Zealand, Wellington
58
SBS design in production
Questionnaire Application Tool (QUAT):
• Generic Tool for generating questionnaires:
-
Editing of questionnaires, letters, and attachments
Uses data bases of populations and variables
Allows for tailored questionnaires, per stratum
Allows for generating questionnaires in mixed-mode design:
paper, online, offline
Based on one set of visual design principles for each mode
• Uniform corporate identity concerning questionnaires
and letters (one look-and-feel)
• Input coordination with regard to variables and
classifications
• QUAT planner: for planning the dispatching of Qs
15 October 2013
OSS Seminar, Statistics New Zealand, Wellington
59
SBS design in production
• SBS design still in production
• But, offline system is out of date:
-
runs only on Windows PC – lot of complaints from Apple users
Software to be downloaded – no open source software
• QUAT is still running
• New system to be developed, on the same principles
15 October 2013
OSS Seminar, Statistics New Zealand, Wellington
61
Overview
1. Overview of developments at Statistics Netherlands:
a. Electronic data collection
b. Organising data collection and data sharing
2. Case study:
Moving the Structural Business Survey to the web
a. Developing and testing the questionnaire
b. Communication strategy to launch the survey and
improving web-take up rates
c. Implementation of the survey
Lessons learned and challenges
3. Looking into the future: future challenges
15 October 2013
OSS Seminar, Statistics New Zealand, Wellington
62
Future challenges for statistics
Global developments:
• Emergence of the information society
• Globalization
• Shift in the balance of power between NSIs and businesses
•
•
Erikson, Haraldsen & Snijkers, 2012, UNECE Seminar, Geneva
Snijkers & Haraldsen, 2013, BLUE-ETS final Conference, Brussels
15 October 2013
OSS Seminar, Statistics New Zealand, Wellington
63
Challenges in modernising statistics
Based on global developments and data coll. strategy:
• Expanding usage of registers
-
data sharing and data warehousing
assessing quality of register-based statistics
• Blending designed and organic data
-
data integration theories
• Modernising current business survey designs
-
tailoring design to the business context, including
collecting data from globalised establishments
using paradata
adopting a communication perspective:
“Improving the Respondent Experience”
• Producing statistics in a globalized world
15 October 2013
developing new statistical indicators
market position of an NSI?
OSS Seminar, Statistics New Zealand, Wellington
64
Consequences …
• Challenges are too big for individual NSIs:
- collaborate on international level
- collaborate with universities
• Position in the information market: Our added value?
- “slow”
or/and “fast” statistics?
accurate or/and timely statistics?
- NSIs have access to and can combine many data sources:
new statistics
- Bench marking of non-NSI statistics
- National Data Archive? National Data Service Center?
15 October 2013
OSS Seminar, Statistics New Zealand, Wellington
65
Future challenges for statistics
Two final statements:
 We are at the beginning of a new era in statistics
production:
The era of data integration
 Are we moving too slow?
The future is already here!
15 October 2013
OSS Seminar, Statistics New Zealand, Wellington
66
References
• Beukenhorst, D.J., and D. Giesen, 2010, Internet Surveys at Statistics
Netherlands. Paper presented at the 2nd International Workshop on
Internet Survey Methods, Statistics Korea, 8-9 september 2010,
Deajon, South korea.
• Haraldsen, G., and M. Couper, 2013, How to Design Effective Business
Web Surveys, Course at the Nordic Meeting, 14 August 2013.
• Haraldsen. G., and G. Snijkers, 2013, Modernisation and Quality of
Business Statistics: The NSI Perspective. Paper presented at the 3rd
European Establishment Statistics Workshop, 9-11 September 2013,
Nuremberg, Germany.
• Groves, R.M. (2011), Three Eras of Survey Research. Public Opinion
Quarterly, Vol. 75, pp. 861-871.
• Erikson, J., G. Haraldsen, and G. Snijkers, 2012, The Future of
Statistical Data Collection? Challenges and Opportunities. Paper
presented at the UNECE Seminar on New Frontiers for Statistical
Data Collection, 31 October-2 November 2012, Geneva, Switzerland.
15 October 2013
OSS Seminar, Statistics New Zealand, Wellington
67
References
• Snijkers, G., H. Haraldsen, J. Jones, and D. Willimack, 2013, Designing
and Conducting Business Surveys. Wiley, Hoboken.
• Snijkers, G., H. Hermans, and R. Göttgens, 2011, Data Collection and
Sharing at Statistics Netherlands, Paper presented at the CES
meeting, 14-16 June 2011, UNECE Geneva, Switzerland.
• Snijkers, G., E. Onat, E., and r. Vis-Visschers, 2007, The annual
structural business survey: Developing and testing an electronic form,
Proceedings of the 3rd International Conference on Establishment
Surveys (ICES-III), Montreal, June 18–21, American Statistical
Association, Alexandria, VA, pp. 456–463.
• Wieling, M., 2013, Electronic business questionnaires at Statistics
Netherlands, Presentation at Statistics Netherlands for guests from
Statistics Canada, 7 June 2013, Heerlen, Netherlands
15 October 2013
OSS Seminar, Statistics New Zealand, Wellington
68