The Future of Statistical
Data Collection?
Challenges and Opportunities
Johan Erikson (Statistics Sweden)
Gustav Haraldsen (Statistics Norway)
Ger Snijkers (Statistics Netherlands)
Seminar on New Frontiers for Statistical Data Collection
31 October – 2 November 2012, Geneva, Switzerland
Outline
1. Fundamental challenges
2. How do we meet these challenges?
3. Are we prepared?
• Traditional survey perspectives:
-
Management
Methodology
• Beyond the traditional survey framework
4. Conclusions
New Frontiers for Stastictal Data Collection, 2 November 2012, Geneva
2
1. Fundamental challenges
Official Statistics in a changing world!
Three main challenges:
1. A shift in the balance of power between
survey organisations and respondents
2. New competitors may make Official Statistics
redundant
3. Globalisation of the economy
New Frontiers for Stastictal Data Collection, 2 November 2012, Geneva
3
A shift in the balance of power
• Dillman et al. (2009):
“Surveys are now respondent driven,
rather than driven primarily by the needs of
survey organizations.”
New Frontiers for Stastictal Data Collection, 2 November 2012, Geneva
4
A shift in the balance of power
•
Traditionally – data collectors in control
- Responding to survey was seen as a duty – not any more
- The change can be seen clearly by studying the attitude of
respondents in different age groups
•
•
•
Business organisations focus more on production and
competiveness
The means of communicating are changing
Respondents are getting harder to reach, and can
control who they want to be reached by
Conclusion:
We need to adapt to respondents more and more in order
to get their cooperation using all kinds of comm. means.
New Frontiers for Stastictal Data Collection, 2 November 2012, Geneva
5
New competitors
• Groves (2011) - “Organic vs. designed data”:
“We’re entering a world where data will be the
cheapest commodity around, simply because
society has created systems that automatically
track transactions of all sorts.”
• E.g. transactions on social media:
3.2 million new entries/day, in the Netherlands
New Frontiers for Stastictal Data Collection, 2 November 2012, Geneva
6
New competitors
•
•
Data are everywhere: “Big Data” Conferences
New competitors use these data to produce new
statistics/information in real time:
- E.g. Google’s stock prices index (GOOG)
•
This challenges the way traditional (NSI) statistics are
produced:
- founded in survey methodology, with a focus on quality
•
The new information maybe directed at different users
Conclusion:
•
•
What is our position in the information market?
Can we produce timely, relevant and competitive stats?
New Frontiers for Stastictal Data Collection, 2 November 2012, Geneva
7
Globalisation
• UNECE (2011) - The Impact on globalization
on National Accounts:
“The increasingly global nature of economic
transactions and arrangements presents a
challenge to the application of national
accounts concepts and the use of data
collection and compilation systems for
measuring developments in
the domestic economy.”
New Frontiers for Stastictal Data Collection, 2 November 2012, Geneva
8
Globalisation
• Traditional indicators that show a society’s development
may no longer be valid
• The concepts and units (needed to compile these indicators)
are becoming irrelevant to globalised businesses:
- If the concepts are irrelevant to providers, data will be hard to get
- Offices in any part of the world may have to be contacted
Conclusion:
• The definitions, measurement and compilation of
traditional indicators needs to be reviewed
• The concepts and units need to be revised,
• The data collection methods and procedures need to be
adapted in international perspective (relates 1st challenge)
New Frontiers for Stastictal Data Collection, 2 November 2012, Geneva
9
2. What is done?
The answer to these challenges so far
Data Collection Strategy - 3 steps:
1. Re-use of available data
•
Data sharing & data warehousing
2. Use of new registers and other
secondary sources
•
•
•
Traditional government-based registers
Information from private businesses
“Organic/Big Data” sources
3. Primary data collection:
1.
2.
3.
•
•
EDI technologies, like XBRL
Web surveys
traditional modes: paper, CATI, CAPI
Using new communication technologies
Reciprocity: report back to respondents
Multi-source
designs
Mixed-mode
designs
New Frontiers for Stastictal Data Collection, 2 November 2012, Geneva
10
3. Is this sufficient?
Are we prepared re: challenges?
• To answer this question we looked at the
internal processes in our Statistical Offices
• What are we actually doing …?
• What is our short-term focus?
• The Quality Diamond:
(Haraldsen; in: Snijkers, Haraldsen, Jones & Willimack, 2013, Wiley)
Integrates four quality perspectives on statistics:
New Frontiers for Stastictal Data Collection, 2 November 2012, Geneva
11
The Quality Diamond
• Management perspective
• Methodology perspective
Specifications
Coverage
Sampling
Nonresponse
Measurement
Processing
Costs
Technology
Ethics
Timeliness
PRB
Question order effects
Mode effects
effects
New Frontiers forCross-cultural
Stastictal Data Collection,
2 November 2012, Geneva
Relevance
Accuracy
Timeliness
Punctuality
Accessibility
Clarity
Comparability
Coherence
(Eurostat Quality
Dimensions, 2011)
12
The Quality Diamond:
The Management Perspective
6 Managing
customer
demands and
expectations
Non-response
1 Costs
2 Timeliness
3 Flexibility of process planning
4 Innovation of process
5 Response Burden
7 Planning staff
8 Systems and tools
9 Culture
Mixed-mode effects
Data
Collection
effects
New Frontiers for
Stastictal
Data Collection,Strategy
2 November 2012,
Geneva
13
The Quality Diamond:
The Methodology Perspective
Accuracy
Non-response
Measurement
Coverage
Sampling
Response Burden
Innovation of data
collection methods
Mixed-mode effects
Questionnaire effects
New Frontiers for Stastictal Data Collection, 2 November 2012, Geneva
Data Collection Strategy effects
14
Differences in focus:
• Management (mostly) interested in becoming better
(more efficient) within traditional survey framework.
Short-term challenges, while running the surveys:
-
Standardisation and integration of processes
Innovating the current processes within the Data
Collection Strategy, like planning mixed-mode designs
• Methodology driven by quality considerations:
-
Reducing survey design effects and Total Survey Error
Survey process constraints not much considered
• The two perspectives hardly overlap
New Frontiers for Stastictal Data Collection, 2 November 2012, Geneva
15
Beyond the traditional survey framework
• Short-term focus – based on traditional
framework
 We need to go beyond that framework:
Considering challenge 1: balance of power
• The end of probability sampling?
- High non-response levels and low representativeness:
responsive designs, web panels, post-survey adjustments
- Use of registers: no sampling involved; non-sampling errors?
Towards multi-source/mixed-mode designs
• The end of standardised questionnaires/data collection?
- From collecting data to collecting {data & metadata} using
data capture methods (like XBRL)
- From standardised interviewing to conversational interviewing
and data mining, using new technologies (smart phones, Skype)
New Frontiers for Stastictal Data Collection, 2 November 2012, Geneva
16
Beyond the traditional survey framework
• Official data requirements and concepts – based
on stove-pipe approach: traditional indicators,
separate surveys
 We need to go beyond that:
• The end of traditional indicators to measure societal
developments?
Considering challenge 3: Globalisation
- What information do we need?
- How do we measure that information?
• The end of official statistics?
Considering challenge 2: New competitors
- What will be our position in the information market?
New Frontiers for Stastictal Data Collection, 2 November 2012, Geneva
17
4. So, are we prepared?
• Standardisation and integration of data collection
processes are a step in the right direction
… but not enough
• Data Collection Strategies towards multi-source/mixedmode designs are a step in the right direction
… but not enough
 Managers and methodologists need to collaborate,
having common short-term and long-term focuses:
a communication perspective!
 We need to go beyond the traditional survey framework
… and act now! The future is already here!
New Frontiers for Stastictal Data Collection, 2 November 2012, Geneva
18