Quality assurance activities
at EUROSTAT
CCSA Conference
Helsinki, 6-7 May 2010
Martina Hahn, Eurostat
Structure of the presentation
 Code of Practice
 Institutional environment
 Process quality
 Output quality
 ESS Quality Architecture
7/23/2016
2
European Statistics Code of Practice
3
Content of the Code of Practice
3 headings – 15 principles – 77 indicators
 Institutional environment
Professional independence, Mandate for data collection, Adequacy
of Resources, Quality commitment, Statistical confidentiality,
Impartiality and objectivity
 Statistical processes
Sound methodology, Appropriate statistical procedures, Nonexcessive burden on respondents, Cost effectiveness
 Statistical output
Relevance, Accuracy and reliability, Timeliness and punctuality,
Coherence and comparability, Accessibility and clarity
4
Institutional environment
 ESS-wide peer reviews during 2006-08
– to monitor the implementation of the Code of Practice
– to boost quality, enhance credibility
– to share ESS best practices and transfer knowledge
 Establishment of the European Statistical Governance
Advisory Board (ESGAB):
– to provide an independent overview of the implementation of the
Code of Practice by Eurostat
– to advise Eurostat and the ESSC on updating and communicating
the Code to users and data providers
 Establishment of general guidelines (e.g. confidentiality
manual)
5
6
ESS Quality Coaching
 Two quality managers work together to address specific
improvement actions
 Based on experience, dialogue and information exchange
 Agreement on commitments from both sides
 Joint working – not doctor/patient relation
 17 coaching pairs took part
 Coaching exercise completed at the end of 2009
 Appropriate where there are clear standards and guidelines
7
Process and output quality:
the Eurostat Quality Assurance Framework
 A system of coordinated methods and tools to ensure a
sustainable level of quality of processes and outputs in line
with the Code of Practice:
–
–
–
–
product/ output quality requirements are explicitely documented
processes are defined and made known to all staff
the correct implementation of the processes is monitored
users are informed on the quality of the products and possible
deficits
– a procedure to guarantee that the necessary improvement
measures are planned, implemented and evaluated
8
The Eurostat Quality Assurance Framework
9
The Eurostat Quality Assurance Framework
User Requirement
Standards
Labelling
Evaluation function
III. CONFORMITY
Quality
assessments
Rolling
reviews
Cost/benefit 5YP
Best practices
ASSESSMENT CHECKLIST
CVD steps
| | | | | |
II. EVALUATION
Quality diagram
Improvement actions
Quality
Barometer
AMP
I. DOCUMENTATION
MEASUREMENT
Process
description
Production
processess
Quality
reports
SDMX
Statistical products
User satisfaction survey
User perception
Eurostat Institutional/Legal environment
10
Process Quality Assessments (1)
 Common tool: The Process Quality Assessment Checklist (based
on the International ISPAC)
 Method: chronologically examining the process
Conceptual framework (2)
Follow-up (10)
User needs
(3)
Data
collection
(4)
Validation
Country level (5)
International level (6)
Confidentiality
(7)
Documentation Dissemination
(8)
(9)
IT conditions (11) – Management, planning and legislation (12) – Staff, work conditions and competence (13)
 Outputs:
– Assessment Diagram
– Summary Assessment report with improvement actions
– Highlight of a good practice
11
Process Quality Assessments (2)
Each process is matched with an assessment category
SelfAssessment
Supported
Self-Assessment
Prod. unit: fills in Checklist,
produces outputs
B1: provides support
on demand
Process 1
Peer review
Rolling review
A comprehensive review
Same as
based on the checklist
Prod. unit: in
Supported
Self-Assessment
implemented
by ext.contractor,
control of assessment,
participates in all steps with one or more peers (int/ext)involving partner & user surveys
taking part in all steps
and cost-benefit analysis
B1: pre-fills the Checklist,
produces the outputs
Process 2
Process 3
Process 4
Process 5
Characteristics:
- Periodicity
- Legal Basis
- Output
- ESTAT intervention
-availability of quality report
12
Process Quality Assessments Implementation
Plan: Principles
 Minimize burden for units
–
Test of the approach in advance
–
Support from the Quality unit
–
Flexibility
 Build on existing information
–
Quality reports
–
Process analysis
 Integrate QA with other horizontal activities
13
Process Quality Assessment Outputs (1)
Summary Assessment Report
14
Process Quality Assessment Outputs (2)
Accessibility and clarity
Assessment
Diagram
Overall quality of metadata
5
Comparability
across countries
Relevance
4
User satisfaction
3
Comparability
2
1
0
Comparability
over time
Accuracy
Overall accuracy
Coherence
General coherence
Timeliness
Timeliness of final publication
15
The Eurostat Quality Assurance Framework
User Requirement
Standards
Labelling
Evaluation function
III. CONFORMITY
Quality
assessments
Rolling
reviews
Cost/benefit 5YP
Best practices
ASSESSMENT CHECKLIST
CVD steps
| | | | | |
II. EVALUATION
Quality diagram
Improvement actions
Quality
Barometer
AMP
I. DOCUMENTATION
MEASUREMENT
Process
description
Production
processess
Quality
reports
SDMX
Statistical products
User satisfaction survey
User perception
Eurostat Institutional/Legal environment
16
ESS Standard & Handbook for Quality Reports (1)
 Provide (mainly producer oriented) recommendations /
guidelines for the preparation of comprehensive quality
reports for statistical processes and their outputs by
distinguishing between 6 types of statistical processes:
–
–
–
–
–
–
Sample survey
Census
Statistical process using administrative sources
Statistical process involving multiple data sources
Price or other economic index process
Statistical compilation
17
ESS Standard & Handbook for Quality Reports (2)
 Promote harmonized quality reporting across statistical
processes and Member States => facilitate comparisons
 Are applicable to National Statistical Offices and Eurostat in
their roles as producers of statistics
 Also include 21 quality and performance indicators
 Handbook: is more detailed, with lots of examples and
explanations, describes the quality concepts, explains how
the different quality documents relate to each other and is
aligned with the 15 Principles of the Code of Practice
18
ESS Quality architecture
ESS level
national / Eurostat level
ESS Statistical Law
ESS Quality Declaration
QAF:
ESS standards
for implementing
the CoP
national Statistical Law
(national TQM framework)
National QAF:
Description of practices
and procedures
for implementing the CoP
19