First results from the in-depth surveys
on quality assurance frameworks
and quality reporting
Conference on Data Quality for International Organisations
27-28 April 2006, Newport (UK)
Håkan Linden,
Statistical Governance, Quality and Evaluation
European Commission, Eurostat
Background
• Key areas of work in the Eurostat project on “the use and
convergence of international quality assurance frameworks”
sponsored by the CCSA.
• Problem statement – quality assurance frameworks:
- lack of use of quality assurance frameworks in most international
organisations
- existence of different quality frameworks and benchmarking activities for
national statistics reported
• Problem statement – quality reporting:
- existence of different tools and practices for collecting quality metadata from
the data providers
- existence of different formats to inform the users on the quality of statistics
Overview in-depth surveys
• Questionnaire templates developed by the task-teams for
structuring the work and collecting the state-of-art on quality
frameworks and quality reporting activities.
• All members of the CCSA included in the surveys
• Data collection: 20 February 2006 – 31 March 2006
• 12 international organisations replied – thank you!
• First results for this conference
• Report with detailed results and recommendations to be prepared
for the 2nd CCSA meeting 2006
Quality frameworks - the template
• Section 6. Quality Assurance Frameworks in place
- brief description of the quality assurance framework
- legal basis
- last update
- coverage of institutional environment, statistical processes and outputs
- quality definition
- quality requirements/ targets
- procedures for evaluation of the adherence to the quality framework
- main strengths and weaknesses of the quality framework
• Section 7. If a quality framework not yet in place
- brief description of the current situation of the quality work
- how weaknesses are identified, impact assessed and improvement actions
- main obstacles for development and implementation of a quality framework
• Section 8. For all international organisations
- the future plans for quality improvements
- additional comments/ recommendations etc. for this project
Quality frameworks - results
• 5 (out of 12) organisations have quality assurance
frameworks in place
• 1 organisation is currently developing an encompassing
quality assurance framework
• 6 organisations have not yet begun to develop formalised
quality assurance frameworks
Quality frameworks - in place
• “Legal” basis:
- endorsed by (internal) Statistical Policy Group and Statistical Committee
- Data Quality Standard under development
- Article
- Policy document
- Adherence to the “Principles Governing Statistical Activities”
- Recommendation
• Last update: 2003 (new update in 2006); May 2004; July 2003;
May 2005; 2003.
Quality frameworks - in place
• Coverage of institutional environment/ settings:
- all statistical activities/ major activities reviewed every 5- years
- statistical data under “direct coordination” but extension of the coverage
planned
- 2 organisations actually cover the institutional settings!
Quality frameworks - in place
• Coverage of statistical processes:
- definition of data requirements
- evaluation of other data currently available
- statistical design and planning
- data and metadata collection
- data and metadata processing
- compilation and estimation
- disclosure control
- data and metadata dissemination
Quality frameworks - in place
• Coverage of statistical output:
- all
- limited to a few domains
- economic monetary statistics
Quality frameworks - in place
• Quality requirements/ targets:
(i) Institutional environment/ settings:
- benchmarks for observation of international good practice or standard with
reference to for example the UN Fundamental Principles of Official Statistics,
the UN Handbook of Statistical Organization, and the Quality Declaration for
the ESS
(ii) Statistical processes:
- a set of best practices is defined for each process. Each activity must be
compared with these practices
- benchmarks …
- standardisation and harmonisation of tools, reports and development of
policies (like revision policy)
(iii) Statistical outputs:
- a set of best practices is defined for each process. Each activity must be
compared with these practices
- punctuality (annually agreed timetables)
- regulations
Quality frameworks - in place
• Procedures in place for the evaluation of the adherence to the
framework:
(i) Institutional environment/ settings:
- surveillance missions may help to identify institutional and legal issues
underlying problems in data quality
- audit exercises
- self-assessments
- peer reviews
- annual report on the implementation and protection of confidential information
(ii) Statistical processes:
- quality reviews of ongoing activities on a 5 year rolling basis (self-evaluation of
strengths and weaknesses by comparing with best-practices)
(iii) Statistical outputs:
- compliance reports
- regular reports on gaps in statistics (as input for medium-term work-pgm)
- quality reports/ quality profiles
Quality frameworks - in place
• Strengths:
- Quality reviews (self-evaluation of strengths and weaknesses by responsible
persons, comparing their activities with best-practices) on a rolling basis give
a good framework for discussing quality problems between domain managers
and statistics and IT- experts
- Data quality monitoring at each stage of a statistical production process
makes it possible to identify and address data quality problems. Incorporated
“feed-back loops” serve as a mechanism for improving data quality
- The implementation of quality framework (phased/ staggered approach)
ease the development of new work processes related to data collection and
processing, IT, and data dissemination
- Existence of transparent and comprehensive legislative framework for the
production of statistics ease the monitoring of data quality
- Quality manager appointed to coordinate all quality work
Quality frameworks – in place
• Weaknesses:
- The resources allocated to corporate quality work are insufficient
- The resources to help solve quality problems are scarce
- The implementation of process-oriented data quality management requires
strong co-ordination and support (IT and statistics)
- Lack of standardisation of data treatment procedures and insufficient
documentation of methods
Quality frameworks – in place
• The main constraints on reaching optimal data quality in the
statistical processes:
- The resources
- Difficult to apply the same quality concept (e.g. dimensions) for less
developed statistical systems as for more developed systems
- The improvement of data quality is viewed as a gradual process that need
to take into account resources constraints and establish priorities
- No direct contact with respondents
- The quality of the country data
Quality frameworks - not (yet) in place
• The current quality work:
- implementation of new Statistical Information System (data validation/
consistency checks and metadata management)
- procedures for validation of statistics/ estimations by countries before
publication
- regular review of data collection activities
- internal quality reports
- methodological aspects are documented and used to identify best practices
- project on “process documentation” to identify weaknesses and define best
practices
Quality frameworks - not (yet) in place
• Identification of weaknesses:
- manual data clearance procedures are well functioning but proves to be
time consuming and intervenes at a late stage in report preparation
- data weaknesses are identified when in-house economists draft their
analytical reports based on the statistics
- data weaknesses are identified on the basis of metadata available and by
comparisons with similar statistics from other organisations
- the impact of data weaknesses is assessed using statistical “judgement”,
improvement actions are discussed between statistician(s) and the supervisor
and implemented according to agreed plans
- weaknesses in data are pointed out by the users (user feed-back or user
surveys)
Quality frameworks - not (yet) in place
• Main obstacles/ critical issues for implementation of quality
assurance framework:
- economists and analysts perceive data quality requirements too excessive
and seen as extreme refinements imposed by statisticians
- relevant IT tools should support the quality framework to efficient process of
data and quality control (QF might be seen as a concept impossible to apply)
- time and resources
- A well-designed quality framework for organisations compiling statistics from
different international organisations would ease the implementation
Quality frameworks – future plans (1)
- review of existing quality framework
- inform senior managers on data quality issues and principles governing
international statistics
- training of staff on quality issues and data processing
- data collection by electronic means will increase and make it possible to
implement more automatic data validation
- SDMX is expected to be a mean for accessing data and metadata from
countries and other international organisations
- improved accessibility to data for users
- adapt the common quality assurance framework that will be proposed by
CCSA
- establishment of a statistical data quality group
Quality frameworks – future plans (2)
- promoting data and metadata dissemination
- regular monitoring, assessment and reporting based on the statistical data
quality framework
- peer reviews of member countries
Quality reporting – the template
• Section 6. Metadata about quality collected from data providers
- brief description of the quality reporting activties
- legal basis
- quality dimensions covered
- frequency
- procedures/ mechanisms in place for quailty reporting
- type of quality information (qualitative/ quantitative)
- the use of the quality information
- strengths and weaknesses of the quality reporting
- quality constraints
- quailty improvements
• Section 7. Metadata about the quality of released statistics
- brief description of the ways of informing the users on the quality
- based on standardised reference metadata
- type of quality information disseminated (qualitative, quantitative)
- strengths and weaknesses of the system of informing users on quality
• Section 8. For all international organisations
- the future plans for improving quality reporting
- additional comments/ recommendations etc. for this project
Quality reporting - results
• “Legal” basis:
- Guidelines for good practices
- Ratification of Convention for data reporting and its implementation
- indirect through Council decision
- Standards
- Regulations
- Recommendation
Quality reporting - results
• Coverage of quality dimensions (1):
- relevance, accuracy, credibility, timeliness, accessibility, interpretability,
coherence
- relevance, accuracy, timeliness, punctuality, accessibility, clarity,
comparbility, coherence, completeness, and sound metadata
- prerequisites of quality, assurances of integrity, methological soundness,
accuracy and reliability, servicability, accessibility
- methodological soundness, accuracy, reliability, consistency, timeliness, and
punctuality
- relevance, accuracy, timeliness, punctuality, accessibility, clarity,
comparability, and coherence
Quality reporting - results
• Coverage of quality dimensions (2):
- relevance, accuracy, interpretability, and coherence
- completeness, accuracy, reliability, comparability, adherence to standards
- data accuracy, comparability and consistency
- data consistency (aggregation) and comparability with standards
- methodological soundness (coverage) and conistency over time
Quality reporting - results
• Frequency of the quality reporting:
- no common rules
- rolling programme
- once a year
- approx. every 5 years
- twice a year
- each time data submitted/ collected
- quarterly
- according to requirements in Regulations
- as part of projects on country assistance/ country reviews
Quality reporting - results
• Procedures/ mechanisms in place for the quality reporting:
- the quality framework provides the theoretical and practical guidance
- metadata questionnaire attached to statistical data questionnaire
- manuals/ guidelines/ handbooks
- clerical and computerised edits
- standardised reporting forms for metadata
- glossary on quality
- bulletin board on data collection, dissmeination and quality of statistics
- standardised data quality assessment framework tool (part of formalised
data quality progam) for assessing member countries data quality
- regulations
- metadata common vocabulary
Quality reporting - results
• Type of quality information collected:
- almost all information provided are qualitative
- very few quantitative measures, like:
- response rates
- data completeness
- data timeliness
- revisions
------------------------- binary indicators
- set of standard quality indicators
- overall assessments:
- allocation of “colour-codes” to country data
- quality ratings (four grade scale)
Quality reporting - results
• Strengths:
- decentralised meaning that activity managers can collect the metadata
- metadata reporting by NSO’s are done in a systematic manner
- allows checking compliance against international standards
- a tool for monitoring where there are trade-offs between data quality
problems and allocation of resources
- standard framework for metadata in general and metadata about quality in
place covering many countries
- good cooperation with data providers
- quality criteria and quality reporting requirements included in legislations for
member countries
Quality reporting - results
• Weaknesses:
- un-coordinated
- too decentralised (no coordination)
- no standards/ standardised reporting
- no central monitoring
- needs further embedding in overall culture of the organisation
- reporting NSO’s not aware of international standard methodology, concepts,
definitions and classifications
- no standardisation of metadata and rules to be followed
- metadata are not processed automatically
- infrequent collection of information
- burden on countries
- lack of resources
- self-assessments of countries not yet done
- not all statistics yet covered
Quality reporting - results
• Coverage of quality constraints:
- none (9 out of 12)
- data sharing and coordination among data producing agencies
- confidentiality
- resources (financial, staff, technical) and efficient use
- planning of statistical programs
- covered by a special “merit and costs” procedure
- burden on respondents
Quality reporting - results
• Coverage of quality improvement actions:
- no (8 out of 12)
- sometimes
- explicit sections on “plans for improvement” and “specific recommendations
for improvement”
Quality reporting - results
• Future plans for improving quality reporting from data providers:
- continued use of electronic dissemination
- investigation on the use of SDMX standards
- development of a metadatabase that allows automatic handling of metadata
for the processing, tabulation and dissemination
- revision of the quality reporting framework in order to improve standardisation
- data quality assessment framework is being incorporated into the data
dissemination standards
- development of a coherent quality framework with the aim to introduce a
harmonised definition of quality
- workshops and seminars for countries on strengthening the capacity to
produce data
Quality information for the users
• Ways of informing the users on the quality:
- quality metadata is embedded in the dissemination system
- attached notes to datasets and tables
- metadata included in “Sources and Methods” publications
- standardised “country notes” on methods and definitions for each release
- Data Dissemination Standard(s)
- dissemination of data only possible if metadata are filled in
- annual quality report published summarising all economic statistics
- annual internal quality report for each statistical area
- quality profiles for key statistical indicators
- domain specific quality reports
Quality information for the users
• Future plans for improving quality information about the published
statistics:
- implementation of new Statistical Information System designed to fully
support metadata (harmonisation and automatisation)
- more use of pre-coded response for metadata to improve standardisation
- adoption of systematic criteria is under consideration (dependent on
international work)
- coverage of all statistical domains
- data quality requirements to be better integrated in the statistical database
dissemination system