The use and convergence of quality assurance frameworks for international and supranational organisations compiling statistics The European Conference on Quality in Official Statistics July 8-11 2008, Rome, Italy Antonio Baigorri and Håkan Linden Statistical Governance, Quality and Evaluation European Commission, Eurostat The context of Total Quality Management To have an encompassing approach with respect to quality work. To implement the principles of institutional quality frameworks and in particular the principles related to statistical processes and outputs. To improve the measurement, monitoring and management of data quality. To coordinate ongoing quality initiatives (process descriptions, quality reports, evaluation activities etc.). To build on existing quality work (standards, best practices etc.). To promote a culture of systematic quality improvement work. 2 Institutional Quality Frameworks TQM TQM TQM User User needs needs Code DQAF of Practice User needs Management Management systems systems andand leadership leadership Management systems and leadership Support Support processes processes Support processes Statistical Statistical products products Statistical products Production Production processes processes Production processes Institutional Institutional environment environment Institutional environment 11 4. Relevance Serviceability 12 Accuracy and reliability 13 Timeliness and CCSA Principles Punctuality 14 Coherence and comparability 15 Accessibility 5. Accessibility And clarity Principles: 1.1, 1.2, 4.3, 4.4, 4.5, 4.6, 5.2, 6.2, 7.1, 7.2, 8.1, 10.2, 10.5 7 Sound 2. Methodological methodologySoundness 8 Appropriate statistical procedures 9 Non-excessive 3.Accuracy and burden Reliability on respondents 10 Cost effectiveness Principles: 4.2, 5.1, 5.3, 5.4, 5.5, 8.3, 8.4, 9.2, 9.3, 9.4, 9.5, 10.1 1 Professional 0. Prerequistes independence of Quality 2(Legal Mandate and for institutional data collection 3 Adequacy environment, of resources Resources, 4 Quality Relevance, commitment Other quality 5 Statistical management6 Impartiality and objectivity confidentiality 1. Assurance of 1.4, Integrity (Professionalism, Principles: 1.1, 1.3, 1,5, 2.1, 2.2, 2.3, 3.1, 3.2, 4.1,Transparency 5.3, 5.6, 6.1, and 6.2, Ethical 7.1, 8.2,Standards 9.1, 10.3, 10.4 Source: Eurostat (2007) Institutional frameworks, like the Principles Governing Statistical Activities, the European Statistics Code of Practice and the IMF Data Quality Assessment Framework, can be seen as general superstructures forming the necessary basis for all other measures an International organisation needs for improving quality at statistical output and product level. 3 Quality Assurance Frameworks Quality assurance frameworks (or frameworks for statistics production) have the objective to establish, in a specific statistical organisation, a system of coordinated methods and tools guaranteeing the adherence to minimum requirements concerning the statistical processes and products. Similarly to institutional frameworks, this includes some kind of assessment. Product/ output quality requirements are being explicitely documented. Processes are defined and made known to all staff. The correct implementation of the processes is monitored on a regular basis. Users are being informed on the quality of the products and possible deficits. A procedure is implemented that guarantees that the necessary improvement measures are being planned, implemented and evaluated. 4 Data quality aspects 1) The perception of the statistical product by the user. 2) The characteristics of the statistical product (or key statistical outputs) 3) The characteristics of the statistical production process. 5 Relationship between process and output quality RELEVANCE ACCURACY ACCESSIBILITY/ CLARITY TIMELINESS/ PUNCTUALITY COMPARABILITY COHERENCE Conceptual framework (2) Follow-up (10) User needs (3) Data collection (4) Validation Country level (5) International level (6) Confidentiality (7) Documentation Dissemination (8) (9) IT conditions (11) – Management, planning and legislation (12) – Staff, work conditions and competence (13) Source: Eurostat Process Quality Assessment Checklist [Eurostat, 2007] 6 Product/ Output Quality Components OECD: relevance, accuracy, credibility, timeliness (and punctuality), accessibility, interpretability, coherence (within dataset, across datasets, over time, across countries) Eurostat: relevance, accuracy, timeliness and punctuality, accessibility and clarity, coherence (within dataset, across dataset), comparability (over time, across countries) ECB: accuracy/reliability, methodological soundness, timeliness, consistency IMF: prerequisites of quality, accuracy and reliability, assurances of integrity, methodological soundness, serviceability (timeliness and periodicity), accessibility, serviceability (within dataset, across dataset, over time, across countries) FAO: relevance (completeness), accuracy, timeliness, punctuality, accessibility, clarity (sound metadata), coherence, comparability UNESCO: relevance, accuracy, interpretability, coherence UNECE: relevance, accuracy (credibility), timeliness, punctuality, accessibility, clarity, comparability (across datasets, over time, across countries) 7 Product/ Output Quality Components Relevance Accuracy (and reliability) Timeliness Punctuality Accessibility Clarity/ interpretability Coherence/ consistency Comparability 8 Quality and metadata Collection and sharing of metadata - SDMX technical standards - SDMX Content oriented guidelines (incl. Cross domains concepts) Dissemination of metadata on quality - Special Data Dissemination Standards (SDDS) - Euro-SDMX Metadata Structure (ESMS) Assessment and monitoring of metadata Integrated information on quality assessment 9 Methods and tools for the assessment of statistics production User requirements Standards III. Conformity Labelling II. Evaluation I. Documentation Measurement Self assessments Process variables Quality indicators Production processes Quality reviews Quality reports User satisfaction survey Statistical products Improvement actions User perception Institutional/ legal environment N.B. Figure derived from draft Handbook on Data Quality Assessment Methods and Tools (DatQAM), version 31.01.2007. 10 How to apply process oriented quality assessment tools The office-wide management approach Institutional preconditions (procedures and legislations) Assessment methods already in use Relevance – size and periodicity Relevance – importance and specific legal frameworks 11 Quality Assessment Packages Labelling Advanced package Key process variables User satisfaction surveys Quality reviews Intermediate package Quality indicators Self assessments Quality reports Fundamental package Process descriptions, product documentation, quality guidelines N.B. Figure derived from draft Handbook on Data Quality Assessment Methods and Tools (DatQAM), version 31.01.2007. 12 The assessment methods and tools Documentation and measurement - process descriptions - quality reports (“Full Quality Report”, “Summary Quality Report”, and “Basic Quality Information”). - user satisfaction surveys Evaluation - self assessments of all production processes (Quality Assessment Checklist) - quality reviews for key statistical outputs Conformity - a process for labelling of key international statistics 13 Principles for implementation Minimise burden for production domains - test the approach in advance - provide support - flexibility Build on existing information - process analysis - metadata on quality (quality reports etc.) Profit from synergies with other horizontal activities - evaluation function requirements - cost/ benefit analysis - input for management programming 14 Data quality assessment recommendations Top management commitment The role of middle managers Data quality assessment is a long term project Most methods should be implemented and fine-tuned in pilot projects Standardise the use of the methods Establish clear responsibilities and authorities Sufficient resources allocated for supporting the assessments 15 Conclusions Each international organisation should have a quality assurance framework in place. The framework and the applied quality principles should be made explicit. A quality assurance framework needs to be compatible with the general quality management model and office-wide procedures and rules. It should be built into the organisational structure. It contributes to increased awareness of quality concepts and promotes best practices. It provides a mechanism for reengineering and quality improvements It should always acknowledge performance/ cost. Convergence of quality assurance frameworks by applying common concepts, standards, methods and tools (both content oriented and technical). Development and sharing of “best practices” for statistics production between all stakeholders is maybe the most important for continuous quality improvement of a global statistical system. 16