Application of a DQAF for Education Statistics
Brian Buffett (b.buffett@uis.unesco.org)
Conference on Data Quality for International Organisations
Newport, Wales
April 27 - 28, 2006
www.uis.unesco.org
Outline
 Assumptions
 Introduction
 Overview of the DQAF
 The DQAF in practice
 Observations & Summary
www.uis.unesco.org
Assumptions
 Familiarity with the concept of Data Quality Assessment Frameworks;
 Awareness of the IMF DQAF;
www.uis.unesco.org
Introduction
1999 ……….
UNESCO Institute for Statistics (UIS) was established
2001 ……….
UIS moved from Paris to Montreal, Canada
2003 ……….
Statistical Capacity Building Programme (SCB) introduced
UIS Mission:
• Internationally comparable statistics in UNESCO’s areas of
competence:
Education; Science & Technology; Culture; Communication
Currently:
• 100 staff members in Montreal
• Regional Advisors in: Chile, Senegal, Thailand, Ethiopia, Samoa
www.uis.unesco.org
Why a Statistical Capacity Building Programme?
 Statistical Capacity Building is one of the four Main Lines of Action in
UNESCO Institute for Statistics statutes.
 The SCB programme serves two purposes:
•
It supports member states to meet their own needs for production
and use of statistics in UNESCO domains.
•
It supports primary UIS data programmes.
» Since countries are the source of UIS primary data –
improving country data is an essential ingredient for improved
international data.
www.uis.unesco.org
Why an Education DQAF?
 The SCB programme needed a mechanism in order to efficiently and
effectively engage countries and assess the statistical systems within
education ministries.
 Within education ministries, what was often lacking was not awareness
of the need for better data and statistics – but an awareness of what
the problems were and a road map for how to go about addressing
them.
 The UIS desired a broad framework that focussed on the qualityrelated features of the governance of statistical systems, their core
statistical processes, and their statistical products.
www.uis.unesco.org
The IMF DQAF
 The IMF DQAF is not limited solely to timeliness and accuracy
 Six dimensions to the IMF DQAF:
• Prerequisites of quality
• Integrity
• Methodological soundness
• Accuracy and reliability
• Serviceability
• Accessibility
www.uis.unesco.org
Why extend the IMF DQAF?
 The DQAF seemed to meet the overall requirements;
 The work was reduced to a fraction – and domain specific;
 Cost and timelines to implement were attractive;
 The IMF was willing;
www.uis.unesco.org
Extending the DQAF to Education
 Developed in 2003 in collaboration with the World Bank.
 Addresses:
• International Standards and Classifications (ISCED)
• Best practices and guidelines specific to education
• Verifies statistical system measures and reports on:
» Structure and normative characteristics of education system
» Supply of education
» Demand for education
» Quality of learning outcomes
» School environment
»…
www.uis.unesco.org
The DQAF in Practice
 Used to diagnose the situation of national information systems on
education, paying particular attention to national information needs.
 These diagnoses are a major element to devise action plans to
strengthen national capabilities on education statistics.
 International reporting requirements are addressed but not the primary
objective.
 Flexibility in developing action plans:
• If there are significant problems related to international data
reporting, ISCED is a critical element;
• if the major problems are related to nationally-specific challenges,
other items are addressed.
www.uis.unesco.org
The DQAF in Practice
 How have the diagnostics been carried out?
• Weighted the DQAF components and developed a scoring guide;
• Development of common methods and best practices;
• Scoring is to a significant extent an expert judgement. Diagnostic
missions carried out by a small number of trained staff using
common methods.
 Regional activities can be facilitated by ensuring coherence across
countries
www.uis.unesco.org
The DQAF in Practice
 In Latin America and the Caribbean region, by the end of 2006:
• Completed in Honduras, Ecuador, and El Salvador; are in revision in
Costa Rica, and Nicaragua, and being prepared for Guatemala,
Uruguay, Paraguay, Peru and Colombia;
• Are being prepared for a similar number of Caribbean countries.
 Will result in systematic diagnoses under a common framework for half
of the region.
 The identification of common challenges will:
• permit grouping of countries;
• provide the basis for country-to-country cooperation;
 Will have the necessary information to support national as well as
regional purposes.
www.uis.unesco.org
Summary – the Country Perspective
 A useful tool to help strengthen the country’s statistical system by
identifying the strengths and weaknesses of the system as well as areas
to be improved.
 Some results:
• adoption of new questionnaires better responding to country
information needs;
• improved collection methodology – including training of respondents;
• more timely completion and return of completed questionnaires;
• more efficient data capture – with edits to ensure data quality;
• more efficient processing of the data and production of outputs;
• improved access to data;
• training of statisticians and policy makers in use and interpretation of
the data.
www.uis.unesco.org
Summary – the UIS Perspective
 Reduced resource costs, timeframe, and skills required for framework
development; Able to focus resources and efforts on subject-matter specificities;
 The framework and diagnostic method are effective;
 Initial results have been achieved more rapidly due to this approach;
 Country quality reports on education can be comparable - providing more
flexible approaches to capacity building and increasing country-to-country
cooperation;
 Common best practices need to be followed, such as:
• Country ownership;
• Broad involvement;
• Assessments combined with audits;
• Consistent application of scoring guides.
 UIS will benefit from future IMF work on the DQAF;
www.uis.unesco.org
Summary – the International Perspective
 An example of collaboration and reuse of existing methods – adapted to
International Org. environment/needs;
 Factor country needs and situation into any approach;
 Statistical activities outside of NSO’s can benefit from the same
practices as NSO’s;
 A broad definition of quality is important;
www.uis.unesco.org
Thank You
www.uis.unesco.org