'Snapshot'
A user friendly tool to assess
strengths and weaknesses of
National Statistical Systems
Eurostat
Content of this presentation
1.
2.
3.
4.
5.
6.
7.
8.
Background
Objectives
Why developed by Eurostat?
Presentation of the tool
How to fill the questionnaire?
How to present the results?
How to use the results of the assessment?
Conclusions
2
Eurostat
Background
•
•
•
•
•
Original context is development cooperation, but this tool can be
used also in a wider context
Results-based management and performance assessments are
central in development
The monitoring of development policies and of aid delivery (and
budget support in particular) require the setting of benchmarks
and the construction of systems of indicators
These benchmarks and systems of indicators are based on official
statistics which are provided by the countries themselves
Countries need baseline and benchmarks for assistance projects
and their own development
=> What do we know about the quality of these
statistics and how can we assess them in a userfriendly way?
3
Eurostat
Objectives
Through the Snapshot, different stakeholders can assess:
• The institutional environment of an NSS
• the quality of data that donors receive from partner
countries to monitor the results achieved by the national
policies and reform programmes
• the performance of the support given by the EU to these
policies and reform programmes
• official statistical data measuring the success of assistance
projects
• the state of play in the development of the NSS of a
country
• the efforts to improve the performance of the NSS
4
Eurostat
Why developed by Eurostat?
• Experience with the European Statistics Code of Practice
• Self-assessments by the EU Member States combined with
Peer reviews (in EU Member States) and Global
Assessments in non-EU countries (for example, in the
Western Balkan, Eastern Europe and North Africa)
• A reference for other quality initiatives in the world (for
example, the African Charter on Statistics, the Code of
Good Practice in Statistics in Latin America and the
Caribbean, the ASEAN Community Statistical System Code
of Practice, the Code of practcioe for the European
neighbourhood South, etc.)
=> A natural vocation for supporting quality
Eurostat
5
Presentation of the tool
The tool consists of a questionnaire (in Excel) divided in two
modules:
• Module 1: analysis of the functioning of the NSS as a
whole, with a special focus on institutional, legal and
organisational aspects
• Module 2: analysis of specific statistical sectors and
related key indicators. A wide range of sectors can be
assessed (such as: education, health, agriculture, food
security, private sector development, climate change and
environment, economic growth, governance, infrastructure,
external trade of goods, employment, living conditions,
6
macro-economic stability, population)
Eurostat
Presentation of the tool
Module 1: assessment of the NSS (61 questions)
Themes
Number of
questions
Content
Country strategy
for statistics
Legal, institutional and strategic foundations (i.e. legal and institutional
framework for the whole NSS, integration and consistency of the statistical
development strategy in the national development strategic framework)
13
Organisation
Organisation and coordination of the NSS
15
Resources
Adequacy of resources for statistics in the NSS (i.e. personnel in adequate
quantity, quality of the personnel involved in statistics, equipment and
infrastructure, financing)
16
Quality
Users
Determinants of data quality (i.e. quality commitment, professional
independence, impartiality, objectivity, methodology and appropriate
statistical procedures)
Relations with users (i.e. relevance, accessibility)
TOTAL
Eurostat
13
4
61
7
Presentation of the tool
Module 1: assessment of the NSS
Examples of questions:
• Are statistics used in the development, implementation and monitoring of
national development policies?
• Are there statements made by high-level officials (president, ministers,
parliament, etc.) on statistics development?
• Is statistical independence a guiding principle for the functioning of the NSS?
• Does the NSS has mechanisms for assigning responsibilities in the national
statistical process?
• Is there a human resources policy (recruitment, career development, education
and training) for the NSI?
• Which proportion of the staff has a (permanent) internet connection?
• Are there internal handbooks / guidelines / recommendations for the statistical
production processes?
• Are there formal processes in place to consult users about their statistical
8
needs?
Eurostat
Presentation of the tool
Module 2: assessment of specific statistical sectors
Number of questions
Themes
Part 1
Content
One data producer
Several data producers
Assessment of sector
Framework
Legal, institutional and strategic foundations (i.e. legal and institutional framework of
the concerned sector information system, integration and consistency with the
strategic framework supporting sector statistics development)
8
8
Resources
Adequacy of resources for statistics within the sector information system (i.e.
personnel in adequate quantity, quality of the personnel involved in statistics,
equipment and infrastructure, financing)
6
9
Quality
Determinants of data quality (i.e. quality commitment, professional independence,
impartiality, objectivity, methodology and appropriate statistical procedures)
21
22
Users
Relations with users (i.e. relevance, accessibility)
10
11
Production
Statistical operations and products
45
53
9
9
3
TOTAL
Part 2
Indicators
Assessment of indicators
Set of key criteria to roughly assess the quality of the indicator
Eurostat
9
Presentation of the tool
Module 2: assessment of specific statistical sectors
Examples of questions:
•
Is the development of sector statistics included in the Sector Development
Strategies?
•
Are there problems caused by inadequate IT resources (hardware, software network)
in any of the agencies involved in the sector statistics?
•
Is information on the methods and procedures used for the sector statistics published
on the NSI website or in a Ministry / observatory website?
•
Do all users get access to statistical releases at the same time?
•
Are there partnerships or joint studies/works with the scientific community?
•
Are there satisfactory quality controls and checks on the administrative data reported
by the units in the sector (e.g. schools, hospitals, health stations)?
•
How much of the statistical outputs in the sector are available via the internet?
•
Is there a contact point to assist users?
10
Eurostat
Presentation of the tool
Easy colour codes:
Not complying with quality requirements
Partially complying with quality requirements
Fully complying with quality requirements
11
Eurostat
Presentation of the tool
Additional codes:
The situation is
expected to improve
No changes are
expected
The situation is
expected to deteriorate
12
Eurostat
Presentation of the tool
13
Eurostat
Presentation of the tool
Module 2
Part 1 - Assessment at Sector/Ministry level
EDUCATION
Theme 1: Legal, institutional and strategic framework at the level of the sector (both the NSI and the sector Ministry)
Q1: Legal and institutional framework supporting the production of the sector statistics
Detailed
questions
Potential
sources
Priority
Is there a law or
an official
agreement that
organizes
efficiently the
production of
sector statistics?
Other
Does the National
Statistical Office
or other body
responsible for
statistics in the
sector have
access to the
relevant
administrative
data?
Is the actual work
organization
(mandate and
attribution) in line
with the law or the
official agreement?
Statistical
laws,
Government
decisions,
Memoranda of
Understanding,
Service Level
Agreements,
Questionnaire
to DG
Situation
Color
coding
Remarks
Perspec
tive
coding
Identified
sources
Initial
assess
ment
No specific law. But there is a
practice withinthe Minsitry of
Education that has been
established during the last 20
years and that involves the
schools, the districts and the
provinces. There are also
regular relations with the LSB
(training, consultation on
statistical operations)
The law addresses the
rights and responsibilities of
the line ministries for the
production and the
dissemination of statistics.
A decree from the Ministry
of Education established
the Information Centre (role
and responsibility) in 2006.
There will be a new decree
soon as ICT and statistics
will be split in two different
bodies.
→
Statistical law
→
Yes. No mandate is
specifically written, but in
practice, data are collected
regularly from schools and
villages.
The system is managed by
the Education Statistics
and Information Technology
Center of Education and
Sport (SITCES). The
system is improving year
after year.
↑
Interview with
Deputy DG
SITCES
↑
Yes
partialy
The LSB is responsible for the
coherence of official statistics
mainly through its
responsibilities in the
Yes
management and direction of
the NSS. The Ministry of
Education gather annual data
from schools and villages.
See Article 32 of the
statistical law.
Eurostat
→
Statistical law
→
Management of the table
In most case, there will not be a
specific law at the sector level. It is
generally expected that the relations
between the NSI and the other data
producers in a country are driven by
memorandum of understanding that
sets the respective responsibilities,
including data sets to be exchanged
and calendar.
To check if mechanisms
(communication, dialogue, exchange
of data …) that ensure coherence of
methods and outputs between the
different structures (NSI, sector
Ministry, ) involved in the production of
sector statistics
14
Presentation of the tool
15
Eurostat
How to fill the questionnaire?
What are the basic sources?
For aspects linked to the legal and policy framework, to the organisation and
functioning of the system:
Country macro-economic and
social plan/strategy (Growth
and poverty alleviation strategy
or any similar policy document)
National Strategy for the
Development of Statistics
(NSDS)
Module 1 (mainly themes 1 and 2)
Generally available on the
Government web site
Information documented and
regularly updated on the
PARIS21 web site. Also
generally available on the NSI
web site
The statistical law and related
application decrees
Generally available on the NSI
Other information on the NSS
web site
(coordination, exchanges
among the data producers/users)
Sector strategy documents/plans Web sites of the related sector
Ministries
Information on the Sector
Module 2 (mainly theme 1)
Information system
16
Eurostat
How to fill the questionnaire?
What are the basic sources?
For aspects linked to practices:
All modules (mainly
themes 3, 4 and 5
for Module 1 and
themes 2, 3 and 4
for Module 2)
Reports by the NSS (annual
reports/plans, self assessments, studies,
guides and methods, etc.)
Reports by other donors
Contacts/discussions with the actors of
the NSS
Not always available on the web (paper
documents)
Not always easy to identify and to
gather particularly when there is no
coordination mechanism for statistics
Direct source
17
Eurostat
How to fill the questionnaire?
Who will do it?
In the framework of a donorbeneficiary relationship
Staff from the EU Delegation or
other donors
Expert statistician mobilized by the
EU Delegation or other donors
EU Delegation with other donors
EU Delegation or other donors and
National Statistical System (NSS)
EU Delegation, other donors and
NSS
In the framework of a self-assessment
of the NSS
Even if only one part of the questionnaire is
answered, the tool can give useful
indications, particularly if the objective is to
support statistical improvement in areas
most in needs
This could help when it is done for the first
time.
Filling in the questionnaire could be the
result of a coordination and information
sharing exercise among a group of donors.
This would be the best way to initiate and
keep a regular dialogue on the strengths and
weaknesses of the NSS and, potentially, to
open the way to concrete support.
Same as the above but involving a group of
donors.
Any member of the NSS
Expert statistician mobilized by
the NSS
This would be the result of a full
appropriation of the tool by the NSS.
Certainly the best solution in the long
run.
This could help when it is done for the
first time.
18
Eurostat
How to present the results?
An example of the results for Module 1
Module 1 - Global results
8
7
6
5
4
3
2
1
0
19
Eurostat
How to present the results?
An example of the results for Module 1
Gap in resources allocated
to statistics
Strategy exists but
not fully
mainstreamed in
policies
Module 1 - Global results
8
7
No rigor in quality
controls
6
5
4
3
Legal framework in
place but poor
coordination
Poor relevance of
data and poor
accessibility to data
2
1
0
20
Eurostat
How to present the results?
An example of the results for Module 2 – part 2
Number of judges
22%
33%
45%
21
Eurostat
How to present the results?
An example of the results for Module 2 – part 2
Definitions and
sources not in line with
international
recommendations
Number of judges
22%
33%
45%
22
Eurostat
How to use the results of the
assessment?
• The assessment could be repeated regularly as a
monitoring instrument for the EU and other
donors
• The results could be used to initiate a dialogue
between donors and the NSS
• The results could be used to identify statistical
areas in need of support and hence to design
statistical capacity building programmes
23
Eurostat
How to use the results of the
assessment?
• The results could be presented and discussed
each year (regularly) at a meeting of the National
Statistical Council or of a similar entity. This
would help in adjusting work plans.
• The results could be used to generate a sort of
literature of good practices from several countries
24
Eurostat
Conclusions
• The Snapshot can be understood and used
quickly
• The Snapshot can be used regularly (for
example, yearly)
• The Snapshot does not replace more in-depth
analysis
25
Eurostat
Conclusions
• The Snapshot is a package that includes:
1.
2.
3.
4.
The
The
The
The
questionnaire in Excel (available in English, French and Spanish)
user manual in pdf (available in English, French and Spanish)
e-learning in PowerPoint and video (available in English)
brochure in pdf (available in English, French and Spanish)
• If you wish to receive it, please write to:
ESTAT-STATISTICAL-COOPERATION@ec.europa.eu
• Probably, when the new website of Eurostat will
be ready, it will be possible to download the
Snapshot online.
26
Eurostat