FOOD AND AGRICULTURE ORGANIZATION
OF THE UNITED NATIONS
Improving the Data Quality monitoring
framework (CCSA/Eurostat self assessment
initiative)
Case of FAO Producer Prices data (methodology and
data quality self assessment):
by Carola Fabi
FAO Statistics Division
AFCAS meeting, Algiers
12 December 2007
FOOD AND AGRICULTURE ORGANIZATION
OF THE UNITED NATIONS
Introduction
• The subject of data quality is consistently
addressed in many international forums
• countries have always addressed data quality
from both practical and theoretical perspectives
• growing consensus that policy and decisions
are increasingly reliant on better data if they are
to be effective
• convergence of data quality models
AFCAS meeting, Algiers
12 December 2007
FOOD AND AGRICULTURE ORGANIZATION
OF THE UNITED NATIONS
Paper outline
• main differences between national and
international criteria
• data quality dimensions of particular relevance
to agricultural price data
• results of the self-assessment exercise on
producer price data based on the CCSA
checklist
• suggestions for improving data quality
AFCAS meeting, Algiers
12 December 2007
FOOD AND AGRICULTURE ORGANIZATION
OF THE UNITED NATIONS
1. General concept of quality of official
statistics
All agree that statistics data quality is multidimensional, but there are differences in the
criteria identified and in the vocabulary used to
define the main data quality axes.
AFCAS meeting, Algiers
12 December 2007
FOOD AND AGRICULTURE ORGANIZATION
OF THE UNITED NATIONS
1. General concept of quality of official
statistics
Eurostat’s 6 dimensions of data quality
• Relevance: degree to which statistics meet users’
needs.
• Accuracy: closeness of estimates or computations to
the exact or true value.
• Timeliness and punctuality: time lag between the
reference period and the release data, time lag between
the release date and the target date.
AFCAS meeting, Algiers
12 December 2007
FOOD AND AGRICULTURE ORGANIZATION
OF THE UNITED NATIONS
1. General concept of quality of official
statistics
Eurostat’s 6 dimensions of data quality
• Accessibility and clarity: physical conditions in which data can be
obtained, data’s information environment, whether they are
accompanied by appropriate metadata; extent to which additional
assistance is provided.
• Comparability: measures of the impact of differences in concepts
and procedures when statistics are compared between
geographical area, non-geographical domains, or over time.
• Coherence: statistics’ adequacy to be reliably combined in
different ways and for various uses.
AFCAS meeting, Algiers
12 December 2007
FOOD AND AGRICULTURE ORGANIZATION
OF THE UNITED NATIONS
1. General concept of quality of official
statistics
• Statistics Canada and OECD call Interpretability what is
Clarity for Eurostat (in particular metadata)
• The IMF includes four dimensions under Serviceability
and Accessibility and adds Pre-Requisites, Integrity,
and Methodological Soundness (procedural
dimensions for data quality).
• FAO Statistics Division is adhering to the Eurostat
taxonomy
AFCAS meeting, Algiers
12 December 2007
FOOD AND AGRICULTURE ORGANIZATION
OF THE UNITED NATIONS
2. Quality dimensions revised in an
international context
• Assessing international databases quality is an
important exercise
• Frameworks need to be adjusted
– because global datasets cover heterogeneous
countries
– international organisations have a limited normative
power on the methodologies implemented in the
individual countries.
AFCAS meeting, Algiers
12 December 2007
FOOD AND AGRICULTURE ORGANIZATION
OF THE UNITED NATIONS
2. Quality dimensions revised in an
international context
1. Shared Fundamental principles
2. Production of international data:
•
•
raw data is the output of national statistical offices
transformation/production process aims at adding value
AFCAS meeting, Algiers
12 December 2007
FOOD AND AGRICULTURE ORGANIZATION
OF THE UNITED NATIONS
2. Quality dimensions revised in an
international context
3. Quality dimensions
–
–
–
At the national level, the debate on data quality is much more
advanced than at the international level.
Many quality dimensions developed for national data sets
carry over directly to international data sets.
There are at least two specific dimensions of quality, which
apply at the international level:
•
•
(i) coverage - for how many countries/regions are data available
and
(ii) comparability - to what extent is the information for different
countries/regions comparable?
AFCAS meeting, Algiers
12 December 2007
FOOD AND AGRICULTURE ORGANIZATION
OF THE UNITED NATIONS
2. Quality dimensions revised in an
international context
4. Degree of coordination/cooperation
– harmonize practices
– develop standards, (Importance of metadata and
information technology)
– create integrated international statistical system, with
an appropriate division of labor
AFCAS meeting, Algiers
12 December 2007
FOOD AND AGRICULTURE ORGANIZATION
OF THE UNITED NATIONS
3. Characteristics of price data
• Price data are generally multi-dimensional
• A substantial amount of work has been done for
collection of price data to compile CPI, PPI and fro
calculating PPP’s
– study poverty issues and international comparison of
purchasing power.
• These studies are oriented towards well organised
consumer markets where agriculture is not clearly
visible.
• Price data concepts depends on the objective of data
collection, frequency depends on the economic
conditions at large
AFCAS meeting, Algiers
12 December 2007
FOOD AND AGRICULTURE ORGANIZATION
OF THE UNITED NATIONS
3. Characteristics of price data
Questions for price data collection
• What type of price? Can we collect actual transaction
prices rather than list price?
• Will we collect basic prices (excluding taxes, including
subsidies on products, and excluding transport costs)?
• At what time?
• How to take into account product variety, quality?
Frequency of price collection will depend on how
frequently prices change.
AFCAS meeting, Algiers
12 December 2007
FOOD AND AGRICULTURE ORGANIZATION
OF THE UNITED NATIONS
3. Characteristics of price data
Guiding principles for data collection
• How tightly defined (narrow) or general (broad)
the item specification should be an issue of
great theoretical and practical importance.
• Selection of items should be based upon a
complete list of important crops and livestock
product being produced by the country.
• The rules for selection should also take account
of the sampling methodology underlying the
selection of markets/traders.
AFCAS meeting, Algiers
12 December 2007
FOOD AND AGRICULTURE ORGANIZATION
OF THE UNITED NATIONS
3. Characteristics of price data
Guiding principles for data collection
• Special attention to seasonal items.
• Considering data collection less often than monthly for
some items, thereby enlarging total sample.
• Regular timing is particularly important when inflation is
rapid.
• Preferably days of the week and times of the month
should be chosen taking into account socio-economic
set-up.
AFCAS meeting, Algiers
12 December 2007
FOOD AND AGRICULTURE ORGANIZATION
OF THE UNITED NATIONS
3. Characteristics of price data
Collection of price data is being done by countries either using a
probability sampling techniques for getting unbiased estimates or
using a purposive sampling selection method.
Reasons for adopting non-probability sampling methods are:
• No sampling frame is available,
• Bias resulting from non-probability sampling is negligible
• Possibility that the sample can not be monitored for long-time
• Availability of staff
• Sample size is too small
• To give freedom to price collector to choose locally popular
varieties. The representative item method is more suitable for
homogeneous products.
AFCAS meeting, Algiers
12 December 2007
FOOD AND AGRICULTURE ORGANIZATION
OF THE UNITED NATIONS
4. Quality criteria applied to price data and
problems with price data received in the FAO
Relevance:
• of FAOSTAT price data: the adopted definition stems
from the SNA93 where producer prices enter into the
compilation of value of production and economic
accounts.
• FAO Statistics division has not yet developed derived
price-based indicators, whose relevance will have to be
assessed in due course.
AFCAS meeting, Algiers
12 December 2007
FOOD AND AGRICULTURE ORGANIZATION
OF THE UNITED NATIONS
4. Quality criteria applied to price data and
problems with price data received in the FAO
Countries responses
Responses
2002
2003
2004
2006
Africa
13
22
18
6
10
48
World
80
101
103
73
100
189
AFCAS meeting, Algiers
2007 total n.
12 December 2007
FOOD AND AGRICULTURE ORGANIZATION
OF THE UNITED NATIONS
4. Quality criteria applied to price data and
problems with price data received in the FAO
ACCURACY: Countries response rate
Response Rate
2002
2003
2004
2006
2007
Africa
27%
46%
38%
13%
21%
World
42%
53%
54%
39%
53%
AFCAS meeting, Algiers
12 December 2007
FOOD AND AGRICULTURE ORGANIZATION
OF THE UNITED NATIONS
4. Quality criteria applied to price data and
problems with price data received in the FAO
ACCURACY: Ratio of official to estimated data
World
Africa
Category 2003 2004 * 2005 * 2003 2004 * 2005 *
Estimate
53%
69%
70% 73%
91%
92%
Official
44%
28%
27% 25%
8%
7%
Semi-offic.
TOTAL
3%
4%
100%
100%
3%
1%
1%
1%
100% 100%
100%
100%
AFCAS meeting, Algiers
12 December 2007
FOOD AND AGRICULTURE ORGANIZATION
OF THE UNITED NATIONS
4. Quality criteria applied to price data and
problems with price data received in the FAO
Sources of inaccuracy:
• doubt on price concept adopted
• use of non standard-units
AFCAS meeting, Algiers
12 December 2007
FOOD AND AGRICULTURE ORGANIZATION
OF THE UNITED NATIONS
4. Quality criteria applied to FAO price data
TIMELINESS AND PUNTUALITY
IDEAL FRAME. DATA REFERENCE YEAR: 2006
2007
2008
May: data collection
01 Jan 2008: 2006 data available
July: replies from countries
December: validation and database
up-date
maximum 2 years-lag
CURRENT FRAME. DATA REFERENCE YEAR: 2006
2007
2008
July: data collection
2006 data available
September: 60% of replies arrived
at earliest on 31 January 2008:
Validation still on-going
AFCAS meeting, Algiers
12 December 2007
FOOD AND AGRICULTURE ORGANIZATION
OF THE UNITED NATIONS
4. Quality criteria applied to FAO price data
Time-lag all countries
Time lag - World
60
50
40
30
Lag2007
Lag2006
Lag2004
Lag2003
20
10
0
months
Lag2003
1
2
3
Lag2004
4
5
Months
Lag2006
6
7
8
Lag2007
9
AFCAS meeting, Algiers
10
11
12 December 2007
FOOD AND AGRICULTURE ORGANIZATION
OF THE UNITED NATIONS
4. Quality criteria applied to FAO price data
Time-lag African countries
Time lag - Africa
8
7
6
5
Data
4
Sum of Lag2007
Sum of Lag2006
Sum of Lag2004
3
Sum of Lag2003
2
1
0
Sum of Lag2003
1
2
Sum of Lag2004
3
4
5
Sum of Lag2006
6
7
8
Sum of Lag2007
9
10
month
AFCAS meeting, Algiers
12 December 2007
FOOD AND AGRICULTURE ORGANIZATION
OF THE UNITED NATIONS
4. Quality criteria applied to FAO price data
Accessibility and Clarity:
• Available on-line and for free
• Complemented by metadata on concepts
and definitions, country notes, currencies
AFCAS meeting, Algiers
12 December 2007
FOOD AND AGRICULTURE ORGANIZATION
OF THE UNITED NATIONS
4. Quality criteria applied to FAO price data
Comparability and Coherence:
• Comparability is impaired by the complexity of prices
that are a dynamic and multi-dimensional dataset
• Countries change concept; six African countries
replaced consumer prices with consumer prices
• Lack of metadata to distinguish between comparability
issues and revisions (accuracy)
• Coherence will be strengthened in FAOSTAT with the
development of derived indicators (Indices, value of
production)
AFCAS meeting, Algiers
12 December 2007
FOOD AND AGRICULTURE ORGANIZATION
OF THE UNITED NATIONS
5. Self-assessment checklist on FAOSTAT’s
price datasets
• tool for the systematic quality assessment of statistics
compiled by international or supranational organisations.
• It is a work in progress based on tests and inputs from
offices such as the FAO
• It has been developed within the Committee for
Coordination of Statistical Activities (CCSA) project,
• on the use and convergence of international quality
assurance frameworks
AFCAS meeting, Algiers
12 December 2007
FOOD AND AGRICULTURE ORGANIZATION
OF THE UNITED NATIONS
5. Self-assessment checklist on FAOSTAT’s
price datasets
• Generic checklist : it is designed to apply to statistics
gathered by international organisations, irrespective of
the subject matter area
• Modular approach: to account for differences in
statistical systems and functions of international
organisations and enable to tailor it to specific needs.
• Full exercise at intervals to identify strengths and
weaknesses
• Annual partial exercise to monitor quality over time
AFCAS meeting, Algiers
12 December 2007
FOOD AND AGRICULTURE ORGANIZATION
OF THE UNITED NATIONS
5. Self-assessment checklist on FAOSTAT’s
price datasets
Chapters
1. Background Information
2. Conceptual Frameworks
3. Users And Customers
4. Data Providers
5. Validation (country data)
6. Validation (international aggregates)
7. Data dissemination
8. Statistical confidentiality
9. It conditions
10. Documentation
11. Follow-up of the statistical production process
AFCAS meeting, Algiers
12 December 2007
FOOD AND AGRICULTURE ORGANIZATION
OF THE UNITED NATIONS
5. Self-assessment checklist on FAOSTAT’s
price datasets
SELECTED INDICATORS
3. Users And Customers
3/6 Availability of information on user satisfaction
3/8 Assess overall user satisfaction
5. Validation (country data)
5/2 Degree of completeness of received country data
5/3 Degree of completeness of received country metadata
5/7 Extent of unit non-response in country surveys
5/19 Distribution of countries by number of revisions
5/20Average size of revisions by the countries
6. Validation (international aggregates)
6/1 Necessity of editing received data
6/18 Average size of revisions by FAO
6/20 Overall assessment of disseminated data accuracy
AFCAS meeting, Algiers
12 December 2007
FOOD AND AGRICULTURE ORGANIZATION
OF THE UNITED NATIONS
5. Self-assessment checklist on FAOSTAT’s
price datasets
SELECTED INDICATORS
7. Data dissemination
7/12 Timeliness of preliminary results
7/13 Timeliness of final results
7/15 Is punctuality kept?
7/19 Coherence of data within the domain
7/20 Coherence in areas where it is applicable
7/23 Comparability over time
7/24 Extent of use of standard concepts and methodologies by countries
7/25 Asymmetries in mirror data
10. Documentation
– 10/6 Assessment of the completeness of metadata
AFCAS meeting, Algiers
12 December 2007
FOOD AND AGRICULTURE ORGANIZATION
OF THE UNITED NATIONS
Assessment diagram - Production (crop & livestock) data (reference
year 2006)
Sec 3/6
Sec 10/6 5
Sec 3/8
Sec 7/26
Sec 5/2
4
Sec 7/25
Sec 5/3
3
Sec 7/24
Sec 5/7
2
1
Sec 7/23
Sec 5/10
0
Sec 7/20
Sec 5/12
Sec 7/19
Sec 5/19
Sec 7/15
Sec 5/20
Sec 7/13
Sec 7/12
Sec 6/1
Sec 6/18
Sec 6/20
AFCAS meeting, Algiers
Production
Prices
12 December 2007
FOOD AND AGRICULTURE ORGANIZATION
OF THE UNITED NATIONS
Assessment diagram - Production (crop & livestock) data (reference
year 2006)
USERS
METADATA
COMPLETENES
Sec 3/6
Sec 10/6 5
Sec 3/8
Sec 7/26
Sec 5/2
4
Sec 7/25
Sec 5/3
3
Sec 7/24
Sec 5/7
2
1
Sec 7/23
VALIDATION–
COUNTRY DATA
Sec 5/10
0
Sec 7/20
TIMELINESS
Sec 5/12
Sec 7/19
Sec 5/19
Sec 7/15
Sec 5/20
Sec 7/13
Sec 7/12
Sec 6/1
Sec 6/18
Sec 6/20
Production
Prices
VALIDATION –
INTERNATIONAL
DATA
AFCAS meeting, Algiers
12 December 2007
FOOD AND AGRICULTURE ORGANIZATION
OF THE UNITED NATIONS
Summarizing quality – Assessment
diagrams and FAO Data quality stamp
Production
Prices
AFCAS meeting, Algiers
Meta data
International
classifications
Update schedule
Global coverage
Integrated - Integrated
Up-to-date
12 December 2007
FOOD AND AGRICULTURE ORGANIZATION
OF THE UNITED NATIONS
6. Suggestions for improvement
•
•
•
•
•
•
sound concepts and definitions followed by sufficient
explanations on country practices
Capacity building for not only producing primary statistics but
also secondary statistics to increase coherence of price data.
Establish a metadata system, which may provide details
about approach (sampling design, questionnaire used,
concepts and definitions) adopted by the countries to collect
data.
Regular feedback between FAO and countries indicating
inconsistency and discrepancy noted in the data.
Review all national price data sources and centralisation in
one office to progressively increase coordination, improve
resource uses, data comparability and coherence
Introduction in FAO’s annual questionnaire of check and prevalidation tools
AFCAS meeting, Algiers
12 December 2007