The South African Statistical Quality
Assessment Framework (SASQAF)
Presentation made at the Conference on Data
Quality for International Organisations
Helsinki, Finland, 6–7 May 2010
Seble Worku
National Statistics System Division
Statistics South Africa
1
Background
• The broader National Statistics System (NSS) is
characterized by three gaps
Capacity gap
- Insufficient statistical skills
- Inappropriate placement of skilled personnel
Quality gap
- Preponderant usage of data of uncertain or
unknown quality
Information gap
- Insufficient supply of statistical information
- Insufficient supply of data on second economy
• To address the Quality Gap the Statistician General has
gazetted through parliament the South African Statistical
Quality Assessment Framework (SASQAF) on the 23rd
September 2009
• SASQAF quality requirements must be met by all statistics
producers in the NSS to qualify as “Official Statistics”.
2
The Statistics Act No. 6 of 1999
• Stats SA is governed by the Statistics Act (No. 6 of
1999)
• The Stats Act covers “Official” Statistics and “Other”
statistics
• Implies, Stats Act covers all statistics produced that
informs policy, planning and monitoring of government
performance.
• Section 14(7) of the Act, empowers the StatisticianGeneral to “designate as official statistics any statistics
or class of statistics” produced by Stats SA or any
organ of state
 Required that a rational, sustainable and
transparent framework for assessing the quality of
those statistics being developed
 South African Statistical Quality Assessment
Framework (SASQAF) has been developed for
this purpose
3
Definition
Official statistics’ definition is statutory – see
Statistics Act [No. 6 of 1999]
Official statistics are statistics designated as official
statistics by the Statistician-General within the provisions of
the Statistics Act
Practical criteria of official statistics
– Must be used in the public domain
– Are from organs of state and other agencies that are
partners in the National Statistics System [NSS]
– Are sustainable
– Have met quality criteria as defined by the StatisticianGeneral [SASQAF]
National statistics’ definition is implicitly statutory
National statistics are statistics not designated as official
Statistics by the Statistician-General
4
Structure of the framework
• Provides a structure for the assessment of statistical
products based on
1. Dimensions of quality,
2. Indicators,
3. Standards,
4. Benchmarks
• Each of the 8 quality dimensions consists of number
of indicators
• Within the indicators a number of benchmarks are
identified relating to a 4-point scale;
1. Quality statistics
(4)
2. Acceptable statistics
(3)
3. Questionable statistics (2)
4. Poor statistics
(1)
5
The Dimensions of Quality
Integrity
Relevance
Methodological
soundness
Free from political interference:
Adherence to objectivity,
professionalism, transparency,
ethical standards
Coherence
Harmonisation of
different info
within broad
analytical
and temporal
framework
SASQAF
Sound methodologies:
•International standards and
guidelines - good practice
•Agreed practices
•Dataset-specific
Accuracy
Interpretability
Accessibility
Availability of
supplementary
info and metadata
Meeting real
needs of clients
Ease of obtaining
info from agency
6
Timeliness
Correctly describes
phenomena it is
designed to measure
Info available at
desired reference
point
Indicators and Standards
1st Edition
Dimension
2nd Edition
Indicators
Indicators
Standards
Prerequisites of Quality
7
8
21
Relevance
7
5
5
Accuracy
7
7
36
Timeliness
5
4
10
13
12
13
Interpretability
3
3
3
Comparability and
Coherence
5
5
12
Methodological
soundness
6
6
14
Integrity
6
6
6
59
56
120
Accessibility
Quality
7
Layout of the framework
Quality dimension
0.
Prerequisites of quality
Description
Refers to the institutional and organisational conditions that have an impact on data quality.
Assessment Levels
Key components
Legal and
institutional
environment
(including
Memoranda of
Understanding
(MoUs) or Service
Level Agreements
(SLAs)
Privacy and
confidentiality
Resources are
commensurate with
the needs of
statistical
programmes
Quality is the
cornerstone of
statistical work
Indicator
Quality Statistics
Acceptable
Statistics
Questionable
Statistics
Poor Statistics
Level 4
Level 3
Level 2
Level 1
The responsibility
for producing
statistics is clearly
specified.
The responsibility
for producing
statistics is
explicitly
specified through
a legal
framework.
The responsibility
for producing
statistics is
specified through
a legal
framework.
The responsibility
for producing
statistics is implied
through a legal
framework.
The
responsibility for
producing
statistics is not
specified.
Standards and
policies are in place
to promote
consistency of
methods and
results.
All standards and
policies are in
place to promote
consistency of
methods and
results, and are
adhered to.
The majority of
standards are in
place to promote
consistency of
methods and
results.
Some standards
are in place to
promote
consistency of
methods and
results.
No standards are
in place to
promote
consistency of
methods and
results.
Data sharing and
coordination among
data-producing
agencies is clearly
specified and
adhered to.
Data sharing and
coordination
among dataproducing
agencies is
explicitly
specified through
a legal
framework.
Data sharing and
coordination
among dataproducing
agencies is
specified through
a legal
framework.
Data sharing and
coordination
among dataproducing
agencies is
implied through a
legal framework.
Data sharing and
coordination
among dataproducing
agencies is not
specified.
8
Quality Standards
•Indicator:
The responsibility of producing statistics is clearly
specified.
•Standard:
A legal arrangement exists that explicitly mandates
the production of statistics.
•Benchmarks:
Quality statistics: A law or legal arrangement
exists that explicitly provides the mandate for the
production of statistics;
Poor statistics: No arrangement exists.
9
Purpose of the framework
SASQAF provides an universal framework for
assessing the quality of statistics within Stats SA
and the NSS
•Reviews within the NSS: certification of data
• Self-assessment by data-producing agencies:
rate own performance
• Assessment by data users based on quality
declaration: brings trust in data
• Assessment by international agencies: ease
international comparison
10
Purpose of the framework
•SASQAF provides an universal framework for
assessing the quality of statistics within Stats
SA and the NSS
•Reviews within the NSS: certification of data
• Self-assessment by data-producing
agencies: rate own performance
•Assessment by data users based on quality
declaration: brings trust in data
•Assessment by international agencies: ease
international comparison
11
Conclusion
Adherence to SASQAF will
•
encourage compliance to the agreed standards,
procedures and guidelines
 resulting in improvement of quality, closing the quality
gap
•
ensure that more statistics are certified as official
 closing the supply gap
•
assist the users in assessing the quality of data and
products
 promoting transparency
•
ensure that all published products include statements
about data quality (quality declaration)
 informing users about the quality of data and products
•
ensure that more statistical products produced in the
NSS are declared as “fit for use”
12
Thank you
13