Evaluating the quality of
vital statistics
UN Expert Group Meeting, 27-30
June 2011
Dr AJ Lanyon
Australian Bureau of Statistics
30 June 2011
Outline
• Chapter V: some comments
• Utility of an assessment framework
and guidance tool
• Example: WHO/UQ HIS-Hub tool
• Country Developmental Pathways
• Conclusion?
2
Chapter V: comments
• Evaluating quality can be
challenging and complex
• Chap V currently provides
comprehensive overview of different
types of approaches and methods
(direct and indirect)
• Should, in general, be carried
forward
3
Chapter V: comments
• However:
– elements of the Chapter are very technical
and dense
– could benefit from reference to an
assessment framework that provides
guidance about why, when and how to
assess different elements of the quality of
vital statistics
– could consider a recommendation on how to
manage revisions of data and subsequently
vital statistics
4
Example framework
• Work by WHO / UQ HIS-Hub across
past 5 years has demonstrated value in
improvement of CR and VS systems of:
– overarching system assessment
framework
– assessment guidance tool.
• Help to:
– Identify strengths / areas for improvement
– Develop feasible action plans for CR and VS
system improvement.
5
Context
• Tool developed in light of slow
progress in a number of countries
in strengthening their CRS
• Framed to empower countries to
develop action-based plan with the
support of key stakeholders in their
CR and VS systems
6
Framework focus?
• Reviewing the system underpinning
the production, quality and use of
birth, death and cause-of-death
information
• Not just focussed on the product or
the output i.e. vital statistics
• Did not include foetal deaths as
many countries not able to collect
relevant data – could be expanded to
7
Evidence that it works . .
• Countries who have successfully used
the framework now have:
– Strong understanding of
strengths/weaknesses of their systems
– Complete understanding of key stakeholders
necessary to support effective system
functioning
– Action plan in place to achieve functional
and sustainable system and quality VS
– Intra-agency co-ordinating committee
moving forward
8
What’s in the framework?
• Inputs – Legal framework for CR and
VS systems; Registration infrastructure
and resources.
• Processes – Registration practices,
coverage and completeness; Death
certification and cause-of-death; ICD
mortality coding practices.
• Outputs – Data access, use and quality
checks.
9
What’s in the guidance tool?
• Comprises series of questions for each
sub-component
• Helps investigate how CR and VS
systems have necessary inputs and
processes in place to produce quality
outputs i.e. vital statistics
• Assists in better description of:
– any issues or problems, and
– possible strategies to address.
10
Review approach
• Promotes establishment of an interagency review team
• Required to gather relevant
(specified) evidence
• Basic assumption:
– Buy-in of key stakeholders to process
and ownership of final action plan
– Collaboration
= KEY TO SUCCESS
11
Component E1
• Data quality and plausibility checks
– Levels of fertility and mortality
– Causes of death
• Asks questions like:
– Are fertility indicators routinely calculated?
– If so, which ones?
– What data sources are used as the
denominators to calculate rates?
– Describe plausibility and consistency checks
carried out (gives examples)
12
Developmental pathways
• Rapid assessment version of the tool
has been applied in 26 Asia Pacific
countries
• Results help to differentiate between
strength of countries CR and VS
systems
• Identified 3 broad groups of countries:
– dysfunctional to weak systems:
– Functional but inadequate;
– Satisfactory (minor adjustments possible).
13
Developmental pathways
• Analysis of 26 assessments
• Cross-cutting issues:
– Coder qualification and training
alongside quality of coding
– Routine procedures to check quality of
fertility and mortality data produced
– Certification practices
– Lack of collaboration across
government departments.
14
Developmental pathways
• Analysis also re-affirmed different
countries:
– different levels of system maturity
– different strengths, weaknesses,
opportunities and threats, and
– different developmental priorities and
pathways.
• Cross-cutting issues could be
addressed through regional
programmes.
15
Conclusion
• Can’t replace technical
sophistication required to evaluate
quality of VS
• Can provide a framework to
support countries to work through
(what can be) a complex and
challenging process to understand
quality strengths and develop
strategies to address issues.
16