A tool for managing the
quality of statistical
processes:
Quality Gates
Paul Schubert
Head, Statistical Services Branch
November 2011 National Statistical Service
Seminar Series
Overview
•
•
•
•
•
•
Motivation
What are Quality Gates
Attributes of Quality Gates
Six Components of Quality Gates
Outcomes of Quality Gates
Further information
Credibility and Integrity
Statistical Risk
• The chance or likelihood that something
could go wrong in the statistical process
that impacts on the quality or integrity of
the statistical outputs produced.
The statistical cycle
Efficiencies
•
•
•
•
Budgets have decreased
Demand for statistics have increased
Timeliness
Getting it ‘right’
What are quality gates?
"checkpoints or decision points at various
strategic places in a statistical process at
which the quality of the process (and data) at
or up to that point is explicitly assessed"
Attributes of Quality Gates
• Planned in advance of issues occurring
• Knowledge Management (e.g. create a
store of corporate knowledge)
• Constantly reviewed and revised
Attributes of Quality Gates
• Collaborative – provide a model of
accountability and responsibility for
collection processes
• Communicate openly the
progress/issues associated with
collections, throughout the cycle
• Proactively managing risk – can save
time and $ in the long run
Six components of quality gates






Placement
Quality measures
Roles
Tolerance
Actions
Evaluation
1. Placement



Assigning priorities
Related to risk profile of collection cycle
Want to detect errors 'upstream‘
What can go wrong?
When can this problem occur?
What impact can it have?
Placement of quality gates
Handover
Within
Areas of risk




changes to processes, systems,
methodologies
data transformations
overlap / coordination / integration with
other areas
knowledge management
2. Quality measures





Choose good indicators of potential
problems
Prioritise!
Definitions
May need to drill down
Time series useful
How would we know if this problem occurred?
3. Roles





Owner area
Operational person / Gate Keeper
Stakeholders
Gate definition
Gate assessment / sign-off
Who is responsible?
Who will this affect?
4. Tolerance





Driven by user requirements
Consider size of natural variation, sampling
errors, level of detail, importance of outputs
Need to form expectations
Historical data may help
Predetermined!
What is an acceptable level of quality?
5. Actions



reflect extent of the problem
reflect consequences
"traffic lights" concept
What will we do (if there is a problem)?
Who needs to be informed?
6. Evaluation


evaluation of processes
evaluation of quality gates
What has this information told us about our
quality?
How can we improve in the future?
Example of a Quality Gate
• Information paper 1540.0 Quality Gate
example
Outcomes from Quality Gates



Quality, Business processes and
responsibilities well-defined and known by
staff
Statistical risk explicitly identified &
assessed
Explicit sign-off occurs at Quality Gates
Outcomes from Quality Gates

Quality Measures clearly defined and
understood
• Errors identified earlier (and fixed)
• Improved analysis of data throughout the
process in relation to fitness for purpose
of outputs
Outcomes from Quality Gates
• Efficiencies long term
– Errors identified earlier
– Processes evaluated in terms of quality
management (improvements identified)
• Possible time reductions in when ‘things’ are done
• Cost savings due to changes in ‘how’ things are
done
– Reduction in ‘wasted’ resources because of
not identifying errors until too late
Outcomes from Quality Gates
• Continuous improvement
– Quality and issues identified during process
can be used to help declare the quality of the
statistical outputs
Further information
• Quality Management of Statistical
Processes Using Quality Gates, Dec
2010, (cat.no. 1540.0) on the ABS
website.
• http://www.abs.gov.au/ausstats/abs@.nsf/mf/1540.0
• Quality Page on ABS website – December 22nd, 2011