Quality Management of
Statistical Processes Using
Quality Gates
Narrisa Gilbert
Australian Bureau of Statistics
May 2011
Overview
•
•
•
•
•
•
Background
What are Quality Gates
Attributes of Quality Gates
Six Components of Quality Gates
Outcomes of Quality Gates
Further information
Different perspectives on quality
ABS approach to quality management
• What?
– Reinforcing a high degree of credibility for the ABS
and its outputs
– Maintaining the relevance of ABS outputs
• How?
–
–
–
–
Effective relationships with respondents
Processes that produce high quality outputs
Regular review and evaluation of statistical activities
Staff skilled and motivated to assure the quality of
ABS outputs
Background
• The Australian Bureau of Statistics produces a
lot of statistics
• On occasion the quality (accuracy) of data has
been called into question at the last minute
(right before release to the public)
•
•
Sometimes no issues are identified
Sometimes errors are discovered
• Sometimes errors in data are identified after
release
What are quality gates?
"checkpoints or decision points at various
strategic places in a statistical process at
which the quality of the process at or up to
that point is explicitly assessed"
Attributes of Quality Gates
• Planned in advance of issues occurring
• Designed to facilitate the detection,
discussion and resolution of issues and
problems with processes earlier
• Constantly reviewed and revised
Attributes of Quality Gates
• Collaborative
• Communicate openly the progress/issues
associated with the collections
• Provide a model of accountability and
responsibility for survey processes
Attributes of Quality Gates
• Knowledge Management
• Create a store of corporate knowledge
• Document and monitor issues and actions
throughout the collection cycle
Six components of quality gates






Placement
Quality measures
Roles
Tolerance
Actions
Evaluation
1. Placement



Assigning priorities
Related to risk profile of collection cycle
Want to detect errors 'upstream‘
What can go wrong?
When can this problem occur?
What impact can it have?
Placement of Quality Gates
• For example
Handover
Within
UNECE (United Nations Economic Commission for Europe) Secretariat 2009, Generic Statistical Business Process
Model, Version 4, April 2009, UNECE, Geneva.
Areas of risk




changes to processes, systems,
methodologies
data transformations
overlap / coordination / integration with
other areas
knowledge management
2. Quality measures





Choose good indicators of potential
problems
Prioritise!
Definitions
May need to drill down
Time series useful
How would we know if this problem occurred?
3. Roles






Owner area
Sign-off person
Operational person / Gate Keeper
Stakeholders
Gate definition
Gate assessment / sign-off
Who is responsible?
Who will this affect?
4. Tolerance





Driven by user requirements
Consider size of natural variation, sampling
errors, level of detail, importance of outputs
Need to form expectations
Historical data may help
Predetermined!
What is an acceptable level of quality?
5. Actions



reflect extent of the problem
reflect consequences
"traffic lights" concept
What will we do (if there is a problem)?
Who needs to be informed?
6. Evaluation


evaluation of processes
evaluation of quality gates
What has this information told us about our
quality?
How can we improve in the future?
Outcomes from Quality Gates



Business processes well-defined and
known by staff
Responsibilities for processes and quality
clearly defined
Statistical risk explicitly identified &
assessed
Outcomes from Quality Gates


Explicit sign-off occurs at Quality Gates
Quality Measures clearly defined and
understood
• Errors identified earlier (and fixed)
Further information
• Quality Management of Statistical
Processes Using Quality Gates, Dec
2010, (cat.no. 1540.0) on the ABS
website.
• http://www.abs.gov.au/ausstats/abs@.nsf/mf/1540.0
Questions
• Contact:
• Narrisa Gilbert:
narrisa.gilbert@abs.gov.au