Process Quality in the Office for National Statistics
Sarah Green1, Rachel Skentelbery and Rachael Viles,
Quality Centre, Methodology Directorate, Office for National Statistics
Paper presented at the Thirteenth Government Statistical Service Methodology
Conference: London, 23rd June 2008.
1. Overview
Quality is one of the key principles of the United Kingdom (UK) National Statistics
Code of Practice. At the European Union (EU) level, the requirement to improve the
measurability of statistical outputs and processes is embedded in the European
Statistical System and also, progressively in legislation.
The aim of the Process Quality Project in the UK Office for National Statistics (ONS)
is to develop a Continuous Quality Improvement (CQI) framework for the statistical
production processes, and tools to enable this with the view to roll out across the
ONS and wider Government Statistical Service (GSS).
An ONS specific version of the Eurostat 'Handbook for Improving Quality by analysis
of Process Variables' has been developed.
The handbook will assist the
implementation of the CQI techniques into the ONS, and includes updated, ONS
specific examples. Additionally it provides guidance on applying the technique for
each step of the Statistical Value Chain (SVC) as well as links to a defined set of
process quality measures and E-Learning courses.
A comprehensive list of quality measures for statistical processes has also been
developed to support the handbook. By providing a list of process measures and
indicators for the statistical production process, it will enable producers to measure
and monitor the quality of their processes over time and to facilitate CQI. Process
measures also provide effective feedback to allow managers to be responsive to
problems and objectively highlight areas for improvement. The document promotes
a standardised approach to measuring process quality.
2. Introduction
The word ‘quality’ has many different meanings, depending on the context in which it
is used. All users have different expectations and as such the definition of quality is a
relative one, allowing for various perspectives on what constitutes quality, depending
on the intended uses of the products. Quality can therefore be most usefully defined
in terms of how well products meet user needs, or whether they are ‘fit for purpose’.
Indeed assessing the quality of statistical outputs is concerned with providing
sufficient information to judge whether or not the data (products) are of sufficient
quality for their intended use(s).
Within an organisation there are a number of different perspectives of quality (figure
1). Ultimately users are concerned with the quality of products but to achieve this
there are other areas where quality must be considered. Firstly, business quality:
how the organisation is run and administered; following this, the quality of processes
Address for correspondence: Sarah Green, Office for National Statistics, Government Buildings,
Cardiff Road, Newport, Gwent, United Kingdom, NP10 8XG (sarah.green@ons.gov.uk)
must be considered. These three aspects all contribute to the reputation of the
organisation which has a direct impact on users’ perceived level of quality of the
organisation and its products. It can be seen, therefore, that process quality is an
integral component of any organisation and as such it is vital it is understood and
controlled within the ONS. The word ‘quality’ has many different meanings,
depending on the context in which it is used. All users have different expectations
and as such the definition of quality is a relative one, allowing for various
perspectives on what constitutes quality,
Figure 1: The relationship between the different types of quality that affect the statistical
outputs produced by ONS (adapted from Statistics Sweden model)
Reputation
Business
Quality
Process Quality
Product Quality
User
Perceived
Quality
The importance of process quality to a statistical organisation was highlighted by the
Eurostat Leadership Expert Group (LEG) final report (Eurostat (2002)) which states
that:
'in theory, good product quality can be achieved through evaluations and rework.
However, this is not a feasible approach since it is costly and time consuming.
Instead, it is believed that product quality will follow from improvements in process
quality'
Therefore for any organisation with constraints on business, process quality is the
sensible way forward. The LEG recognised that improving process quality has a
direct effect on product quality, and that this is often more feasible than improving
product quality directly.
The LEG called for the development of a handbook on the identification,
measurement and analysis of key process variables, which are those factors that can
vary with each repetition of the process and have the largest effect on critical product
characteristics. The “Handbook on improving quality by analysis of process
variables” (Eurostat (2004)) was produced by a project team consisting of members
from the National Statistics Institutes of Greece, Portugal, Sweden and the UK, who
led the project. The Eurostat handbook consists of guidance on how to identify,
measure and analyse process variables. Once a general approach was developed,
examples of applying the method to statistical processes were identified. These
examples are presented in the handbook, categorised into the key processes used to
produce statistics.
This paper summarises two of the initiatives taken by ONS under the process quality
project to develop a continuous quality improvement framework for statistical
production processes, and the tools to enable this, with the view to roll out across
ONS and wider GSS.
3. ONS Process Quality Handbook
The “Handbook on improving quality by analysis of process variables” is a long and
technical document. While it is ideal for reference it is not easily accessible in order
to roll out process quality techniques across ONS. Due to this, one of the initiatives
from the process quality project was to produce a summarised version of the
Eurostat handbook for ONS. This was to include updated guidance on improving
quality by analysis of process variables and examples relevant to ONS.
The ONS Process Quality Handbook describes methods for improving quality by
analysis of processes based on the Eurostat handbook. This handbook applies the
general approach taken in the Eurostat handbook to the ONS and describes useful
tools for the task of identifying, measuring and analysing key processes in order to
achieve Continuous Quality Improvement (CQI). Section 2 includes guidance and
tools while Section 3 looks specifically at examples of the application to ONS
statistical processes using the ONS Statistical Value Chain (SVC). The ONS SVC
was developed during 2001 and describes the key ONS processes to produce
statistics. There exist several other similar representations of the statistical
production cycle, for example Eurostat uses a tool called the Cycle des Vie de
Données (CVD) or Data Life Cycle.
The ONS handbook does not aim to provide a list of recommended variables to
measure process quality across all statistical processes; such measures are being
developed in a separate process quality measures document. It is however a
component of the process quality tool kit being developed in the ONS.
Process quality can be judged using Key Process Variables. A process variable is a
factor that can vary with each repetition of the process; those that are key are those
which will have a large effect on key characteristics of the output. Morganstein and
Marker (1997) developed a standardised approach to process quality, that will
produce an output (product) meeting user needs. The ONS handbook uses an
adaptation of this approach, Figure 2, tailored to be specific to ONS needs. The
standardised approach is planned in a process map made up of six steps. The first
three steps cover identifying, the fourth measuring, the fifth and sixth analysing key
process variables.
Figure 2: Implementing and improving process quality for ONS processes, a step-by-step guide
(adapted from Morganstein & Marker, 1997)
Step 1
Identify key characteristics that
will make the product of the
process fit user requirements
Step 2
Develop a process map
Step 3
Determine key process
variables that will impact on the
key characteristics
Step 4
Evaluate how to measure the
key process variables
Adequate measures
available?
NO
Make Changes
NO
Review or develop
process to eliminate
instability
NO
Make process change
YES
Step 5
Monitor the stability of the key
process variables
Stable process variables?
YES
Step 6
Determine if the process
produces an product that, over
time, continuously meets user
requirements
Process capable?
YES
Establish a system for the
continuous monitoring of the
process
Clearly, a vital part of this approach is the ability to identify the key characteristics of
the process that will produce an output to fit user requirements. It is not enough to
identify a process; in order to implement process quality and effectively achieve CQI
it is essential to understand the internal and external output characteristics that are
important to users; these are the key characteristics.
An important consideration when identifying key characteristics is to ensure adequate
coverage of the European Statistical System (ESS) dimensions of quality through the
processes to be monitored. The six dimensions of quality for statistical outputs are:






Relevance: a statistical product is relevant if it meets user needs.
Accuracy: accuracy is the difference between the estimate and the true parameter
value.
Timeliness: this is important for users since it is linked to an efficient use of the
results.
Accessibility and Clarity: results have high value when they are easily available
and understood.
Comparability: reliable comparisons across space and time are often crucial.
Coherence: statistics originating from different sources are coherent insofar as
they are based on common definitions, classifications and methodological
standards.
Identifying the key characteristics will also identify the processes to focus on. Once a
process has been identified for monitoring and the key characteristics identified, the
next step is to map the process by developing a comprehensive flow chart of the
process, known as a process map. Process maps can be used to take an objective
look at a process and identify:




who owns the process or parts of it
sources of variation in the process
ways to monitor the process
any activity that uses resource but adds no value
An effective process map will establish the following:




pictorial representation of a process
clearly defined start and end points
the flow of the process and decision points within it
what actually happens in a process - not what 'ideally' should occur.
Once the process is fully understood it can be looked at more closely to identify,
monitor and analyse key process variables.
3.1 Identifying, Monitoring & Analysing of Key Process Variables
Process variables observed conveniently and continuously during the process can be
used to assess the quality of the process and as such control the quality of the
output.
It would not be efficient or effective to measure all process variables and so the aim
must be to identify those factors that can vary with each repetition of the process and
have the largest effect on key characteristics of the output, that is the key process
variables.
A simple and effective tool for identifying key process variables is the Pareto chart.
This tool tries to find the relatively few error types that account for the majority of all
errors, hence enabling staff to be more effective in allocating resources. The error
types identified, that account for the majority of all errors can be labelled as the key
process variables to measure over time.
Figure 3 is an example of using a Pareto chart to identify key process variables in
ONS e-commerce survey. With an aim of reducing item non-response for ecommerce it can be seen that the process variable is the contribution to item nonresponse of each question. The Pareto chart shows that out of thirty questions (on
the e-commerce questionnaire) four are identified as contributing to 33% of the total
item non-response for the survey. As such these four questions will be the key
process variables to consider.
Pareto
of contribution
toVariables
total itemin E-Commerce Survey
Figure 3: Example of Pareto
Chart chart
to Identify
Key Process
80%
100%
70%
90%
70%
50%
60%
40%
50%
30%
40%
33%
30%
20%
Cumulative %
80%
60%
Contribution
Contribution to Non-response
non-response by question
20%
10%
10%
0%
0%
11
16
27
25
All other
Question
Some process variables can not be summarised quantitatively in a Pareto chart.
Cause & Effect diagrams can identify key process variables when numeric
information on the variables is not available.
After identifying the key process variables, it is important to know how accurately
they can be measured. Ideally this measurement should require as little effort as
possible on the part of the producers. The key points to consider when evaluating
how to measure the key process variables are:




what can be measured that is useful to understanding and monitoring
the system
can the measures be used to assess changes in the quality
what does success look like
can the measures lead to potential for improvement
In the same way that efficiency and effectiveness are ensured by identifying a subset
of process variables that are key to measure, it may not be efficient or effective to
measure a key process variable continuously in order to assess the quality of the
process. If this is the case then a sampling and verification approach could be used:
assessing a sample of work from the process.
Figure 4 is an example of a measuring key process variables for ONS UK Census,
monitoring the accuracy of occupation coding for each estimation area.
Figure 4: An example of measuring key process variables in ONS UK Census
UK Census uses occupation coding and the accuracy of this was identified as a key
process variable when looking to monitor and improve processes.
Rather than checking all codes a 2% sample of codes was verified for each
estimation area, independently of coders.
Sample data were used to estimate the key process variable which could be
measured as a consistency rate.
The consistency rate can be considered over time to identify any positive or negative
changes in quality.
Success will be consistently reaching an agreed level of consistency in the accuracy
of coding.
If the consistency rate is found to be too low or drops then the process can be
considered to see where problems may occur that lead to this and how these
problems can be rectified. In this case data checks may be required at more
strategic points in the process.
Once key process variables have been identified and measured, analysis of them
can begin. This begins by testing them for stability: process stability is a state where
the process variation consists entirely of random components, that is when the
variation is not systematic.
In the ONS handbook, the main tools highlighted to monitor stability are control
charts and analysis of Pareto charts. Through this analysis step, we can identify the
sources of process and output variations, and examine the effects that changes to
the process have on variation. Control charts have control limits that are typically
calculated at three standard deviations from a centre line (possibly a group average).
Observations that fall out of the control limits are deemed incapable or 'out-ofcontrol'. A special cause for out-of-control observations should be identified and
addressed. Tools such as control charts can be used to measure the effectiveness of
improvements and accurately predict likely outcomes in the future.
Figure 5 is an example of a control chart for the ONS UK Census, monitoring the
accuracy of occupation coding for each estimation area as discussed in Figure 4.
Figure 5: Example of control chart monitoring consistency rates in ONS UK Census
Control Chart of Consistency Rate
0.99
Consistency
Lower bound
Upper bound
Centre Line
0.98
0.97
0.96
Rate
0.95
0.94
0.93
0.92
0.91
0.90
0.89
06-Apr-2002
06-Mar-2002
03-Feb-2002
03-Jan-2002
03-Dec-2001
02-Nov-2001
02-Oct-2001
01-Sep-2001
01-Aug-2001
01-Jul-2001
0.88
Scanning End Date
Points of interest in this example are clearly from February to April 2002 where the
consistency rate is out of control and over the top limit. This examples shows that
the limits are in place in order to help spot good and bad special causes of variation
in processes. The higher-than-anticipated rate should be investigated in as much
detail as if the consistency rate had dropped below the lower limit: when things get
better spontaneously it is important to investigate the reasons for this as it may be
something that can be incorporated permanently into the process.
Once the key process variables have been identified, measured and monitored then
it must be determined whether the process produces an output that, over time,
continuously meets user requirements. For processes that are reasonably stable,
staff can determine the limits of the expected process variation, or evaluate the
capability of the process to predictably meet specifications defined and driven by
user needs. A stable process is ‘capable’ if its random variation is such that the
system will consistently meet the user requirements or limits for that process.
Once a process has been shown to be capable of meeting user requirements, it is
important to ensure that a system of continuous quality improvement is implemented
and maintained. A continuous monitoring system is needed to:





keep staff informed
provide feedback
assist in controlling process variation
help achieve continuous reduction in process variation through
improved methods
evaluate process changes
It should also aim to reduce the burden of measuring and monitoring by automating
or improving procedures. Effective procedures for managing the implementation of
identified improvements are important.
4. Process Quality Measures
To complement the ONS Process Quality Handbook, a definitive set of quality
measures are being developed. This set of process quality measures will enable
producers of ONS statistics to measure and monitor processes over time, be
responsive to problems and highlight areas for improvement. Moreover, it should
create a standardised approach to measuring ONS process quality and ensuring
comparability between outputs, since standard quality measures could be checked
across ONS. It will also provide a means for users to quality assure their processes,
again in a standardised manner across the organisation.
The ONS process quality measures are being designed to be counterpart to the
statistical output ONS Guidelines for Measuring Statistical Quality. Measures will link
to steps in ONS SVC and one of the six process quality attributes documented by the
Eurostat LEG. The process quality attributes are an accompaniment to the ESS six
dimensions of quality for outputs. As with the dimensions the attributes are
definitions to usefully describe and determine process quality, which would otherwise
be intangible and certainly inconsistent across processes.
The definitions, from the LEG, for each of the process attributes are as follows:






Effectiveness: successfully delivering the desired outcomes, meeting business
needs.
Efficiency: producing the desired outcomes cost effectively.
Robustness: delivering results against challenging demands.
Flexibility: processes that readily adapt to changing needs and demands.
Transparency: processes that are open, visible and easily understood.
Integration: processes that are complementary and consistent with other
processes.
In terms of considering standardised approaches for measuring process quality the
most important aspect of the process quality measures work will be to derive a
subset of key process quality measures (KPQMs). For the Guidelines for Measuring
Statistical Quality, out of over 160 quality measures and indicators a subset of 12 key
quality measures (KQMs) has been identified (Annex A). The KQMs are considered
to be the most important and informative in giving users an overall summary of output
quality and it will be important to have the same approach for process quality.
5. Implementation
Process quality is not a new concept in any organisation.
However the
implementation of process quality measurement, monitoring and improvement, as an
inherent part of business-as-usual in organisations is challenging to achieve. Even in
organisations or areas where variables within processes are measured it can often
be seen that this information is not used to its full advantage to achieve continuous
quality improvement.
As an organisation ONS wishes to continually achieve acceptable quality processes
and make continuous quality improvement where necessary. This being the case, it
is important that the ONS Process Quality Handbook and the associated process
quality measures are rolled out effectively to ONS. As such a progressive approach
to roll out across ONS is to be taken.
Roll out of process quality will largely be achieved through piloting the ONS
Handbook in specific statistical output areas. If gains in process quality can be made
in these pilot areas and they can quantify the benefits they have found from
implementing the approach then these success stories can be used to create interest
from other areas of the office.
Bespoke support to statistical output areas will be provided from ONS Quality Centre,
since from examination it can be seen that processes fall into two categories:
 those amenable to the whole process quality approach, including identifying
measurable process variables
 those where mapping the process and identifying key factors may lead to
valuable quality improvement actions
Such support will broaden the process quality knowledge of Quality Centre as well as
enable all areas of the office to have sufficient and effective process quality
procedures in place.
Detailed information about when and how to use the corporate process quality tools
described in the ONS Handbook will be addressed through e-learning courses and
supported by expert led training sessions where required.
References
Eurostat (2002) Quality in the European Statistical System - The Way Forward,
Office for Official Publications of the European Communities: Luxembourg.
Eurostat (2004) Handbook on improving quality by analysis of process variable.
Jones N and Lewis D (2004) Developing a handbook on improving quality by
analysis of process variables, Paper for the European Conference on Quality and
Methodology in Official Statistics: Mainz, Germany, 24-26 May 2004.
Morganstein D and Marker D A (1997) Continuous Quality Improvement in Statistical
Agencies, in Lyberg L, Biemer P, Collins M, De Leeuw E, Dippo C, Schwarz N, and
Trewin D (eds.), Survey Measurement and Process Quality, New York: Wiley, pp.
475-500.
ONS (2007) Guidelines for Measuring Statistical Quality v3.1
ONS (2008) ONS Process Quality Handbook, forthcoming publication.
Annex A
UK Office for National Statistics: Key Quality Measure (KQMs)
Key Quality Measures have a dual function: to provide a summary of overall output
quality, and to provide management information to monitor office performance. In
their latter function, they fulfil the requirement for statistical key performance
indicators (other non-statistical key performance indicators may include Human
Resources or financial information, but these are not covered here).
Key Quality Measures can be either quantitative (that is calculated with standard
formulae) or qualitative (that is textual statements). For both, guidance is available:
formulae are given for quantitative measures; for qualitative measures examples are
given as well as 'things to consider' when completing the quality measure. Guidance
is available the National Statistics website in the Guidelines for Measuring Statistical
Quality at http://www.statistics.gov.uk/qualitymeasures.
It is important to note that not all of the KQMs will be relevant to all outputs. When
writing a quality report only relevant KQMs should be reported upon.
Key Quality Measures
Where possible, describe
how the data relate to the
needs of users
Provide a statement of the
nationally/ internationally
agreed definitions and
standards used
Unit response rate by subgroups, weighted and
unweighted
Key item response rates
Editing rate (for key items)
Total contribution to key
estimates from imputed
values
Estimated standard error for
key estimates
Compare estimates with
other sources on the same
theme
Estimated mean absolute
revision between provisional
and final statistics
Identify known gaps between
key user needs, in terms of
coverage and detail, and
current data
Time lag from the reference
date/period to the release of
the provisional output
Time lag from the reference
date/period to the release of
the final output
Quantitative / Qualitative
(Included in)
ESS Quality Dimension
Qualitative
(Summary Quality Report)
Relevance
Qualitative
(Summary Quality Report)
Comparability
Quantitative
(Basic Quality Information)
Accuracy
Quantitative
(Basic Quality Information)
Quantitative
(Basic Quality Information)
Quantitative
(Basic Quality Information)
Quantitative
(Basic Quality Information)
Accuracy
Accuracy
Accuracy
Accuracy
Qualitative
(Summary Quality Report)
Coherence
Quantitative
(Basic Quality Information)
Accuracy
Qualitative
(Summary Quality Report)
Relevance
Quantitative
(Summary Quality Report)
Timeliness and punctuality
Quantitative
(Summary Quality Report)
Timeliness and punctuality