FOCUS 2011
NATIONAL AUDIT OF INTERNAL
QUALITY CONTROL PRACTICE in UK.
David Housley.
Luton and Dunstable NHS Trust.
Teresa Teal.
National ACB Audit Group.
INTRODUCTION
• A previous audit of IQC within the Thames Region in
2006 showed significant variation in practice.
• 29 returns were received.
• Some areas of very poor performance were identified.
• 9 audit standards were developed and published 2008.
Housley D G et al; Ann Clin Biochem; 45: 135–139.
• AIM: To re-audit IQC practice nationally and compare to
previous audit data from a single area.
METHODS
• Questions based on the previous audit, with
minor modifications to improve clarity, facilitate
electronic data gathering and to increase focus
on audit standards.
• Questionnaire designed using SurveyMonkeyTM
• Distributed by e mail in March 2011 via ACB
office with a hyperlink to the questionnaire.
• Circulated to one individual in each UK hospital.
• Respondents were instructed to restrict answers
to routine chemistry systems only (not endocrine
or esoteric).
Q. 1 & 2
• 86 responses received.
• ANALYTICAL SYSTEMS IN USE;
–
–
–
–
–
–
•
41%
16%
8%
14%
13%
8%
Roche
Beckman - Coulter*
Olympus*
Abbott
Siemens
Ortho
*note change of ownership since previous audit.
Q.3
• How frequently is internal quality control material run
through your analyser (assume standard core hours
(weekday) with no analytical, technical or procedural
problems that require extra analysis)?
• Please answer for serum / plasma assays only.
• 53%
• 64%
• 6%
At set time intervals
At fixed times of the day
After a set number of samples or tests
– Responses ranged from hourly through to 24 hourly.
– Most common responses ranged from 4 to 8 hourly.
86 answered question
Q.4
• Please state what your lab considers to be its core
working hours and whether you operate a shift system or
on-call system outside of this period?
• Core working hours of labs are variable but most
common answers approximated too:
– 62%
09:00 – 17:30
– 26%
08:00 – 20:00
– 3%
08:00 – 18:00
– 5%
07:00 – 22:00
– 4%
24/7 shift
85 answered question
Q.5
• Does frequency of IQC outside of weekday core
hours differ to the frequency within core hours?
24.5%
75.5%
YES (21% previously)
NO
Mostly this is a reduction in the frequency of
IQC, but significant evidence of discretionary
testing still remains.
Q. 6
• Does your laboratory have a policy for IQC practice
outside of weekday core hours ?
•
YES = 58% (30% previously)
– NO = 42%
86 answered question
Q.7
• According to your laboratory sample throughput, what is longest run
length (number of patient specimens processed between IQC
assessment) for a plasma / serum sodium measurement ?
1400
9000
1200
8000
7000
6000
1000
5000
4000
800
3000
2000
1000
600
0
Roche
Beckman
Abbott
Olympus
Siemens
Ortho
400
200
0
Roche
Beckman
Abbott
Olympus
Siemens
Ortho
59 answered question
Q.8
• Does your laboratory preferentially utilise IQC material
from a third party source if it is available ?
• YES = 85%
• NO = 15%
• Unchanged from previous regional audit.
85 answered question
Q.9
• Which of the following are used routinely (every IQC
assessment)?
• 44%
• 27%
• 61%
Unassayed (no values assigned) material
Assayed (quoted values used)
Assayed (values assigned by lab)
If you assign your own values, how many measurements are
made to derive the target and SD ?
Replicates
Number of
labs
100
3
80
1
50
3
40
1
30
10
25
1
20
23
10
6
Q.10
• How are the target concentrations for IQC
material chosen ?
–
–
–
–
–
–
60%
81%
86%
42%
35%
5%
What is available on the market
Clinically important decision levels
To cover low, normal and high values
Detection limits of assay
To assess areas of poor assay performance
Other reasons
– custom made material, stability, lot size, cost.
85 answered question
Q.11
• Are multi-rules (e.g. Westgard rules) used as the primary
form of QC data evaluation for accepting / rejecting QC ?
YES = 60.5%
NO = 39.5%
Free text replies showed a lot of confusion.
Very similar percentages to previous regional audit.
57 answered question
Q.12
• If you do use multi-rules (e.g. Westgard
rules), do you apply the same rules to all
analytes, or do you use method (analyte)
specific rules ?
• 89.5%
• 10.5%
Same for all
Method specific
• Free text replies showed a lot of confusion.
Q.13
• Do you use single 2 SD rules ?
• 55%
• 45%
YES
NO
(76% in previous regional audit)
• Fewer laboratories using single 2SD rules.
• Free text replies varied from ‘all analytes’ to single
analytes, most commonly sodium / ISE.
85 answered question
Q.14
• How are your control limits set ?
– 92%
– 46%
– 31%
– 11%
Data generated within laboratory
Cumulative data
Manufacturer derived ranges
Peer group or EQA data
– Varies with different analytes.
Q.15
• Do you consider the IQC data on your main
chemistry analyser easy to access ?
• 84%
• 16%
YES
NO
86 answered question
Q.16
• What is the typical response when IQC data
suggests a method is out of control (e.g. failed QC) ?
–
–
–
–
67%
73%
87%
6%
Repeat using existing control material
Re-run using new control material
Re-calibrate then re-run control
Over-ride then continue with patient samples
– Free text comments from all.
– Would depend on whether 2SD being used as a warning or multi
rules are violated.
– Question could have been more specific.
Q.17
•
What grade / type of staff make decisions about whether to accept or reject
IQC data in real time ?
6%
5%
95%
96%
82%
44%
26%
20%
36%
32%
Trainee / MTO
Qualified (not registered)
BMS 1
BMS 2
BMS 3
BMS 4
Locum BMS
Quality manager
Clinical Scientist
Consultant / Specialist registrar
Very similar results to previous regional audit.
85 answered question
Q.18
• If IQC data showed that a method was out of control (i.e. QC had
failed), what grades of staff would make the decision to continue
processing patient samples ?
18%
WOULD NEVER HAPPEN IN MY LAB
0%
1%
15%
61%
65%
40%
3.5%
20%
49%
50%
MTO / Qualified (not registered)
Trainee
BMS 1
BMS 2
BMS 3
BMS 4
Locum BMS
Quality manager
Clinical Scientist
Consultant / Specialist registrar
85 answered question
Q.19
• How often is out of control (non-ideal) IQC accepted
(e.g. in order to ensure work is completed) ?
–
–
–
–
–
–
7%
7%
2%
55%
26%
5%
Daily
Weekly
Monthly
Rarely
Never
Other
– Other reasons were ‘known problem’; ‘reagent or calibrator
change’; ‘depends on clinical significance’; ‘only for non-critical
tests such as protein or bilirubin’; ‘we know we get fliers’.
84 answered question
Q.20
• What grade of staff monitors overall IQC
performance ?
–
–
–
–
–
29%
86%
53%
61%
26%
Quality manager
Senior in charge of section
Clinical scientist
BMS3 or BMS 4
Consultant
85 answered question
Q.21
• Do you formally compare performance between
multiple analysers (same site or across sites)?
• 71%
• 29%
YES
NO
• Very similar to previous regional audit.
• Very wide range of statistical processes in use.
84 answered question
Q.22
• If your service is split between geographically distant
sites (e.g. different hospitals) are there barriers that
prevent IQC performance being compared for the same
type of analysers located on the different sites ?
• 20%
• 80%
YES (34% in previous regional audit)
NO
• Barriers include lack of time, poor IT, staff
pressures, time pressures, poor cooperation.
60 answered question
Q.23
• If an assay is judged not to be performing
adequately (poor IQC performance), what systems
are used to prevent patient results being released ?
–
–
–
–
79%
33%
32%
39%
Chemistry is disabled
Data not transferred to LIS
Results not validated
Relies on operator not to report
85 answered question
Q.24
• Please state how long your laboratory stores IQC data
for. If the data is not readily retrievable for the whole
duration, please additionally state how long it is in a
readily retrievable form?
–
–
–
–
–
–
–
7%
10%
13%
1%
24%
1%
28%
– 16%
<1 year
<3 years
5 years
7 years
10 years
30 years
Indefinite
For lifetime of analyser
76 answered question.x
Q.25
• Does your audit trail allow you to identify
retrospectively for any patient sample, the
IQC data/performance relating to analysis?
• 89% YES
• 11% NO
• Free text comments gave as little as 2 hours and
one week and several said a month.
• Many said indefinitely or for lifetime of analyser.
Q.26
• Does your laboratory use six sigma ?
• 8%
• 92%
YES (3% in previous regional audit.)
NO
85 answered question
Q.27
• Does your laboratory hold regular dedicated
IQC meetings ?
• 68%
• 32%
YES
NO
• Very similar results in previous regional audit.
Frequency ranged from daily up to 6 monthly.
Monthly meetings are the most common scenario.
Q.28
• Is IQC a standing agenda item on governance or
management meetings ?
• 81%
• 19%
YES
NO
(72% in previous regional audit.)
• Is IQC discussed at dedicated meetings or general
senior management meetings ?
–
–
–
–
55%
14%
26%
6%
Both
Dedicated meeting only
Senior management meeting only
Neither dedicated meeting or senior management meeting
86 answered question
STANDARD 1
• IQC policies should cover a 24 hour period.
– 25% of labs still show clear evidence of having IQC practices
that vary according to the time of day or whether samples are
being run during core hours versus ‘on-call’ or other shift.
There is an apparent improvement compared to the
previous audit in which 70% had no policy, BUT Remaining labs should develop their policies for 24/7.
Labs should review the validity of IQC testing practice out
of hours if it differs to the routine day.
Discretionary IQC testing out of hours MUST BE
discouraged.
STANDARD 2
• Target values and ranges for IQC material should be
assigned locally for each instrument.
• Manufacturer target levels and ranges should not be used.
• Significant variability remains in the way that labs establish
the target value and SD ranges for IQC materials.
• This is unchanged since the previous audit.
STANDARD 3
• Laboratories should use IQC material from
a third party source.
• The proportion of labs not using third party
QC material remains unchanged since the
previous audit.
STANDARD 4
• Use EP5-A2 or equivalent to determine IQC
values within each individual laboratory.
• There is clear evidence of significant variability
in the way that labs assign target values and
ranges to QC material. Since this is vital to the
ability of a IQC process to detect true error, this
should be addressed.
STANDARD 5
• Encourage method specific rules (single and multi)
based on the required quality of the assay, and known
method bias and coefficient of variation.
• There has been an apparent increase in the number of
labs that utilise method specific rules, but scope for
significant further work.
• Other concerns (see later)
STANDARD 6
• Discourage inappropriate use of single rules, but when
used, ensure they have the required degree of error
detection.
• 55% stated they use single 2SD rules, compared to
previous 76%. This is an apparent improvement, but
again concerns (see later).
STANDARD 7
• Review grades of staff accepting and rejecting IQC.
• The use of trainee and other staff who are not stateregistered to accept and reject QC should be
discouraged or they must be supervised.
• Still approximately 10% of labs are allowing either
trainees or non-registered staff to make key decisions.
This is unchanged since the previous audit.
• A weakness of the audit is that we do not know about the
degree of supervision, but we suggest labs review and
ensure appropriate supervision is in place or change of
practice occurs.
STANDARD 8:
• If IQC policies are robust, and the correct
IQC with appropriate error detection is
used, failed IQC should not be accepted
(e.g. to continue processing patient
samples).
• ONLY 18% of respondents said
continuing analysis would never happen.
• This remains very disappointing. The
reasons quoted are as unconvincing as
they were in the original audit.
STANDARD 9
• For analysers that have multiple modules, each module
should be assessed individually with IQC on each
assessment.
• Not measured in either audit, but principle still proposed.
DISCUSSION POINTS
IQC testing intervals are still very variable.
It is unclear how much relates to lack of confidence in
methodology versus poor IQC planning.
This is reflected in the large variation in the number of
sodium measurements that labs are making between
IQC measurements. Scope for standardisation.
There is a need for our professions to work with
manufacturers of IQC material to refine the target
concentrations used.
DISCUSSION POINTS
Some free text answers imply a lack of understanding of
IQC. For example, some of the labs that stated they use
single 2SD rules are using multi-rules. Most of the labs
that state they use method specific rules are not.
• Would there be any value in the ACB and/or IBMS
designing a questionnaire to evaluate core knowledge
and understanding of IQC within the professions ?
There is lack of clarity on what statistical tests best allow
services operating on multiple sites to compare data
across their whole service see recent paper.
DISCUSSION POINTS
40% of labs still rely on human intervention to prevent
patient results being released when IQC indicates a
method is out of control. This should be minimised in
favour of automated IT systems.
Several labs to need to review the length of time that IQC
data is stored for – RCPath guidance exists.
6% of labs that do not formally discuss IQC at either a
dedicated meeting or management / governance
meetings should review the feasibility of doing this.
COMMENTS FROM WESTGARD
• Analytical failures can be invisible to the laboratory until
they escalate to become serious. Maintaining quality in
the light of workload pressures is essential.
• The UK are doing more for IQC than in the US.
• What the UK have done is very helpful and we would be
keen to ensure labs around the world can learn from it.
• Interested to use their website as a forum to disseminate
and discuss findings in the form of articles and essays.
• They have data from a UK lab and a European lab that
shows introduction of a robust IQC programme can
reduce the number of calibrations by 50% and can
reduce expenditure on IQC material by 50% (reduction in
number of controls, repeat runs and calibrations).
IQC: planning and implementation
strategies. James Westgard.
• Ann Clin Biochem. 2003. 40. 593-611.
• Do not use 2SD control limits.
• Do not use the same control rules for all tests.
• Do select IQC procedures on the basis of the quality
required and precision and accuracy of the method.
• Do minimise false rejections in order to maximise
response to real problems when they occur.
• Do build in the error detection necessary to detect
medically important errors by selection of appropriate
control rules and numbers of control measurements.
Clinical Biochemistry.
Leeds General Infirmary. UK.
• Ann. Clin. Biochem. March 2011. P136-146.
• Jassam N, Lindsay C, Harrison K, Thompson D,
Bosomworth MP, Barth JH.
• The implementation of a system for managing
analytical quality in networked laboratories.
• Defined quality requirements for 71 analytes and
developed software to monitor analytical indices.
• 5 sites, single manufacturer.
• 9 general chemistry, 7 immunoassay analysers.
The way forward
• Adopt a total IQC programme.
• IQC policies should cover 24 hours.
• Establish IQC procedures that minimise
false rejections and maximise the
detection of medically important errors.
• Encourage method specific IQC rules.
• Define control limits based on own data
and reject results which are our of control.
• Preferentially use IQC from a third party.
THANK YOU
Thank you for listening.
A big thank you to the 86 labs who
responded to the survey and to
Westgard for helpful comments.