User verification of
performance
Zahra Khatami/Robert Hill
BHR University Hospitals/Sheffield Teaching
Hospitals
Focus 2013
Aim and objective
• Provide an overview of measurement
uncertainty
• Offer practically achievable procedures
• Recommend the minimum testing
requirements also ensuring reliability
• Work through pre-programmed
spreadsheets
Communication language
“International Vocabulary of Metrology”
(VIM) JCGM 2008
www.bipm.org/utils/en/pdf/JCGM
(BIPM, IEC, IFCC, IUPAC, IUPAP, ISO, OIML,
ILAC)
VIM definition
Verification:
Provision of objective evidence that a given
item fulfils specified requirement
Example: Confirmation that performance
properties or legal requirements of a
measuring system are achieved
Common quandary
Which performance criteria?
What type of material?
How many samples/measurements?
How many days?
What statistical tools?
Performance properties
In the context of clinical chemistry
this equates to the uncertainty
associated with the measurement
VIM definition: Measurement
Uncertainty
Non-negative parameter characterizing
the dispersion (imprecision) of the
quantity values being attributed to a
measurand (an interval of values within
which the true value lies with a stated
probability)
VIM definition: Trueness
Closeness of agreement between the
average of an infinite number of
replicate measured quantity values and
a reference quantity value (in absence
of reference method the difference
between specified methods)
Which performance criteria?
Precision (Imprecision)
Trueness (Bias)
Estimating the measurement uncertainty
Guide to the expression of uncertainty measurement
(GUM)
Bottom–up approach which derives the
uncertainty of a measurement result by
combining the uncertainties related to
the uncertainty sources of the
measurement process
Estimating the measurement uncertainty
Guide to the expression of uncertainty measurement
(GUM)
Alternatively
Top-down approach which uses the
reproducibility as uncertainty estimate
Reference publications
EN/ISO 15189 (Requirements for
quality & competence specific to the
quality management system
requirements particular to medical
Laboratories)
CLSI (Clinical Laboratory Standards
Institute) Former NCCLS
IFCC
ECCLS (European Council for Clinical
Laboratory Standards) obsolete since
mid 90’s
Reference publications
CLSI EP15-A2 (User verification of
performance for Precision and Trueness
2005)
CLSI EP9-A2 (Method Comparison and
Bias Estimation Using Patient Samples
2002)
Reference publications
• Thorough document
• Lengthy document
• Large number of tests
• Detailed statistical processing
• Lack of accessible pre-programmed
spreadsheets
Authors
Zahra Khatami
Robert Hill
Catherine Sturgeon
Edward Kearney
Peter Breadon
Anders Kallner
Aim
• Provide practical recommendations
• Minimum requirements/ensuring
•
•
•
•
reliability
User friendly
Full pre-programmed spreadsheets
Free access for all
Regular review
Measurement verification in the
clinical Laboratory:
A guide to assessing analytical performance
during the acceptance testing of methods
(quantitative examination procedures) and
/or analysers
http://www.acb.org.uk/An%20Ver/Measurement1.asp
Assumptions
Adequate verification by manufacturer,
accessible by the user (IVD directive)
Established performance specifications
(precision, bias, linearity, measurement
interval, interference, etc)
Imprecision
• What type of material?
• Patient Pool or IQC material
• How many measurements?
• Five replicates at two concentrations
• How many days?
• Five days
The statistical test
ANOVA
Tests for a difference between the means
of a number of groups of observations
It partitions the total variance into the
components: variation between the
within and between each series of
measurements
It offers an estimate of within and
between series variation
Refer to Calcium data word file
Bias: Patient comparison
• How many measurements?
20 patient samples
• How many days?
Paired samples should be measured within a
close time interval
The statistical tests
T-test
Ordinary linear regression
Deming regression
Difference plot
Refer to amylase word file
Bias: using reference material
• What type of material?
Reference material with assigned value
(CRM)
• How many measurements?
Two concentrations, duplicate measurements
• How many days?
3-5 occasions/days
• What statistical tool?
T-test
Bias: using EQA material
• What type of material?
Material with peer group values from EQA
organizers
• How many measurements?
7-10 concentrations, duplicate measurements
• How many days?
N/A
• The statistical tool?
T-test
Group
mean
6
7
12
47
95
127.5
160
180
C omponent: T E S T
P eer group data
Us er D ata
C alc ulated
N:o of
P eer
S tand
R es ult R es ult
C V% obs in
Mean group Z-s c ore
dev
1
2
group
SEM
2 16.67
9
2
5
3.5
0.67
-1.25
1
6.5
3.5
5
-2.00
10
15
12
13.5
1.25
2
42.5
47.5
45
-1.00
5.26
5
100
90
95
2.23
0.00
6
105
95
100
-4.58
150
145
147.5
7
180
170
175
-0.71
1
35
10
175
Abs
R el
deviadevia
tion
tion
-2.50 -41.67
-2.00 -28.57
1.50 12.50
-2.00
-4.26
0.00
0.00
-27.50 -21.57
-12.50
-7.81
-5.00
-2.78
170
172.5
MeanZ :
-1.19
Mean peer goup:
79.3
S tandard deviationZ :
1.82
S E MP eer
25.1
S E MZ
0.69
Mean us er res ults :
73.1
k=tcrit:
2.447
S E MUser
23.5
tdep :
-1.726
D iff:
6.3
df:
6
tcrit:
2.45
α %:
5
p-value(two-s ided):
0.135
Bias not different from zero
tdep :
α %:
p-value(two-s ided):
1.847
5
0.107
1.6803
Thank you
Everything should be made as simple as possible,
but no simpler