Center for Biofilm Engineering
Statistically assessing
limits of detection and
performance standards
Albert Parker
Biostatistician and Research Engineer
Montana State University
July 2010
Standardized Biofilm Methods Laboratory
Diane Walker
Paul Sturman
Lindsey Lorenz
Marty Hamilton
Kelli Buckingham-Meyer Darla Goeres
Statistical thinking
 Limit of Detection
 Dealing with counts below the detection limit
 Performance Standards
Single tube test method
1. inoculate
Coupon w/
biofilm
2. expose
& neutralize
3. enumerate
Rinse
Sonicate,
vortex
Dilute, plate
10-fold Dilution series of concentrated cells
Treated
coupon
1 mL
Concentrated
Cells
Dilution: 0
1 mL
1 mL
1 mL
1 mL
1 mL
9ml buffer
9ml buffer
9ml buffer
9ml buffer
9ml buffer
9ml buffer
Dilution:1
Dilution: 2
Dilution: 3
Dilution: 4
Dilution: 5
Dilution: 6
Counting cfu’s
EPA 2007
Dilution
0
1
2
3
4
5
6
Counts
1
2
NP
NP
NP
NP
NP
NP
TNTC TNTC
126
181
16
17
2
1
If 40ml is the original volume and 0.1ml is the volume plated,
then using the counts of 126 and 181 at the 4th dilution,
it is estimated that there were
average(126,181) x 40/0.1 x 104 = 6.14 x 108
biofilm bacteria on the carrier.
What to do when there is nothing to count?
Dilution
0
1
2
3
4
5
6
Counts
1
2
NP
NP
NP
NP
NP
NP
0
0
0
0
0
0
0
0
Limit of Detection in Chemistry
In chemistry, Currie (1968 and 1995) used the term
detection limit to refer to a true concentration that has
a small probability of generating measured values
smaller than some critical value (such as 0).
Excerpt from the Technical Support Document for the Assessment of Detection and
Quantitation Approaches, Engineering and Analysis Division, Office of Science and
Technology, EPA, 2003.
Limit of Detection in Chemistry
X
= measured concentration
LOD = true concentration
p(X<=0 | LOD = 3.29) <= .05
Analogous Limit of Detection in Microbiology
X
= cfu’s at dilution d
LOD = true number of bacteria
in the original beaker
p(X = 0 | Count = 3) <= .05
LOD = Count x 40/.1 x 10d
LOD = 3 x 40/.1 x 10d
Julia Sharp and Marty Hamilton,
Detection Limits in Microbiology,
MSU masters project, 2001.
Conventional “Limit of Detection” in Microbiology
X
= cfu’s at dilution d
“LOD” = true number of bacteria
in the original beaker
p(X = 0 | Count = 1) <= .37
“LOD” = Count x 40/.1 x 10d
“LOD” = 1 x 40/.1 x 10d
Counts below the detection limit in Microbiology
When all zero cfu’s are observed for all
dilutions plated in this example …
This happens 5% of the time when the
number of bacteria in the original
volume is
LOD = 3 x 40/0.1 x 103
= 1.2 x 106
This happens 37% of the time when
the number of bacteria in the original
volume is
“LOD” = 1 x 40/0.1 x 103
= 4 x 105
Dilution
0
1
2
3
4
5
6
Counts
1
2
NP
NP
NP
NP
NP
NP
0
0
0
0
0
0
0
0
Counts below the detection limit in Microbiology
What to do?
 Substitute a 1
 Substitute a 1/2
Dilution
0
1
2
3
4
5
6
Counts
1
2
NP
NP
NP
NP
NP
NP
1
0
0
0
0
0
0
0
Dilution
0
1
2
3
4
5
6
Counts
1
2
NP
NP
NP
NP
NP
NP
.5
0
0
0
0
0
0
0
Dilution
0
1
2
3
4
5
6
Counts
1
2
NP
NP
NP
NP
NP
NP
0
0
0
0
0
0
0
0
2.0 x 105
1.0 x 105
Values below the detection limit in Chemistry
When values are observed below the LOD:
 Substitute a small number such as LOD/2 or LOD
(analogous to what is done in microbiology)
 Use a “robust method” which ignores these values
 Random substitution
 Bayesian methods
Values below the detection limit in Microbiology
There is no single solution for all scenarios!
The appropriate approach depends on:
 The percentage of values below the LOD
The EPA recommends substitution rules if <15% of the
values are below the limit of detection.
- Guidance for Data Quality Assessment, Office of Research and Development, EPA, 1998.
- Singh and Nocerino, Robust Estimation of Mean and Variance Using Environmental Data
Sets with Below Detection Limit Observations, EPA, 2001.
 The goal of the study!
- Mean estimation?
- Variance estimation?
Values below the detection limit in Microbiology
 Substitute a small number such as 1/2 or 1
- Introduces bias into biofilm density estimates
- Biases variability
 Use a “robust method” which ignores these values
- Biases variability
 Random substitution
- Non-unique biofilm density estimate
 Bayesian methods
- Area of research at the CBE
Performance Standards for a Quantitative Disinfectant Test
Performance standards are moving to quantitative tests.
Understanding how the limit of detection affects quantitative
measures is important.
 Currently, performance standards for a disinfectant are
“semi-quantitative” – e.g., the use dilution method.
 Future performance standards may require that a mean
quantitative efficacy measure, such as the log reduction,
exceeds a target value with confidence.
 To meet a target value with confidence, disinfectant mean
efficacy and variability must both be estimated.
Example of a Performance Standard for the LR
Controls
Dilution
0
1
2
3
4
5
6
Counts
1
2
NP NP
NP NP
NP NP
NP NP
TNTC TNTC
40
42
2
2
0
1
2
3
4
5
6
NP
NP
NP
NP
TNTC
36
2
NP
NP
NP
NP
TNTC
48
4
0
1
2
3
4
5
6
NP
NP
NP
NP
TNTC
51
2
NP
NP
NP
NP
TNTC
37
5
Treated
Log Reduction
8.36
Dilution
0
1
2
3
4
5
6
Counts
1
2
3
1
0
0
0
0
0
0
0
0
0
0
0
0
2.02
8.38
0
1
2
3
4
5
6
TNTC TNTC
56 52
11
8
0
2
0
0
0
0
0
0
4.53
8.40
0
1
2
3
4
5
6
TNTC TNTC
142 148
13 16
4
1
0
3
0
0
0
0
4.93
log
mean 8.38
LD
LR
= 8.38 – 3.82
= 4.55
SELR = .9111
mean 3.82
CI95% = (1.89, 7.21)
The LR is not
significantly larger
than the
target value = 2
Non-detect effect on Performance Standards for the LR
Controls
Dilution
0
1
2
3
4
5
6
Counts
1
2
NP NP
NP NP
NP NP
NP NP
TNTC TNTC
40
42
2
2
0
1
2
3
4
5
6
NP
NP
NP
NP
TNTC
36
2
NP
NP
NP
NP
TNTC
48
4
0
1
2
3
4
5
6
NP
NP
NP
NP
TNTC
51
2
NP
NP
NP
NP
TNTC
37
5
Treated
Log Reduction
Counts
1
2
NP NP
NP NP
0.5
0
0
0
0
0
0
0
0
0
8.36
Dilution
0
1
2
3
4
5
6
8.38
0
1
2
3
4
5
6
NP
NP
11
0
0
0
0
NP
NP
8
2
0
0
0
8.40
0
1
2
3
4
5
6
NP
NP
13
4
0
0
0
NP
NP
16
1
3
0
0
log
mean 8.38
LD
LR
= 8.38 – 4.28
= 4.10
SELR = .5865
3.12
4.74
4.99
mean 4.28
CI95% = (2.38, 5.81)
The LR is smaller
SELR is smaller
CI95% is more narrow
The LR is
significantly larger
than the
target value = 2
Performance Standards for a Quantitative Disinfectant Test
 Confidence intervals to assess performance are affected by
substitution rules for values below the limit of detection.
 Instead of confidence intervals, requirements could be placed
on the LR and on SELR. For example, if the
- target value is 2
- then the requirements
LR > 3.5
SELR < 0.5
would alleviate effects of values below the limit of detection
on the test outcome.
Summary
 In other fields of science, the limit of detection is a statement
based on a probability. For microbiology, this statement is:
LOD = 3 x (scale-up factor) x 10d
 Substitution rules affect biofilm density and variability
estimation. Appropriate action on values below the limit of
detection depend on the goals of the study.
 Performance standards unaffected by values below the limit of
detection are desirable.
limit of detection