Module
PANEL TESTING
High level
1
Content Overview
•
•
•
•
•
•
•
•
•
•
What is panel testing?
What is panel testing used for?
PT advantages and disadvantages
Preparation of test smears
Validation of panel batches
Panels’ composition
Organization of a panel testing round
Analysis of results; scoring system
Interpretation and feedback
Forms
2
What is Panel Testing?
•
One of EQA methods
•
System of sending stained
and/or unstained smears
from NRL to peripheral
laboratories to check
proficiency in performing
AFB smear microscopy and
reporting AFB results
•
Tests individual
performance of a laboratory
worker, not the laboratory
overall
3
What is Panel Testing Used For?
•
Rapid assessment of performance of a laboratory
staff to prioritize training and supervisory activities
•
Quick detection of problems associated with very
poor performance
•
Evaluation of competency of laboratory technicians
prior to and following training
4
What is Panel Testing Used For? (2)
• A minimal step for EQA with limited resources
• Monitoring performance of individuals in
absence of a rechecking program
• Supplements rechecking programs
• Investigation of excessive errors found in rechecking
5
Advantages of Panel Testing
• Low workload for a peripheral center
• Improves laboratory credibility
• Rapid response countrywide possible
• Possible identification of gross deficiencies
• Use of stained and unstained smears can
help identify source of problem
6
Disadvantages of Panel Testing
• Technicians know they are being evaluated
• Does not measure routine performance
• High workload for NRL
•
need for additional resources - appropriate equipment, highly
qualified staff to produce panels
•
a system for panel sets distribution, data collection, analysis
and feedback
• May not be motivating to improve daily
performance
7
Preparation of Test Smears
• Routine patient smears:
• Problems with consistency
• Only stained smears available
• Smears of the required AFB quantification may not
be easily available in the needed quantity
• Specially manufactured smears:
• Can be stained or unstained
• Provide uniformity of the test for technicians
• Provide known quantification of AFB required
8
Manufacturing Smears for PT
•
Should be done at NRL
• requires a safety hood, centrifuge,
vortex, water bath, lab supplies
(pipettes, tubes, slides, boxes etc.)
•
•
•
• BIOSAFETY MEASURES!
Smears are prepared from
known positive and negative
sputa
Reference for the manufacturing
procedures: AFB Smear
Microscopy EQA Guidelines
Requires time practice and
expertise
9
Validation of Panel Batches
• Mandatory requirement!
• Pre-validation:
• Validation of consistency of panel batches prior to
sending test panels out to periphery
• Post-validation:
• Validation of panel slides / batches after receiving
aggregate results from all laboratories
• Keep accurate records of batches prepared
and detailed results of the validation process
10
Pre-validation
• Stain at least 6 slides from each batch to be
•
•
•
examined independently by 3 or more
technicians
Calculate the average results and standard
deviation (SD)
The average minus 2 SD should be > 0 to
accept the batch
Use the Validation Log to record results
11
Normal Distribution
•
•
symmetrically
distributed around
the mean
Characteristic “bellshaped” curve
Assumed for all
quality control
statistics
Frequency
• All values are
x Variable
12
What is Standard Deviation?
The principle calculation used in the
laboratory to measure dispersion of a group
of values around a mean
Standard Deviation – Statistical Formula
SD 
 ( Xi  X )
2
n 1
13
Standard Deviation and Probability
•
•
For a set of data with a normal
distribution, a value will fall
within a range of:
• +/- 1 SD 68.2% of the time
• +/- 2 SD 95.5% of the time
• +/- 3 SD 99.7% of the time
Laboratories use the +/- 2 SD
criteria for the limits of the
acceptable range for a control
value
When the QC measurement
falls within that range, there is
95.5% confidence that the
measurement is correct
X
Frequency
•
68.2%
95.5%
99.7%
-3s-
2s
-1s
Mean
+1s
+2s +3s
14
Sample Form: Validation Log
for AFB Panel Testing Slide Batches
(pre-validation)
Batch №
VALIDATION LOG FOR AFB PANEL TESTING SLIDE BATCHES
Slide evaluation
Slide Preparation
Number
Date slides
of slides
made
made
Slide test results (AFB per 100 fields)
1
2
3
4
5
Standard
deviation (SD)
6
Average
Consistency
(average minus
2 standard
deviations)
ACCEPT or
REJECT?
Report result
1
7/6/2006
17
10
10
50
15
10
11
17.7
15.96
-14.2
reject
1+
2
9/6/2006
34
0
0
0
0
0
0
0.0
N/A
N/A
accept
negative
3
9/6/2006
40
7
2
4
3
9
2
4.5
2.88
-1.3
reject
scanty
4
10/6/2006
23
5
6
2
9
12
9
7.2
3.54
0.1
accept
scanty
5
15/6/2006
17
30
27
28
36
43
50
35.7
9.22
17.2
accept
1+
6
16/6/2006
30
3500
3700
1500
1700
2600
2900
2650.0
907.19
835.6
accept
3+
7
8
9
10
• Intended positives should never be negative
• Intended negative smear should never be positive
• Quantification differences should not reach 2 steps on scale
15
Post-validation
•
The same smear error reported by a majority of
technicians may represent a problem with the panel
slide / batch:
•
•
•
•
Technical difficulties in preparing panel slides
Error in the pre-validation
Incorrect recording of the expected result
Fading of smears during transportation to
peripheral sites
16
Logbook of Panel Slides Sets /
Post-validation
Form PT2: PANEL SETS' LOGBOOK / AGGREGATE RESULTS
(Record of a set of 10 slides selected from Form PT1)
Central Laboratory administering panel test:
District where panel testing is conducted:
Date slide set(s) sent to peripheral laboratories:
Slide set(s)' number(s):
Slide Sets Numbers
Slide
No
Batch No
Stained or
unstained
Expected
result (from
Form PT1)
Comments
Peripheral Laboratories' Results
1
2
3
4
5
6
7
8
9
10
1
2
3
4
5
6
7
8
9
10
17
Supervisor:
Coding of Panel Smears
•
•
Ensure that result can not be guessed by an
examinee – to avoid reading bias
Make identification of a panel smear clear to a
supervisor in charge of a panel testing exercise
Example of a smear’s code:
a panel set
number
62-45-1
a smear serial
number in a panel
a batch number
18
Panels’ Composition
•
The composition of a panel set is determined by NRL
•
Number and types of slides to reassure that correct
or incorrect results are not accidental
• At least 10 slides provides a valid and fair test
• Batch of stained and unstained smears
•
Unstained smears:
• Evaluate staining technique; provide information about
stain preparation and quality
19
Examples of Panel Sets Compositions
A panel test should represent a challenge in
terms of difficulty:
- some scanty and low-positive smears
1 slide graded 3+
1 Slide graded 2+
1 slide graded 1+
2 slides graded
1-9 / 100 fields
5 negative slides
1 slide graded 3+
1 slide graded 2+
2 slides graded 1+
3 slides graded
1-9 / 100 fields
3 negative slides
1 slide graded 2-3 +
2 slides graded 1+
3 slides graded
1-9 / 100 fields
4 negative slides
20
Getting Started: Issues to Consider
•System for sending slides
•Frequency of testing
•Forms to record and report results
•Time allowed for technicians to complete PT
•Availability of microscopes
•Performance criteria
•Feedback and corrective action if needed
•Mechanism to resolve discrepant results
21
Implementation of Panel Testing
•
•
•
•
•
Responsibility of the NRL- from preparation of slides to
analysis of results and feedback
Determine the number of AFB technicians who will
participate in PT (ensure preparation of the needed
number of panels)
Communicate with Public Health Directors regarding
EQA activities
Prepare the schedule for panel testing in each location
Collaborate with intermediate laboratories
22
Sending Slides
•
Delivery system based on services, regulations,
resources available:
• mail/post
• courier
• supervisory visit
•
Turnaround time
•
Safe package to prevent breakage of slides:
• strong plastic slide holders
23
National reference laboratory
A POSSIBLE SCHEME OF A
PANEL TESTING ROUND
Intermediate laboratory
Peripheral laboratories
Peripheral laboratories
24
Performing a Panel Test Round
•
Frequency: at least one to two times a year
•
A standardized PT reporting form / an accompanying letter to
provide instructions
•
Individual, not group work
•
No incentives or punitive actions as a result of the PT exercise
•
Time allowed to complete the PT exercise,
maximum:
• 2 hours for a stained slide set
• 3 hours for an unstained slide set
25
Individual Results of Panel
Testing / Feedback
Form
Form PT3: PANEL TESTING INDIVIDUAL RESULTS AND FEEDBACK FORM
Central Laboratory Use Only:
Test slide set No:
Passsing score:
Date of sending set:
Date results received:
Peripheral Laboratory:
District:
Date PT conducted:
Name of technician reading test smears:
Note: If more than one technician performs AFB microscopy in the laboratory, each technician should read all 10 smears and
record their results on a separate form. Technicians should not discuss results or share forms until all results have been sent
back to the central laboratory. Forms for all technicians should be sent to the central laboratory for evaluation.
Central Laboratory Only
Slide Number
Result
Expected result
Error Type
Points
Feedback:
Total Points:
HFP:
Pass / Fail:
HFN:
LFP:
LFN:
QE:
Recommended Action:
Supervisor:
26
Management of PT During a Supervisory Visit
•
•
Administration of PT during on-site visits:
•
•
•
•
•
can be effective in some circumstances
provides direct observation of work under PT exercise
corrective action may be easily facilitated
BUT: may be impractical in routine conditions
can be done in a special survey
Important: PT must not disrupt routine patients’
examinations, therefore consider:
• Careful planning of a supervisory visit
• Allocating sufficient time for a visit
27
Analysis of PT Results
•
A scoring system is to be developed prior to test
•
Distinguish major and minor errors
• false positive/negative related to 1+, 2+ or 3+ errors are
major errors
• quantification errors (at least a 2 grade difference) and
false positive / negative errors in the scanty group (1-9
AFB) are considered minor
•
Determine successful score
•
Determine plan of action for poor performances
28
Types and Classification of Errors
Result of
Technician
Negative
1-9 AFB/100 f
1+
2+
3+
Negative
correct
LFP
HFP
HFP
HFP
Correct:
QE
LFN
LFP
HFN
HFP
Result of Controller
1-9 AFB/100 f 1+
2+
LFN
HFN
HFN
correct
correct
QE
correct
correct
correct
QE
correct
correct
QE
QE
correct
No errors
Quantification error
Low False Negative
Low False Positive
High False Negative
High False Positive
3+
HFN
QE
QE
correct
correct
Minor error
Minor error
Minor error
Major error
Major error
29
Example of PT Scoring
•
Set of 10 slides, each slide is worth 10 points, total
possible score = 100
• HFP and HFN scores 0
• LFP, LFN and QE scores 5
(QE = 2 grades difference)
• Passing score = 80 – 90
30
Analysis of PT Results
•
•
Study the aggregate results from all laboratories
Post-validate panel slides/batches
•
If a majority of technicians fail to report correct
results for the same slide/batch it may represent a
problem with panel slide preparation:
• Assure that poor performance is not due to panel slide
problems
→ exclude this slide from scoring
→ check returned discrepant slides
→ detect problems in preparation of panel smears
→ undertake measures to improve the quality of panel
smears preparation
31
PT– Interpretation of Results
• False positive and negative errors should be
considered separately
• False positives - lack of proficiency / faulty
microscope
• False negatives - poor stain / inadequate
examination time / poor microscope
32
Feedback to Laboratories on PT results
•
Timely and confidential
•
Individual and aggregate test results
•
Criteria for acceptable performance
•
Reports to TB program coordinator should provide
appropriate background information and
recommendations and not simply scores
•
Poor performance often requires a visit to laboratory
33
PT Aggregate Results of Multiple Laboratories
Form PT4: PANEL TESTING REPORT OF MULTIPLE LABORATORIES FOR DISTRICT SUPERVISOR AND NTP
District:
District Supervisor:
Supervising Laboratory:
Period PT conducted:
Panel test slide set(s):
Passing score:
Peripheral Lab Annual volume
SPR, %
Technician(s)
participated in PT
PT score
HFP
HFN
LFP
LFN
QE
Total errors
District
Averages
SPR: slide positivity rate; PT - panel testing; Annual volume - annual volume of smear examinations; HFP - high false positives; HFN - high false negatives; LFP - low false positives;
LFN - low false negatives; QE - quantification errors
Report submitted by:
Date:
34
PT Aggregate Results Report: Example
35
Conclusion:
Laboratories submitting unacceptable PT
results with documented consistency and
quality of PT slides experience serious
problems with AFB microscopy.
Additional resources should be obtained for
supervisory visits, correction of problems
identified, including replacement of faulty
microscopes (and/or stains), retraining if
needed, and follow-up panel testing.
36
Key Messages (I):
•
•
•
PT is an effective method when it is necessary to
quickly obtain information about capabilities of
individual laboratory technicians to read smears and
report results according to standards approved by
NTP.
PT is considered to be less effective than rechecking
because it does not monitor routine performance; BUT
PT can be more effective than rechecking in the areas
where prevalence of positives is low.
37
Key Messages (II):
•
The main prerequisite to start a panel testing program
in a country is availability of a laboratory with a highly
qualified staff capable to safely produce panel sets of
appropriate quality and required composition.
•
Validation of panel batches (pre- and post-validation) is
the mandatory requirement for PT.
•
A well functioning system should be established to
distribute panels, collect and analyze data; provide
timely feedback to peripheral laboratories.
38