EQA Overview Module
OVERVIEW OF EXTERNAL QUALITY ASSESSMENT
Purpose:
To provide participants with an overview of the basics of external
quality assessment (EQA)
Learning
objectives:
At the end of this module, participants will be able to:
 Define EQA
 Explain how EQA is an indispensable part of a laboratory
quality management system
 Name three methods of EQA that are used for AFB sputum
smear microscopy.
 Describe the indications for use, and the advantages and
disadvantages of each method
Content Outline:




Handout and
exercises:
Appendix:
What is EQA?
EQA for AFB Smear Microscopy
o Panel testing
o Blinded rechecking
o On-site evaluation
EQA method: advantages and disadvantages, indications for
use
Identification of the most feasible EQA method for a country
None
None
WHAT IS EXTERNAL QUALITY ASSESSMENT (EQA)?
External Quality Assessment (EQA) can be defined as a system for objectively checking
the laboratory’s performance using an external agency or facility. EQA is an
indispensable part of a laboratory quality management system as it ensures customers
(physicians, patients and health authorities) that the laboratory can produce reliable
results. An independent external quality assessment of laboratory performance by a
recognized scheme is the requirement of the international standard ISO 15189:20031,
which states that “The laboratory should participate in interlaboratory comparisons such
as those organized by external quality assessment schemes. Laboratory management shall
monitor the results of external quality assessment and participate in the implementation
of corrective actions when control criteria are not fulfilled”.
1
ISO 15189:2003, Medical laboratories – Particular requirements for quality and competence.
1
EQA Overview Module
EQA strongly contributes to the continuous quality improvement of laboratory
performance. Participating laboratories benefit because external quality assessment:
 serves as an early warning system,
 identifies systematic problems,
 provides objective evidence of laboratory quality,
 serves as an indicator for focusing improvement efforts,
 identifies training needs,
 provides a source for continuing education,
 provides a source of materials for practice.
EXTERNAL QUALITY ASSESSMENT FOR AFB SMEAR MICROSCOPY
The EQA consensus document (“External Quality Assessment for AFB Smear
Microscopy” guidelines) defines EQA for AFB smear microscopy as a process which
allows participating laboratories to assess their capabilities and performances by
comparing their results with those of other laboratories in the network. EQA focuses on
the identification of laboratories where there may be serious problems resulting in poor
performance. EQA for AFB smear microscopy consists of three methods that can and
should be combined to evaluate laboratory performance. These three methods are:
 Panel testing
 Blinded rechecking
 On-site supervision
PANEL TESTING
This is the process of sending stained and/or unstained smears from the NRL to the
peripheral sites to check proficiency of laboratory technicians in reading smears and
reporting results.
In a panel testing program, the NRL sends out a set of sputum smears for testing.
Laboratories then process and analyze smears, return results to the NRL which evaluates
their results, and provides feedback to participating laboratories. Panel testing for AFB
sputum smear microscopy is strictly an individual task for a laboratory technician; it
evaluates the individual performance of a laboratory worker. NTP managers and
supervisors use panel testing results to plan and undertake appropriate corrective actions,
if needed, to improve performance.
Panel testing may be useful:
 in a test situation such as at the end of a training course,
 to perform a quick and relatively easy survey of the network to gain information
on training needs, and the condition of microscopes and stains. The latter requires
that stained as well as unstained smears are included in the panel sets and that the
slides are returned with the results to make investigation into the source of errors
possible.
2
EQA Overview Module
However, the usefulness of panel testing is limited, since it presents a mock (simulated)
situation, and it cannot check technician’s performance for all steps, such as smearing.
The test gives an indication of capability rather than of true performance, and it provides
little motivation to perform better in daily work. Technicians may occasionally score
extremely well in this test because they know their work is being evaluated, but their
daily work may remain unsatisfactory because of carelessness. Although panel testing
cannot control everything, it should be organized periodically, especially when an EQA
rechecking program is not feasible, for instance in case of insufficient resources. Once
rechecking is being performed efficiently, routine panel testing may be unnecessary. In
areas with low incidence of tuberculosis and low positivity rates in laboratories, a
rechecking program may not be feasible, thus making panel testing a preferred EQA
method.
BLINDED RECHECKING
Rechecking is another EQA method to check the laboratory’s performance. This method
is a systematic sampling of routine smears from the peripheral laboratory for rechecking
by a higher-level laboratory. For this method to be effective, there are a few important
considerations.
Rechecking must be blinded, which means that the controller (i.e. a supervisory
laboratory or a technician responsible for rechecking slides) does not know the results
from the peripheral laboratory. The blinded rechecking method should be based the Lot
Quality Assurance System (LQAS). LQAS is a method to determine an optimum sample
size of slides to be rechecked which, when applied properly, yields statistically
acceptable samples to assess quality of work. The former sampling scheme of rechecking
10% of negatives and 100% of positives is no longer recommended for AFB sputum
smear microscopy. Other considerations when applying the blinded rechecking system
include proper storage of routine smears in peripheral laboratories, random sampling of
smears, and establishment of procedures to resolve discrepancies.
This system is considered to be the best method for evaluating performance because it
reveals the reality of the daily performance of the laboratory. Experience shows that
blinded rechecking can be highly motivating for peripheral staff, and can be seen as a
method of continuous education. Properly executed, rechecking is capable of gradually
bringing the quality of the network to a high level.
However, this method is the most resource intense and the most expensive to implement,
and constitutes a great deal of work for intermediate and central level. Although the
concept of rereading smears to check performance may seem simple, expertise is needed
to calculate the proper sample size, set up a system for blinding samples and resolving
discordant results, and implementing an equitable program for different labs. Blinded
rechecking cannot be used when almost no positives are detected. Therefore, in areas
where the smear positivity rates in the laboratory registers are low, around 5%, stratified
sampling focusing on follow-up smears may be more appropriate.
3
EQA Overview Module
ON-SITE SUPERVISION
The systemic process of making periodic visits to the laboratory to assess laboratory
performance in order to obtain a realistic picture of laboratory practices and assist with
problem solving is called on-site supervision. This EQA method is especially helpful to
resolve problem areas that have been identified by other EQA schemes (blinded
rechecking and/or panel testing), and allows for provision of training. On-site supervision
includes regular visits by TB supervisors as well as visits by a laboratory supervisor from
a higher-level laboratory. The types and frequency of field visits include:
– a monthly/quarterly visit by a TB supervisor (TB control DOTS strategy
requirement), and
– a comprehensive evaluation by a laboratory specialist from a higher level
laboratory (intermediate or NRL) routinely at least annually, and more
frequently if very poor performance if identified based on results of panel
testing or blinded rechecking.
In order to ensure that on-site evaluations are carried out in a consistent and structured
way, the NTP should develop standard check-lists to assist both laboratory and TB
supervisors during their field visits. Planning should also consider the need for follow-up
visits to monitor corrective actions and provide additional training if needed. Although
conducting on-site supervision visits can be costly and time-consuming, they are an
effective and efficient use of time and resources if personnel are trained, motivated and if
on-site supervision applies very targeted problem-oriented supervision to resolve
problems identified from other EQA schemes, particularly rechecking.
EQA METHODS: ADVANTAGES AND DISADVANTAGES, INDICATIONS FOR USE
Each EQA method has distinct advantages and disadvantages. This information is
summarized in the Table 1 below. Effective use of each of these methods will depend on
the availability of resources, goals set up by the NTP, TB prevalence,
centralization/decentralization of health services and the stage of laboratory quality
assurance activities development in a country. An important step in any process of
detecting performance problems is the application of appropriate problem solving
strategies. Many factors may contribute to poor performance, and training cannot be
considered the universal solution. Therefore, resources to implement quality
improvement are a critical consideration when designing a step-wise approach to EQA.
Resources will also be necessary for ongoing performance assessment to evaluate the
success of problem solving strategies.
4
EQA Overview Module
TABLE 1. COMPARISON OF EQA METHODS WITH REGARD TO THEIR ADVANTAGES, DISADVANTAGES AND APPLICABILITY
Method
Advantages
Disadvantages
Uses
Panel testing
•
•
•
•
•
•
•
Blinded
rechecking
•
•
•
On-site
supervision
•
•
•
•
•
Low workload for a peripheral
center
Improves laboratory credibility
Rapid response countrywide
possible
Use of stained and unstained
smears can help identify the
source of a problem
May lead to identification of
faulty equipment
Low workload for a peripheral
laboratory
Motivates to improve daily
performance
Reflects reality of routine
performance
Direct personal contact
Motivating to staff
Observation of actual work
Identifies causes of errors
Permits verification of
equipment quality and function
•
•
Does not measure routine
performance
High workload for NRL
May not be motivating to
improve daily performance
•
•
•
•
•
•
•
•
•
•
•
•
Higher workload for a higher
level center
Needs close adherence to
elaborated procedures
Can not be used with very low
positivity rates
Usually poor coverage (not
provided for all peripheral labs)
Labor intensive
Costly
Ineffective unless done by a
well experienced / educated
personnel
Needs very good supervisors
•
•
•
•
•
•
•
Minimal first step for EQA with
limited resources
Rapid assessment of gross
deficiencies
Evaluates proficiency of laboratory
technicians prior to and following
training
A tool during problem-oriented
supervision
Countrywide
Standard for monitoring laboratory
performance
Ongoing and permanent
Complementary to rechecking and
panel testing for constructive
feedback and problem solving
Implementation and monitoring of
quality improvement measures
Data collection and flow of
information among laboratory levels
•
5
EQA Overview Module
IDENTIFICATION OF THE MOST FEASIBLE EQA METHOD FOR A COUNTRY
Which EQA method(s) should be recommended for a country? To answer this question
and make a proper decision, the NTP managers and NRL staff should collect and analyze
comprehensive data on TB prevalence, AFB smear positivity rates, health laboratory
services infrastructure (centralized/decentralized), as well as the status and effectiveness
of existing laboratory quality assurance activities, taking into consideration goals and
objectives determined by the NTP and the availability of resources. The analysis of
situations in three hypothetical countries (Country A, Country B and Country C) can help
illustrate the decision-making process regarding selection of the EQA method(s) for a
specific country. The brief background information per each country is summarized
below:
Country A
 Populous country; vast
territory
 High burden of TB
 Several hundreds
peripheral laboratories,
geographically
dispersed
 Intermediate
laboratories’
infrastructure is not well
developed; many
intermediate labs lack
sufficient qualified staff
 NRL is well equipped
and staffed
Country B
 Less populated
(compared to Country
A); compact territory
 High burden of TB
 Smear microscopy
laboratory network: one
microscopy center for
100,000 population
 High to medium
positivity rates
 Well developed
structure of intermediate
laboratories, supervised
by NRL
 Lack/rotation of staff is
a typical problem for
many laboratories
Country C
 Populous country
 Low burden of TB
 Only few laboratories
dealing with sputum
smear microscopy
 Low volume sputum
smear examinations;
very low positivity rates
 AFB smear-positive
specimens are
rechecked at NRL
In selecting options for an appropriate strategy for developing EQA activities in Country
A, consider:
1. The large numbers of decentralized peripheral laboratories and, in particular,
poorly developed infrastructure of intermediate laboratories makes
implementation of all EQA methods difficult.
a. Panel testing will result in high workload for NRL (preparation and
distribution of uniform panels, review of results and preparation of
feedback reports).
b. Blinded rechecking is impossible without a properly established and
functioning intermediate laboratory network. It will be difficult to assign
the required number of controllers to ensure an even distribution of the
rechecking workload and, consequently, appropriate quality of controllers’
work.
6
EQA Overview Module
c. On-site supervision annually will be difficult as the NRL staff will not be
able to visit all peripheral laboratories alone.
2. A thorough and critical assessment of all strengths and weaknesses of the
situation should be recommended; including collection of data on smear positivity
rates.
3. In this situation it will be too difficult to implement the blinded rechecking
program. Instead, it can be recommended to start with a panel testing program.
Every effort should be made to develop intermediate level laboratories for the
eventual gradual implementation of the blinded rechecking program.
4. Information on AFB smear positivity rates will help determine the sampling
approach for the blinded rechecking program. In areas with medium positivity
rates (around 5%), the stratified sampling would be applicable.
In Country B, where the situation is characterized by a relatively well developed TB
laboratory network, consider these options.
1. Panel testing, especially if there is only sketchy information available on
laboratory proficiency, and there is need for a rapid assessment in order to
prioritize training and supervisory activities.
2. Gradual implementation of the blinded rechecking program. For medium
prevalence situations where using a non-stratified sample would lead to the
controllers’ overload, the stratified sampling is a possible solution.
3. With continued program development and intermediate laboratories
strengthening, the use of panel testing may be limited to evaluation of proficiency
of laboratory technicians prior to and following training while blinded rechecking
system is further developed and expanded countrywide.
The analysis of Country C situation would result in the following conclusions:
1. Blinded rechecking would not be efficient due to the low volume of TB sputum
smear microscopy examinations and the very low smear positivity rates.
2. Panel testing would be more cost-effective. Moreover, a panel test may be of an
added advantage in that it can provide “refresher” training on the appearance of a
positive smear for those laboratories where only rare positives are seen.
Additionally, NTP managers and NRL staff in all three countries - Country A, Country
B, Country C, should provide training for laboratory and TB supervisors to effectively
implement very targeted, problem-oriented on-site supervision activities to complement
other EQA methods and provide strategies for problem solving and laboratory quality
improvement.
7
EQA Overview Module
Key messages:
- EQA can be defined as a system for objectively checking the laboratory’s
performance using an external agency or facility.
- EQA for AFB smear microscopy allows participating laboratories to assess their
capabilities and performance by comparing their results with those of other
laboratories in the network.
- EQA for AFB sputum smear microscopy includes three methods:
o Panel testing
o Blinded rechecking
o On-site supervision
- Effective use of each EQA method will highly depend on the availability of
resources, goals set up by the NTP, TB prevalence, centralization/decentralization
of health services, and the stage of laboratory quality assurance activities in the
country.
- Blinded rechecking or panel testing should be complemented by very targeted
problem-oriented supervision conducted by trained staff.
8