Exploring the Medical Laboratory
Quality ToolBox-A
Michael A Noble MD FRCPC
Clinical Microbiology Proficiency Testing program
Program Office for Laboratory Quality Management
University of British Columbia
Vancouver BC
Management
by the
Moment
Management
by the
Moment
Management
by the
Moment
So the better approach is…
An systemic approach of
organization, plan, review, and
action gives you the
best chance of success.
The Quality Toolbox
Lean
What are quality tools?
LEAN
Six Sigma
ISO 9001: 2000
A quality tool is a supplement or component part
of a quality program that usually will not stand alone
but can enhance the total quality system
January 2009
7
Tools in the Toolbox
• Mega-Tools
– Lean
– Six Sigma
– Priority Matrices and
Risk Assessment
• Tools/Techniques
–
–
–
–
–
–
Balanced Scorecards
Brainstorming
Control Charting
Flow Charting
Quality Indicators
Surveys
Quality Indicators are Metrics
(measured process information)
• Determine quality of services.
• Highlight potential quality concerns,
• Identify areas that need further study and
investigation, and
• Track changes over time.
Amazon.com
$30.00
A really
good,
inexpensive
reference
book
Keeping Score
Using the Right Metrics to Drive World Class Performance
1996
Many organizations spend thousands of
hours collecting and interpreting data.
However many of these hours are nothing
more than wasted time because they
analyze the wrong measurements,
leading to inaccurate decision making.
– Mark Graham Brown.
Characteristics of Good Metrics
Measurable
objective
Achievable
contained
Timed
short and long term
Engaging
all levels
Good
Metrics
Interpretable
specific
Balanced
full cycle
Actionable
action oriented
Seven Steps to Successful
Indicators
Do not start data collection until these are addressed
1. Objective
2. Methodology
3. Limits
4. Interpretation
5. Limitations
6. Presentation
7. Action plan
Developing Indicators
Objective
What are you trying to measure?
Methodology
How to capture the data
Limits
Can I preset levels for:
1.
1.
2.
3.
4.
1.
Why am I collecting this information? Be specific
What data needs to be captured
Who (or what) to capture the data
How often to capture the data
Is it achievable (time, resources, revenue)?
Acceptable, Concern, Unacceptable, Critical
Presentation
Graphic or Text
Interpretation
1.
2.
3.
4.
What does it mean?
Does it reflect on YOUR quality?
Can I compare it?
Can I trend it?
Limitations
1.
2.
Unintended variables
What does it not mean?
1.
2.
What will I do if it indicates acceptable performance?
What will I do if it does not?
Action Plan
Developing Indicators
Objective
What are you trying to measure.
Methodology
How to capture the data
Limits
Can I preset levels for:
1.
1.
2.
3.
4.
1.
Why am I collecting this information? Be specific
What data needs to be captured
Who (or what) to capture the data
How often to capture the data
Is it achievable (time, resources, revenue)?
Acceptable, Concern, Unacceptable, Critical
Presentation
Graphic or Text
Interpretation
1.
2.
3.
4.
What does it mean?
Does it reflect on YOUR quality?
Can I compare it?
Can I trend it?
Limitations
1.
2.
Unintended variables
What does it not mean?
1.
2.
What will I do if it indicates acceptable performance?
What will I do if it does not?
Action Plan
Setting Relevant Limits and Ranges
•
•
•
•
•
•
Set Objectively
Validate by Study
Clinical Relevancy
Customer Expectation
Matched Benchmarks
Regulation
60 minutes
Relevant or Easy?
Assessing Quality Indicators
•
•
•
•
Importance
Scientific Acceptability
Feasibility
Usefulness
Potential for Improvement
Reliability and Validity
Implementation and cost
Comprehensive
Having Quality Quality Indicators
IQLM Indicator List
1.
2.
Diabetes monitoring (system)
Hyperlipidemia screening (system)
•
•
•
•
•
•
•
•
•
•
Test Order Accuracy and Appropriateness (pre-analytic)
Patient Identification (pre-analytic)
Adequacy and Accuracy of Specimen Information (pre-analytic)
Blood Culture Contamination (pre-analytic)
Accuracy of point-of-care testing (analytic)
Cervical cytology/biopsy correlation (analytic)
Critical Values Reporting (post-analytic)
Turnaround time (post-analytic)
Clinician satisfaction (post-analytic)
Clinician follow-up (post-analytic)
CLMA Survey
Pre-examination Phase Indicator List
•Ordered test is appropriate
for patient care
•Patient consent
appropriately collected
•Test utilization by clinician
for best patient care
•Physician written order
with every specimen
•Cost/benefit assessment
for laboratory test menu
•Patient identification and
its accuracy
• Preparation of patient for
specimen collection
•Appropriate specimen container
•Timing of specimen collection
•Phlebotomy success
•Specimen integrity
•Specimen quantity
•Specimen transportation
•Accuracy of specimen
identification
•Condition for specimen storage
CLMA Survey
Examination Phase Indicator List
• Quality Control
• EQA-external quality
assessment
• Time to first result
availability
• Specimen
contamination
• Laboratory injuries or
accidents
• Competency of
testing personnel
• Vacancy of technical
staff
CLMA Survey
Post-examination Phase Indicator List
• Result reporting accuracy
• Adequacy of information
for interpretation of
laboratory tests
• Report delivery
turnaround time
• Consistency of critical
values reporting
• Result interpretation by
physician
• Patient’s satisfaction with
laboratory services
• Patient’s satisfaction
specifically with
phlebotomy services
• Physician’s satisfaction
with laboratory services
Contmination Rate: Blood Culture Sets
4.0%
Percent
3.0%
2.0%
1.0%
0.0%
1
2
3
4
5
6
7
8
9
10 11 12 13 14
Time Period
2002-2003
2003-2004
2004-2005
Contmination Rate: Blood Culture Sets
4.0%
Percent
3.0%
Objective:
to ensure that blood
culture results reflect
sepsis.
2.0%
Methodology: Count single bottle
1.0%
0.0%
1
2
3
4
5
6
7
8
9 10 11 12 13 14
positives of common skin
flora/ total sets
Time Period
2002-2003
2003-2004
2004-2005
Limits:
Below 2%
Interpretation: Meeting accepted limits all the time
Limitations Definition may include some true infections and
may miss others
Presentation: Linear time graph
Action plan: Identify and educate blood collector group.
If stable for 2 years, then consider dropping from routine
Quality Indicators and Timing
Use an indicator
only as long as
it provides
you with
useful
information.
Don’t get tied to
your indicators
Underfill Blood Collections
(As a percent of collections per site)
16.0
14.0
12.0
RF2
10.0
RF1
8.0
ER
ICU
6.0
Phlebotomists
4.0
2.0
0.0
1
3
5
7
9
11
13
15
17
19
21
23
25
27
29
31
33
35
37
39
Underfill Blood Collections
(As a percent of collections per site)
Objective:
16.0
14.0
to ensure that blood
culture are properly filled.
12.0
RF2
10.0
RF1
8.0
ER
ICU
6.0
Phlebotomists
Methodology: Count underfilled bottles /
total bottles collected
4.0
2.0
0.0
1
3
5
7
9
11
13
15
17
19
21
23
25
27
29
31
33
35
37
39
Limits:
Below 2% (?)
Interpretation: Wards with inexperienced collectors have problems
Limitations Some frail and elder people have very weak veins and may be
impossible to collect
Presentation: Linear time graph
Action plan: Identify and educate blood collector group.
Caution about
patient outcome indicators
Theoretically, outcomes best assess quality, but
they are the most difficult to measure
– too many confusing variables
• Age, underlying conditions, therapy, circumstance
– require high volumes of detailed data
– need long collection periods
David Hsia
Medicare Quality Improvement
Bad Apples or Bad Systems?
JAMA. 2003;289:354-356.
FACT:
Quality Indicators, Done Well,
Will Consume More Time Than
You Have
•Set Priority
•Set Limits
•Drop Non-Productive Activity
•Target: 10-12
Computer Nonsense Metrics
[urine culture] * [glucose] * [INR]
X100
[NUPA hr] * [Telephone minutes]
Just because a
computer can
calculate a value,
doesn’t mean that
it should.
The BIG SECRET for
Quality Indicator Team
Engage the folks
who do the work,
because they
know what
they do!
Risk management is activity
directed towards assessing,
mitigating (to an acceptable level) and
monitoring of risks to an enterprise.
Risk Management
helps define
PRIORITIES
and helps
PREVENT ERROR
Tough Decisions 1
• A group of 10 laboratories over 300
square miles considers centralizing all
tests to a single facility.
Improved cost efficiencies
Simplified process control
Closer oversight
Improved utilization management
Improved Patient Care
Reduced clinical-lab interface
Extended pre-examination phase
Test delays
Impeded Patient care
Tough Decision 2
• A laboratory considers laboratory redesign
based on LEAN analysis.
Improved workflow
Improved time efficiencies
Solve ergonomic challenges
Improved cost efficiencies
Improved Patient Care
Expensive renovations
Construction interruptions
Staff retraining
Equipment specificity
Test menu specificity
Impeded Patient care
Tough Decision 3
• Laboratory considers implementing a
quality management team.
Better information for better decisions
Better process control
Continual Improvement Process
Improved Patient Care
Staff Reallocation
Increased immediate costs
Quality Commitment
Impeded Patient care
Tough Decision 4
• A laboratory proposes the need for an
on-site level 3 (increased biosafety level)
microbiology laboratory.
Improved biosafety
Reduced laboratory infections
Faster diagnosis
Improved Patient Care
Construction costs
Operating costs
Potential Bioterror target
Impeded Patient care
Risk and Priority and Prevention
are Relative Terms
• Risk tolerance
• Risk transfer
PRIORITY
• Risk aversion
COMFORT
• Risk reduction
• Risk prevention
• Risk avoidance
ACTION
Risk Management Documents
for the Medical Laboratory
• ISO 14971:2007
Medical devices -- Application of risk management to
medical devices
• ISO 20993:2006
Biological evaluation of medical devices -- Guidance on
a risk-management process
• ISO WD TS22367:2007
Medical laboratories -- Reduction of error through risk
management and continual improvement
Linking Quality
and Risk
Managements
Introduction to ISO TS 22367
“Risk management framework has been described as being of
steps, planning for risk, identifying risk and its impacts,
developing risk-handling strategies, and monitoring for risk
control. These steps are consistent with management
requirements as described in ISO 15189:2003 including
identification and control of non-conformities, establishment of
preventive and corrective actions, performance of internal
audit and management review and continual improvement.
It is the objective of this technical report to link these in the
context of the medical laboratory.”
Linking Prevention and Risk
ISO TS 22367
•
•
•
•
Assessment of risk of deviations from standard
The quality manager should establish and maintain a process
for identifying incidents associated with deviations from standards
requirements, estimating and evaluating the associated risks to
patient care, and controlling these risks and monitoring
effectiveness of the control.
The process should include a prospective risk assessment
method for processes considered as high risk.
Assessment of potentially high risk processes may be based
upon previous audit, survey, experience, or evidence-based
literature on procedures where a failure may lead to significant
safety risk to patients.
Linking Prevention and Risk
ISO TS 22367 (Continued)
– The quality manager should identify a team of people to
study the selected process.
Note 1: The team should have personal knowledge of the process.
Note 2: The team should be comprised of people with different levels
and types of knowledge.
– The team should organize a thorough investigation of the
process to include:
a) each step of the process
b) how each step of the process may fail
c) how each failure at each step of the process may affect patient safety
d) the rank of severity of each failure mode effect
e) the most critical failure mode effects
f) potential root causes of the most critical failure mode effects
g) procedures to address the potential root causes.
– The analysis of this Failure Mode Effects Analysis should
form the basis of a Prevention Action Plan.
Occurrence – Severity Grid
(Qualitative)
Remote
Rare
Common Frequent
Insignificant
ACCEPTABLE
RISK
ACCEPTABLE
RISK
ACCEPTABLE
RISK
DEBATABLE
RISK
Mild
ACCEPTABLE
RISK
ACCEPTABLE
RISK
DEBATABLE
RISK
UNACCEPTABLE
RISK
Moderate
ACCEPTABLE
RISK
DEBATABLE
RISK
DEBATABLE
RISK
UNACCEPTABLE
RISK
Severe
DEBATABLE
RISK
UNACCEPTABLE
RISK
UNACCEPTABLE
RISK
UNACCEPTABLE
RISK
Occurrence – Severity Grid
(Semi-Qualitative)
Insignificant
(0.1)
Mild
(0.2)
Moderate
(0.8)
Severe
(0.9)
Remote
(0.2)
Rare
(0.4)
Common Frequent
(0.8)
(0.9)
ACCEPTABLE
RISK
ACCEPTABLE
RISK
ACCEPTABLE
RISK
DEBATABLE
RISK
ACCEPTABLE
RISK
ACCEPTABLE
RISK
DEBATABLE
RISK
UNACCEPTABLE
RISK
ACCEPTABLE
RISK
DEBATABLE
RISK
DEBATABLE
RISK
UNACCEPTABLE
RISK
DEBATABLE
RISK
UNACCEPTABLE
RISK
UNACCEPTABLE
RISK
UNACCEPTABLE
RISK
Severity – Occurrence Grid
• Numerical Values
– Literature Benchmarks
– Consensus
– Experience
• Regardless of source, values should be
both verified, and validated to fit the FMEA
being performed.
HIV Testing Risk Evaluation
Canadian versus American Risk Analysis
Insignificant
(0.1)
Mild
(0.2)
Moderate
(0.8)
Severe
(0.9)
Remote
(0.2)
Rare
(0.4)
Common Frequent
(0.8)
(0.9)
ACCEPTABLE
RISK
ACCEPTABLE
RISK
ACCEPTABLE
RISK
DEBATABLE
RISK
ACCEPTABLE
RISK
ACCEPTABLE
RISK
DEBATABLE
RISK
UNACCEPTABLE
RISK
ACCEPTABLE
RISK
DEBATABLE
RISK
DEBATABLE
RISK
UNACCEPTABLE
RISK
?
UNACCEPTABLE
RISK
UNACCEPTABLE
RISK
UNACCEPTABLE
RISK
Quality Approaches Look Alike
Strategy
Event Steps
EXAMINE
REMEDIATE
AND
CORRECT
CONTROL
IDENTIFY
EXAMINE
DEVELOP
STRATEGY
MONITOR
PLAN
IDENTIFY
EXAMINE
DEVELOP
STRATEGY
MONITOR
DEFINE
MEASURE
ANALYZE
IMPROVE
CONTROL
Corrective
Action Plan
PLAN
MONITOR
Preventive
Action Plan
PLAN
Risk
Management
Six Sigma
Failure Mode Effects Analysis
• FMEA is a systematic method of studying failure.
• 1940s
1960s
1970s
1990s
US Armed Forces
Aerospace (Apollo Space program)
Ford Motor Company (post-Pinto)
semiconductors, food service, plastics,
software, and diagnostics equipment.
• Documented FMEA is a REQUIREMENT in the
automotive, in-vitro diagnostics, and
pharmaceutical industries.
October 1, 2008
FMEA:
Failure Mode Effects Analysis
In a process:
– where are points that the process is likely to
break down?
– what is the probable occurrence of the
different possible break-downs?
– what are the likely severity and consequences
of the different possible break-downs?
– What is the priority list for addressing the
different possible break-downs?
FMEA:
Failure Mode Effects Analysis
• FMEA
– Failure Mode Effects Analysis
• FMECA
– Failure Modes Effects and Criticality Analysis
• FTA
– Fault Tree Analysis
• HACCP
– Hazards Analysis and Critical Control Points
• PDPC
– Process Design Program Chart
A Medical Laboratory FMEA
Physician
Requisition
Sample
Collection
Sample
Transport
Sample
Set-up
Sample
Analysis
A Medical Laboratory FMEA
Site Identification
Good Sample
Physician
Requisition
Sample
Collection
Expertise
Proper Container
Equipment and stains
working properly
Labelling
Sample
Transport
Workload
Time issues
Sample
Set-up
Courier competent
and available
Proper Packaging
Sample
Analysis
A Medical Laboratory FMEA
Site Identification
0.5
Good Sample
0.8
Physician
Requisition
0.5
Expertise
0.3
Proper Container
0.6
0.4
0.2
Equipment and stains
working properly
Sample
Labelling
Transport
0.9
Workload
0.5
Time issues
0.5
0.5
Sample
Collection
1.5
Sample
Set-up
1.9
Courier competent
0.2
and available
SEVERITY
Proper Packaging
0.3
Sample
Analysis
1.3
A Medical Laboratory FMEA
Site Identification
0.05
Good Sample
0.1
Physician
Requisition
0.5
Expertise
0.05
Proper Container
0.1
0.05
0.3
Equipment and stains
working properly
Sample
Labelling
Transport
0.1
Workload
0.5
Time issues
0.5
0.8
Sample
Collection
0.45
Sample
Set-up
0.25
Courier competent
0.4
and available
OCCURENCE
Proper Packaging
0.4
Sample
Analysis
1.05
Pre-Examination Grid
Remote
0.2
Rare
0.5
Common
0.8
Frequent
>1
Insignificant
(0.2)
Requisition Transport
Mild
(0.5)
Moderate
(0.8)
Severe
(>1)
October 1, 2008
Set-up
Collection
Analysis
A warning about FMEA
• If done as a structured “top-down” exercise, many critical
break points will be missed.
• If done as an unstructured “bottom-up” exercise, the
team will get bogged down in minutiae
FMEAs should ALWAYS be:
✔Directed
✔Team Efforts
✔Planned
✔Structured
The BIG SECRET for
The FMEA Team
Engage the folks
who do the work,
because they
know what
they do!