A SystemsSystems-engineering Approach to
Establishing Quality Assurance
Bruce Thomadsen
University of
Wisconsin -Madison
Question
The greatest source of radiotherapy failures come
from which of the following?
Hardware failures
1.
Software failures
2.
Human failures
3.
4.
Organizational failures
Learning Objectives
To understand a systems engineering
approach to QM in a medical setting,
To understand the nature of QM, and
To understand approaches to developing
QM.
Where the Failures Occur
Keeping equipment in calibration is essential!
Few events resulted from machine failures (e.g.,
Therac 25, Omnitron, Varian IMRT MLC).
Frequently, events evolve from some design
problem.
Where the Failures Occur
Design Problem
During the programming, the physicist entered
260 second for one dwell position instead of 26.
Seconds
7 8 9
4 5 6
1 2 3
=
0
Calculator
.
7
4
1
8
5
2
0
9
6
3
.
HDR Unit
1
4
7
*
2
5
8
0
3
6
9
#
Phone
A Problem with Most QA
It focuses on hardware rather
than process.
Keeping equipment in calibration is essential!
Few events resulted from machine failures (e.g.,
Therac 25, Omnitron, Varian IMRT MLC).
Frequently, events evolve from some design
problem.
All events have a strong component of human
failure.
The vast majority of events begin with an operator
error, unlike environments such as Nuclear
plants.
However
ALL failures are organizational failures because:
The system should be such that failures are
expected to happen;
The system should be built to prevent the
propagation of the failures into patient injury.
Question
Working within a System
The greatest source of radiotherapy failures come
from which of the following?
Hardware failures
1.
Software failures
2.
3.
Human failures
Organizational failures
4.
SEIPS Model Work system
(Balance Theory; Smith and Carayon, 1989; Carayon and Smith, 2000)
2000)
Slide from Pascale Carayon
Technology
and Tools
Organization
Person
Tasks
Recognize that failures will occur, human and
mechanical.
Trying harder and paying better attention will
not prevent failures.
Recognizing that failures will occur allows a
focus on how to intercept failures.
But, looking at the process as a system also
allows design to help make failures less likely.
Person approach
versus System approach
Person approach
Focus on errors of individuals
Blaming individuals for
forgetfulness, inattention, or
carelessness
Methods: poster campaigns,
writing another procedure,
disciplinary measures, threat of
litigation, retraining, blaming
and shaming
Target: Individuals
Environment
Slide from Pascale Carayon
System approach
Focus on the conditions under
which individuals work
Building defenses to avert
errors or mitigate their effects
Methods: limiting the incidence
of dangerous errors, creating
systems that are better able to
tolerate the occurrence of
errors and contain their
damaging effects
Target: System (team, tasks,
workplace, organization)
Conventional Latent Error Picture
How to Address Hazards
During the design of a procedure (or looking
at a procedure in place), assess the risks.
For the high risks, try to design them out.
When not possible, develop quality
management.
Reason. Human Error
Error 1990
Quality Management
Shift in focus:
Quality Management – All activities designed to
achieve the desired quality in treatments.
Quality Control – Activities that force specific
quality on a process.
Quality Assurance – Activities that demonstrate
the level of quality of a process.
Organizational Difference
between QA and QC
Input
1
Input
2
Input
3
Input
4
QC
QC
Process
QA
QC
QC
Different Approaches
QC tends to be failure prevention.
QA tends to be failure interception.
Frequency for QM
One problem in clarifying QA and QC is that
some authors use the terms in exactly the
opposite sense, so be careful when reading.
QC — every time a procedure is performed
QA — with a period such that the worse
possible conditions for which the QA screens
would produce no harm. (Maybe each
occurrence…
occurrence…or not.)
Comparison of QA and QC
Important Guidance
QC procedures often require more resources
than QA (to cover large numbers of inputs) but
failures detected by QC less costly to correct.
Relying on QA can add time to a procedure
since failures detected often require repeating
the process.
QA and QC work best together.
• If you are picking up many
problems with your QA, you should
move some resources to QC
• If you are not picking up problems
with your QA, question its utility.
Quality Management Program
Good Beginnings
Acceptance Testing is making sure the
procedure works as specified.
Quality
Management
Quality
Planning
Usually applied to hardware and software.
Also applies to walking through new procedures.
Commissioning gets a procedure started
Quality
Assurance
Quality
Improvement
Quality
Control
Process
Audit
Gathers all the data needed.
Gives confidence that systems work as expected.
Finds when the system fails.
Quality
Audit
Product
Audit
Essential to any procedure.
Development of QM
Our focus now will be on the
periodic QM and the perper-case QM
What to Do?
1.
2.
3.
4.
RiskRisk-based QM Development!
Understand the process
Assess the hazards
Establish the failure propagation patterns
Address the hazards
a.
b.
5.
From the greatest risk and most severe
Use the most effective tools
Test and evaluate
Understand the Process
LDR Brachytherapy Process Tree 1:
Placement followed by dosimetry
Application
Usually best in the form of a process map or
chart.
That allows visualization of relationships.
It will come in handy later in placement of QM
procedures.
Here are some examples.
Hardware operation
Applicator
Patient
identification
Correctness
Software operation
1
Applicator
check
Satisfaction
1
Procedure
correct Correct
target anatomy Correct
applicator
placement
Identity
Placement
Fiducials
Source verification
10
Consistence
Source loading
4
Dummies
Source fixation
Anatomy
Geometry
Identification
Correct
selection
Time recording
Completion
Applicator
fixation
Per
algorithm
1
Data entry
1
Entering data
in computer
factors
Targeting
date
Correct films Interpretation
Calibration factors
Identification
Dummies
Image quality
Record
1
3
Geometry
source
strength
Record Setup
Localization
factors
patient
data
input
Dose / time
calculation
2
EPID imaging Place patient on table
For localization
Align all
3x7x4=84
Emergency response
Insertion in
carrier
1
10 Source
selection
Source
preparation
1 Post Tx
monitoring
Source removal
Removal preparation
1
Removal time
verification
Treatment
termination
Verify beam outlines 4x5x5=100
3x7x4=84
b
Align mold marks
2x7x2=28
marks
3x7x4=84
Recording
source
integrity
Source Radiation
survey
count
Limitations of
algorithms
Dosimetry
calculation
8
3
Placement in well
Set source
2
clinical
Protocol
stage
Anatomic
information
Image quality
Reading
Calibration
Select beam in
4x5x5=100
record & verify
Image 4x5x5=100
Pt in mold
3x7x4=84
a
Make AP image 1x2x1=2
Set
Make
exposure mu
1x2x1=2
Set gantry
Set field size
Register beam
outline c plan
Set machine 1x2x1=2
Verify clearance
and achievability
2x5x2=20
Set
Make
exposure mu
17
Set
parameters
4x5x5=100
3x5x1=15
Make lateral image 1x2x1=2
Repeat for
each beam
Set gantry
b
Set machine
Verify images are adequate
Day 1 imaging verification
1x2x1=2
Approve patient position
Approve treatment
if good
Reimage if necessary
a
4x5x5=100
Verify patient
setup
3x6x4=108
3x6x4=108
Determine patient
Shifts and rotations
2x5x2=20
Do not try to read the labels; enjoy the organization
Successful
Treatment
treatment duration
executed
9
Dose
Other Tx
information information
Prescription
Set chamber
Monitoring
Duration calculation
Optimization
2 Specification
strength
Calculation of strength
1
Removal
calculation
1
Per localization
1
Procedures
leading to an
LDR Patient
treatment
Measurement
APBI with Contura Process Map
Source
loading
Planning quality
assurance
Reconstruction
Review beam images
Register EPID and
4x5x5=100
pseudoradiograph
Load
Pseudoradiograph
4x5x5=100
Load
EPID
2x4x3=24
2x5x4=40
Review setup images
2x6x4=48
Approval to treat
Brachytherapy Information Flow
Intraoperative
Imaging
Perform
Simulation
Imaging
Applicator
Identity
Select &
Identify
Patient
Patient ID
Select Applicator
type, template,
and transfer tube
length
Applicator Type
Applicator
Location
Insert Applicator
Label Applicators
Applicator orientation
Simulation Prep
-Insert dummies
-ID dummies
-Sound Appl
Axial Dwell Localization
-contour CTV
-match images to Ch. No.
-Calculate treatment length
-Number dwell positions
S Funct tion
e
p
Applicator-Channel
Association
Intraop Documentation
Annotated Sim images/Documentation
-Dummy ID-Channel correspondence
-Numbered Applicator images
-CTV contours
-first-last dwell positions
-Dwell spacing & dwell numbering
-Treatment length
-Patient ID
-Applicator type & No.
-Transfer tube length
-Diagram: channel numbering, anatomic site
-Imaging study ID
Assess the Hazards or Risks:
Failure Modes and Effects Analysis
1
Potential
Failure
Potential
Cause of
Failure
Potential
Effects of
Failure
Current
Controls
O S D RPN
QA Data: SK
1
Prepare Rx: Specify
-Dos/Fx and total dosee
-Dose specification
-Opt method
-Opt constraints
Treatment Planning
-Imput simulation images
-Digitize dwell position, target volume, critical anatomy
-Enter channel & dwell nos, Tx length, and dwell spacing
-Select isodose planes/views
-Enter dose opt & specification points
-Enter dose and select Opt Method
-Enter
Local
Plan Approval
Process
Complete Tx
documentation/
setup Forms
2
Intermediate
Hardcopy Rx and
Tx forms
Electronic
plan storage
Hardcopy of plan
End
(Patient)
Time
From J. Williamson
Risk Probability Number
O = likelihood of
occurrence;
S = severity of the effects of
the failure;
D = likelihood failure would
go undetected.
O S D
RPN
Values for O, S, and D between 1 and 10,
(1 = low danger, 10 = high).
How to determine values?
Probability that a Specific Cause
will Result in a Failure Mode (O)
Qualitative
Review
Ranking
Frequency of
Occurrence
Failure is unlikely
1
2
3
4
5
6
7
8
9
10
1/10,000
2/10,000
5/10,000
1/1000
<0.2%
<0.5%
<1.0%
<2.0%
<5.0%
>5.0%
Relatively few
failures
Occasional failures
Repeated failures
Failures are
inevitable
Severity of the Effects Resulting
from a Specific Failure Mode (S)
Not noticeable, no effect on
the patient or on the
department
Inconvenience
Minor dosimetric error
Limited toxicity (may not
require medical attention) or
minor underdose to PTV
Potentially serious toxicity or
injury (may require medical
attention) or major underdose
to PTV
Possible serious toxicities
(requires medical attention)
Catastrophic
1
Detection Ability of
Failure Mode in %
2-3
4
99.99
99.80
99.50
99.00
98.00
95.00
90.00
85.00
80.00
Extreme likelihood
5-6
7-8
9
10
Risk Probability Number
O = likelihood of occurrence;
S = severity of the effects of the
failure;
D = likelihood failure would go
undetected.
O S D
Probability that a Failure Mode will go
Undetected (D)
RPN
RPN = risk probability number = product of
OxSxD.
OxSxD.
In industry, RPN <125, little concern,
however, in medicine, RPN > 40 might
warrant some consideration.
Probability that
failure mode
goes undetected
in %
0.01
0.20
0.50
1.00
2.00
5.00
10.00
15.00
20.00
>20.00
Ranking
1
2
3
4
5
6
7
8
9
10
Current Controls
Usually, there are already some controls in place,
for example, in any new externalexternal-beam
procedure, there is the normal machine QA.
Sometimes you may want to ignore the current
controls to evaluate if they serve the
departmental QM well.
Understand the Propagation of Failure
Example: Calculation Fault Tree
Error in PT
Calculation
Example:
Calculation Fault
Tree with QM
Error in
Input Data
Error in
Data Entry
Example:
Calculation Fault
Tree with Error
Mitigation
Error in
Calculation
Error in QA
Error in
Prescription
Error in QC
Error in QC
Error in
Calculation
Algorithm
Error in QC
Error in QC
Wrong units
Calibration error
Wrong
calibration
or
and
Calculation error
Failure of
verification
3/44
3/44 Source strength
E
or
error
10/44
Dosimetry Error
3/44 Wrong source
data
Failure of
verification
F
Go to
Page 5
Wrong data
format (US/Euro)
Wrong data
(wrong decay
factor)
and
Incorrect entry
Page 4
Error in data
entry
3/44 Failure to enter
or
or alter data (unit
default)
Wrong source in
device
3/44
Dose calculation
error
Erroneous
strength for
source data
and
or
7/44
Measurement
error
Discrepency in
strength between
device and
planning system
Address Risks with QM
1.
2.
APBI with Contura Process Map
Sort the FMEA by RPN, and separately by S.
Mark the top ranked steps on the process map.
Do not try to read the labels; enjoy the organization
Steps for Addressing the Risks
1.
2.
Sort the FMEA by RPN, and separately by S.
Mark the top ranked steps on the process map.
Steps for Addressing the Risks
4.
5.
This helps identify critical regions where group
solutions are possible.
Eventually, all the significant risks will be
addressed.
3.
Maybe the risks are already covered.
Maybe some current controls are unnecessary.
Maybe some current controls are not well suited to
the task.
Mark the top ranked steps on the fault tree.
Look for common causes.
Look for junctions.
Consider placement of QM to cover the risks.
Consider current controls:
6.
Select appropriate tools.
Question
The most effective types of QM make use of
which of the following?
Policies and procedures
1.
Computerization
2.
3.
Education
Interlocks and barriers
4.
Independent checks
5.
Not on the List
Motivation
Ranking of QM Tools
The strength of actions varies:
1. Forcing functions and constraints
2. Automation and computerization
3. Protocols and standard order forms
4. Independent check systems and other redundancies
5. Rules and policies
6.
(Re)Education and Information
From the toolbox of the Institute for Safe Medical
Practices toolbox (ISMP, 1999)
Two Midrange but Useful Tools
And inexpensive
Forms – help eliminate omissions and better
communication.
Independent review – a good QA procedure.
Question
The most effective types of QM make use of
which of the following?
Policies and procedures
1.
Computerization
2.
3.
Education
Interlocks and barriers
4.
Independent checks
5.
Compliance Activities
Some activities required by regulations.
For example, required by NRC
With source changes, surveys around unit
Survey around room
Measurement of transfer tube and applicator length
These add no value to QM but you need to do
them.
Test and Reevaluate
Plans don’
don’t always work
and sometimes create
unintended
consequences.
QI and QP address this.
Steps:
Plan
Do
Check
Act
Quality
Planning
Quality
Assurance
Quality
Improvement
Quality
Management
Quality
Control
Quality
Audit
Process
Audit
Product
Audit
Conclusions
In the new paradigm for quality:
Consider a process a system.
Understand the process and assess the
risk.
Think about QA and QC separately but
together.
Use the most effective and efficient tool.
Review and reevaluate the QM.