Putting the A in ADCL – the work of the Calibration

advertisement
Putting the A in ADCL – the
work of the Calibration
Laboratory Subcommittee
Malcolm McEwen, Donna Stevens,
Tom Slowey and Jan Seuntjens
AAPM Calibration Laboratory Sub-committee
1
What is an ADCL?
‹ Accredited
Dosimetry Calibration Laboratory
‹ Provides calibrations to users for instruments and
radioactive sources for dosimetry in radiotherapy and
diagnostic imaging
‹ Currently three ADCLs:
‹ University
of Wisconsin (Madison)
‹ K&S Associates (Nashville) MD Anderson Cancer
Centre (Houston)
http://uwrcl.medphysics.wisc.edu/
http://www.kslab.com
http://rpc.mdanderson.org/adcl/
2
What does the CLA do?
“Oversee all activities regarding the operations of
the ADCLs. The Subcommittee’s task is to
accredit, supervise and maintain the highest level
of confidence in the quality of the ADCL system,
with sufficient capacity in the system to prevent
undue delays in satisfying the membership’s
calibration needs while providing a choice of
ADCLs.”
Main forum for discussion of issues relating to calibration
of ion chambers and brachytherapy sources
3
Some history
‹ In
1971 the AAPM formed a task group to develop
guidelines for the establishment of a system of
secondary standard calibration laboratories.
‹
The laboratories would be accredited by the AAPM to provide
high precision dosimetry calibrations.
‹ They would be outside of the National Institute of Standards and
Technology (National Bureau of Standards as it was then) but
traceable to NIST.
‹ Purpose:
‹ To reduce the time required for precision calibrations
‹ To create a system of secondary standard laboratories,
‹ To establish a technical resource for the membership of the
AAPM
4
Time progresses …
‹
‹
‹
A sub-committee of the Radiation Therapy Committee of
AAPM has overseen the development of the program to this
day.
Names have changed but the focus is still the same
At the summer meeting of the AAPM in 1995, the
Subcommittee initiated a major revision of the accreditation
protocol which has resulted in the present criteria that now
conform to ISO standard 17025:2005.
5
What does the CLA do?
Develops criteria
Accredits laboratories
Carries out assessment
visits
Monitors performance
Makes recommendations
6
What does the CLA not do?
‹ Does
not set prices
‹ Does not recommend any one ADCL over another
‹ Does not distinguish between commercial and
academic organizations
‹ Does not try to “spread the business around”
7
Registration vs Accreditation
‹ We
all see signs saying “An ISO9001
registered company” but what does it mean?
‹ Registration:
‹ Procedure
used to give written assurance that a system
conforms to specified requirements. Such systems
include those established for the management of product,
process or service quality, and environmental
performance
8
Quality System Registration
‹ ISO
9000 quality systems are generic
‹ Applies to all types of organizations
‹ Recognition is of a quality system that’s defined in
terms the output (products/services) of the quality
system
‹ Registration does not certify or guarantee the quality
of products or service for compliance with specific
technical specifications
9
Accreditation
‹Accreditation:
‹ Procedure
by which an authoritative body gives
formal recognition that a body or person is
competent to carry out specific tasks
10
Laboratory Accreditation
‹ Formal
recognition that a testing laboratory is
competent to carry out specific tests or types of
tests
‹ Key Words:
‹ competent
‹ specific tests (scope of testing)
‹ Accreditation is having a Quality System and
demonstrating competency.
‹ Requires assessment (audit) visits to the laboratory
to verify competency.
11
What does the CLA do?
Develops criteria
Accredits laboratories
Carries out assessment
visits
Monitors performance
Makes recommendations
12
Laboratory Accreditation:
System Auditing
Malcolm McEwen, Donna Stevens, Tom Slowey and
Jan Seuntjens
AAPM Calibration Laboratory Sub-committee
Based on a course by:
PROFESSIONAL QUALITY SYSTEMS
Gary D. Scalise, Instructor
Technical Lead Assessor A2LA
13
What Really Is An Audit?
‹ Audit
or Assessment: to verify conformance
based on objective evidence
14
Types of Audits
‹ First Party
‹ Internal
‹ The laboratory’s own audit
‹ Second Party
‹ External
‹ Usually by a client to their own specific requirements
‹ Third Party
‹ External
‹ By independent organization, e.g. AAPM
‹ Full quality system evaluation
‹ Objective – done by trained and qualified assessors
15
ISO17025 and the AAPM
‹
ISO standard 17025:2005 is the international standard
covering calibration laboratories
‹ The AAPM has a set of Criteria which are used for
the accreditation of ADCLs
‹ The Criteria are based on the requirements set out in
ISO17025
‹ A lab accredited according to the AAPM Criteria is,
by implication, following ISO17025
‹ But – the Criteria have additional requirements that
must be met.
‹ Therefore ISO17025 accreditation is not enough on
its own to satisfy use of the ADCL logo
16
The AAPM Criteria
‹ The
Criteria are based on the requirements
set out in ISO17025
‹ In addition there are detailed technical
requirements for each specific area that will
be covered by the scope of accreditation:
‹
‹
‹
‹
Co-60 absorbed dose to water
LDR and HDR brachytherapy
Diagnostic radiology dosimetry
Electrometer calibration, etc
17
Audits – the details
‹ The
slides will focus on the AAPM criteria and what
the CLA does
BUT
‹ They can be applied to any assessment of a quality
system, quality assurance program, procedure, etc
‹ To both:
¾
¾
external audits (such as carried out by the CLA)
internal audits (checking before someone else does)
18
First - an audit needs Auditors!
‹ Understanding
of quality system philosophies
‹ In-depth knowledge of standards
‹ Technically competent on test methods
‹ Excellent communication skills (written and oral)
‹ Ability to probe and analyze – not obnoxious
‹ Good logic, judgement and integrity
‹ Poise, tact, persistence and maturity
‹ Easy going and positive attitude
‹ Not easy to find!
19
Auditors need to be trained on…
‹ The
standard (AAPM Criteria, ISO17025…)
‹ The organization’s auditing procedures
‹ Information gathering techniques
‹ The human relation aspects of auditing
It unfortunately takes more than 1 lecture
20
Phases of an Audit
Determine if
Quality System
Documentation
meets intent of
the standard
(AAPM Criteria)
Evaluate how well
Quality System
Policies and Procedures
are put into practice
Reading beforehand
“Live”
21
Pre-Audit Preparations
‹ Review
•
Goal: awareness of specific policies and procedures
‹ Prepare
•
all pertinent quality system documents
a detailed gap analysis vs. requirements
Goal: understanding of what is there compared to what
should be
‹ Prepare
questions and checklists for use during audit
‹ Review all documents once again
•
•
Goal: complete familiarity with the laboratory procedures
Don’t start on the plane
22
Scheduling the Audit
‹ Contact
the auditee
‹ Provide results of your document review (if possible)
‹ Request any additional documents
‹ Establish the date and duration of the audit
‹ Provide checklists you want to have filled out
‹ Ensure there is a correct understanding
‹ Send auditee an agenda of the audit
23
Typical Audit Sequence
Opening Meeting
Introduction of each auditee
Gathering the
information and evidence
Summary with each auditee
Compiling Report
Closing Meeting
24
Opening Meeting
‹ Gather
all the major participants
‹ Laboratory
director and quality manager
‹ Supervisors,
key personnel, analysts as needed
‹ Discuss
the scope of the audit
‹ Review the agenda of the audit
‹ Discuss how deficiencies will be handled
‹ No surprises at the closing meeting
25
What’s being looked at?
1.
Management Requirements
¾ From Section 4 of AAPM Criteria
¾ Organizational Structure, Quality Policy, Technical
Management, Records Management, Purchasing
Policy, Internal Audit, etc.
2.
Technical Requirements
¾
¾
From Section 5 of the Criteria (and Appendices)
Test methods, Equipment, Analysis, Reporting, etc.
26
Gathering Information and
Evidence
‹ Asking
Questions
‹ Inspecting Facility and Equipment
‹ Examining Documents
‹ Examining Records
‹ Observing Activities
27
Limitations of Questions
‹ Can’t
always take answers at face value
‹ Answers may not be the truth
‹ May be intentionally misleading
‹ Could be coached responses
‹ But remember:
Auditors are in the evidence business
hunches and opinions don’t count
28
Technical Proficiency
‹ Beyond
system auditing, an audit of a
laboratory requires demonstrated technical
proficiency and competency (Section 5.0)
‹ Auditors can verify (see) this four ways:
1.
2.
3.
4.
Verify suitable equipment
Verify appropriate procedure/test method
Interview personnel to verify knowledge
Observe the test to verify competence
29
Inspect Facility & Equipment
‹ Evaluate
design layout of laboratory
‹ Proper environmental conditions and monitoring?
‹ Sufficient lighting? Temperature?
‹ Effective separation between areas?
‹ Good housekeeping? Appropriate security?
‹ Suitable equipment – range, precision?
‹ Status of equipment – maintenance, calibration?
30
Examining Documents
‹ Documents
available for use? Implemented?
‹ Is there proper document control for
‹ Internal
documents?
‹ External documents?
‹ Document
changes properly made?
‹ Draft procedures in use?
‹ Obsolete documents removed?
‹ Verify that testing follows written procedures
31
Examining Records
‹ Quality
and technical records in place?
‹ Records allow traceability and repetition?
‹ Procedures being followed for archiving?
‹ Records include identity of all personnel?
‹ Contemporaneous records? Complete?
‹ Error corrections properly made?
32
Observation of Activities
‹ Witness
demonstrations of procedures by staff
‹ Confirms what we heard from our questions
‹ Confirms that analysts are following procedures
‹ Determines status of equipment and setup
‹ Enables us to judge technical skills
‹ Establishes the level and depth of training
‹ Gains information on supervision
33
Aside – CLA surveillance
‹ There
is no requirement in a surveillance visit to
witness a procedure (calibration)
‹ However, it may be useful:
‹ if
there is any confusion with documentation
‹ if it’s a simple procedure
‹ to help familiarize the auditor with the lab facilities
and personnel
‹ if the lab requests it
34
Compare Audit Findings to…
‹ AAPM
Criteria requirements (or relevant
standard)
‹ Need
specific clause #
‹ The
internal quality system requirements
‹ Technical laboratory procedures
‹ Agreed methods with the client
‹ Compare reality to requirements,
‹ But
remember the rule…
35
#1 Rule of Auditing
If you cannot express the non-conformance
in the words of the quality system standard,
the appropriate technical specification or
test method, or the company’s own policies
and procedures,
THEN YOU DO NOT HAVE A
NON-CONFORMANCE
36
Audit Reporting
‹ Reporting
‹ Verbal
has more than one aspect
reporting
‹ Feedback
‹ Closing
to the auditee at time of finding
meeting
‹ Summarize
findings to management at close of audit
‹ Written
deficiencies statements
‹ Documented audit report
‹ Documented
as soon as possible (same day)
37
Recommendation
‹ Don’t
keep tight-lipped until the presentation of
the report because:
‹ Shock
and Awe are not legitimate assessor tools
‹ Information may be power but you are really there to
assist not control
‹ If a finding is simple to rectify then it can be cleared
before you leave (and that’s easier for everyone)
‹ You may be mistaken than it’s a non-conformance
38
When Something Seems Wrong
‹ Is
it really wrong?
‹ Do the lab personnel know its wrong?
‹ What is their explanation?
‹ Is it an isolated event, or a symptom of a deeper
problem?
‹ Why didn't the quality system detect it?
‹ What lapse in the quality system allowed this to
happen?
39
Compiling the Audit Report
‹ Writing
the Non-Conformances (Deficiencies)
‹ Assembling the complete audit report
‹ Scope
of Audit: What, How, Why
‹ Agenda of Audit: When, Where, How Long
‹ Identification of auditor(s) and auditee(s): Who
‹ Non-Conformances
‹ Completed checklists
‹ Time frame for corrective action responses
Make sure you leave time for this!
40
What is a nonconformity?
‹ Definition:
‹ “non-fulfillment
of a requirement”
‹ An
equivalent term is “deficiency”
‹ “Non-compliance” should be avoided as it
has a legal implication
41
What is an “audit finding?”
‹ Definition:
‹ “result
of the evaluation of the collected audit
evidence against audit criteria”
‹ Findings
can be good or “not so good”
‹ Can indicate conformity, nonconformity or
opportunities for improvement
42
Writing Deficiencies
‹ Clear
‹ Concise
‹ Non-confrontational
‹ Non-blaming
‹ Non-personal
43
Writing a Deficiency
Structure of a deficiency statement
The Requirement
Description of the deficiency
The objective evidence
44
Writing Deficiencies
‹ Directly
related to the relevant quality
system standard, technical procedures,
laboratory policies and procedures, etc.
‹ And based on a clear statement of objective
evidence recorded during the audit
45
Recommendations
‹ Auditors
are free to make recommendations
that result from audit findings or general
observations of the laboratory
‹ Laboratories are free to ignore any
recommendations!
‹ But recommendations can be a real help to the
development and improvement of a QS (the lab
is effectively getting free consultancy) and
therefore should be considered
46
Closing Meeting
‹ Gather
participants – everyone if possible
‹ Review the compliance areas – first
‹ Review the non-compliance issues
‹ Be careful to distinguish between non-compliances
and recommendations
‹ Complete any records
‹ Thank people for their assistance and time
47
Corrective Actions
‹A
deficiency triggers the need for corrective action
‹ The auditee is responsible for corrective action
‹ Conduct
a root cause investigation for each
‹ Identify potential corrective actions
‹ Determine the significance and resources
‹ Evaluate possible effects on other departments or other
elements of the quality system
‹ Establish a time frame for implementation
‹ Reference each corrective action to one deficiency
48
Follow-Up on Deficiencies
‹ Auditee
should first monitor that the corrective
action brought desired results
‹ Verify that the agreed corrective action has been
effective in the long term
‹ Additional audits may be warranted if the
laboratory is still out of compliance
49
Audit Closure
‹ An
audit is formally closed when…
‹ All
reported nonconformities are addressed by
corrective action plan
‹ Responsibility
‹ Corrective
action is implemented
‹ Responsibility
‹ Corrective
of auditee
of auditee
action is verified as effective
‹ Responsibility
of audit program manager (CLA)
50
Audit summary
‹ Audits
are a valuable tool for any process-based
organization
‹ Audits are compliance based
‹ An auditor only writes deficiencies when
objective evidence has been gathered
‹ All deficiencies must directly relate to the
relevant standard, test method, etc.
‹ Remember the #1 Rule of Auditing
51
What does the CLA do?
Develops criteria
Accredits laboratories
Carries out assessment
visits
Monitors performance
Makes recommendations
52
Thank you for your attention
53
Download