Putting the A in ADCL – the work of the Calibration Laboratory Subcommittee Malcolm McEwen, Donna Stevens, Tom Slowey and Jan Seuntjens AAPM Calibration Laboratory Sub-committee 1 What is an ADCL? Accredited Dosimetry Calibration Laboratory Provides calibrations to users for instruments and radioactive sources for dosimetry in radiotherapy and diagnostic imaging Currently three ADCLs: University of Wisconsin (Madison) K&S Associates (Nashville) MD Anderson Cancer Centre (Houston) http://uwrcl.medphysics.wisc.edu/ http://www.kslab.com http://rpc.mdanderson.org/adcl/ 2 What does the CLA do? “Oversee all activities regarding the operations of the ADCLs. The Subcommittee’s task is to accredit, supervise and maintain the highest level of confidence in the quality of the ADCL system, with sufficient capacity in the system to prevent undue delays in satisfying the membership’s calibration needs while providing a choice of ADCLs.” Main forum for discussion of issues relating to calibration of ion chambers and brachytherapy sources 3 Some history In 1971 the AAPM formed a task group to develop guidelines for the establishment of a system of secondary standard calibration laboratories. The laboratories would be accredited by the AAPM to provide high precision dosimetry calibrations. They would be outside of the National Institute of Standards and Technology (National Bureau of Standards as it was then) but traceable to NIST. Purpose: To reduce the time required for precision calibrations To create a system of secondary standard laboratories, To establish a technical resource for the membership of the AAPM 4 Time progresses … A sub-committee of the Radiation Therapy Committee of AAPM has overseen the development of the program to this day. Names have changed but the focus is still the same At the summer meeting of the AAPM in 1995, the Subcommittee initiated a major revision of the accreditation protocol which has resulted in the present criteria that now conform to ISO standard 17025:2005. 5 What does the CLA do? Develops criteria Accredits laboratories Carries out assessment visits Monitors performance Makes recommendations 6 What does the CLA not do? Does not set prices Does not recommend any one ADCL over another Does not distinguish between commercial and academic organizations Does not try to “spread the business around” 7 Registration vs Accreditation We all see signs saying “An ISO9001 registered company” but what does it mean? Registration: Procedure used to give written assurance that a system conforms to specified requirements. Such systems include those established for the management of product, process or service quality, and environmental performance 8 Quality System Registration ISO 9000 quality systems are generic Applies to all types of organizations Recognition is of a quality system that’s defined in terms the output (products/services) of the quality system Registration does not certify or guarantee the quality of products or service for compliance with specific technical specifications 9 Accreditation Accreditation: Procedure by which an authoritative body gives formal recognition that a body or person is competent to carry out specific tasks 10 Laboratory Accreditation Formal recognition that a testing laboratory is competent to carry out specific tests or types of tests Key Words: competent specific tests (scope of testing) Accreditation is having a Quality System and demonstrating competency. Requires assessment (audit) visits to the laboratory to verify competency. 11 What does the CLA do? Develops criteria Accredits laboratories Carries out assessment visits Monitors performance Makes recommendations 12 Laboratory Accreditation: System Auditing Malcolm McEwen, Donna Stevens, Tom Slowey and Jan Seuntjens AAPM Calibration Laboratory Sub-committee Based on a course by: PROFESSIONAL QUALITY SYSTEMS Gary D. Scalise, Instructor Technical Lead Assessor A2LA 13 What Really Is An Audit? Audit or Assessment: to verify conformance based on objective evidence 14 Types of Audits First Party Internal The laboratory’s own audit Second Party External Usually by a client to their own specific requirements Third Party External By independent organization, e.g. AAPM Full quality system evaluation Objective – done by trained and qualified assessors 15 ISO17025 and the AAPM ISO standard 17025:2005 is the international standard covering calibration laboratories The AAPM has a set of Criteria which are used for the accreditation of ADCLs The Criteria are based on the requirements set out in ISO17025 A lab accredited according to the AAPM Criteria is, by implication, following ISO17025 But – the Criteria have additional requirements that must be met. Therefore ISO17025 accreditation is not enough on its own to satisfy use of the ADCL logo 16 The AAPM Criteria The Criteria are based on the requirements set out in ISO17025 In addition there are detailed technical requirements for each specific area that will be covered by the scope of accreditation: Co-60 absorbed dose to water LDR and HDR brachytherapy Diagnostic radiology dosimetry Electrometer calibration, etc 17 Audits – the details The slides will focus on the AAPM criteria and what the CLA does BUT They can be applied to any assessment of a quality system, quality assurance program, procedure, etc To both: ¾ ¾ external audits (such as carried out by the CLA) internal audits (checking before someone else does) 18 First - an audit needs Auditors! Understanding of quality system philosophies In-depth knowledge of standards Technically competent on test methods Excellent communication skills (written and oral) Ability to probe and analyze – not obnoxious Good logic, judgement and integrity Poise, tact, persistence and maturity Easy going and positive attitude Not easy to find! 19 Auditors need to be trained on… The standard (AAPM Criteria, ISO17025…) The organization’s auditing procedures Information gathering techniques The human relation aspects of auditing It unfortunately takes more than 1 lecture 20 Phases of an Audit Determine if Quality System Documentation meets intent of the standard (AAPM Criteria) Evaluate how well Quality System Policies and Procedures are put into practice Reading beforehand “Live” 21 Pre-Audit Preparations Review • Goal: awareness of specific policies and procedures Prepare • all pertinent quality system documents a detailed gap analysis vs. requirements Goal: understanding of what is there compared to what should be Prepare questions and checklists for use during audit Review all documents once again • • Goal: complete familiarity with the laboratory procedures Don’t start on the plane 22 Scheduling the Audit Contact the auditee Provide results of your document review (if possible) Request any additional documents Establish the date and duration of the audit Provide checklists you want to have filled out Ensure there is a correct understanding Send auditee an agenda of the audit 23 Typical Audit Sequence Opening Meeting Introduction of each auditee Gathering the information and evidence Summary with each auditee Compiling Report Closing Meeting 24 Opening Meeting Gather all the major participants Laboratory director and quality manager Supervisors, key personnel, analysts as needed Discuss the scope of the audit Review the agenda of the audit Discuss how deficiencies will be handled No surprises at the closing meeting 25 What’s being looked at? 1. Management Requirements ¾ From Section 4 of AAPM Criteria ¾ Organizational Structure, Quality Policy, Technical Management, Records Management, Purchasing Policy, Internal Audit, etc. 2. Technical Requirements ¾ ¾ From Section 5 of the Criteria (and Appendices) Test methods, Equipment, Analysis, Reporting, etc. 26 Gathering Information and Evidence Asking Questions Inspecting Facility and Equipment Examining Documents Examining Records Observing Activities 27 Limitations of Questions Can’t always take answers at face value Answers may not be the truth May be intentionally misleading Could be coached responses But remember: Auditors are in the evidence business hunches and opinions don’t count 28 Technical Proficiency Beyond system auditing, an audit of a laboratory requires demonstrated technical proficiency and competency (Section 5.0) Auditors can verify (see) this four ways: 1. 2. 3. 4. Verify suitable equipment Verify appropriate procedure/test method Interview personnel to verify knowledge Observe the test to verify competence 29 Inspect Facility & Equipment Evaluate design layout of laboratory Proper environmental conditions and monitoring? Sufficient lighting? Temperature? Effective separation between areas? Good housekeeping? Appropriate security? Suitable equipment – range, precision? Status of equipment – maintenance, calibration? 30 Examining Documents Documents available for use? Implemented? Is there proper document control for Internal documents? External documents? Document changes properly made? Draft procedures in use? Obsolete documents removed? Verify that testing follows written procedures 31 Examining Records Quality and technical records in place? Records allow traceability and repetition? Procedures being followed for archiving? Records include identity of all personnel? Contemporaneous records? Complete? Error corrections properly made? 32 Observation of Activities Witness demonstrations of procedures by staff Confirms what we heard from our questions Confirms that analysts are following procedures Determines status of equipment and setup Enables us to judge technical skills Establishes the level and depth of training Gains information on supervision 33 Aside – CLA surveillance There is no requirement in a surveillance visit to witness a procedure (calibration) However, it may be useful: if there is any confusion with documentation if it’s a simple procedure to help familiarize the auditor with the lab facilities and personnel if the lab requests it 34 Compare Audit Findings to… AAPM Criteria requirements (or relevant standard) Need specific clause # The internal quality system requirements Technical laboratory procedures Agreed methods with the client Compare reality to requirements, But remember the rule… 35 #1 Rule of Auditing If you cannot express the non-conformance in the words of the quality system standard, the appropriate technical specification or test method, or the company’s own policies and procedures, THEN YOU DO NOT HAVE A NON-CONFORMANCE 36 Audit Reporting Reporting Verbal has more than one aspect reporting Feedback Closing to the auditee at time of finding meeting Summarize findings to management at close of audit Written deficiencies statements Documented audit report Documented as soon as possible (same day) 37 Recommendation Don’t keep tight-lipped until the presentation of the report because: Shock and Awe are not legitimate assessor tools Information may be power but you are really there to assist not control If a finding is simple to rectify then it can be cleared before you leave (and that’s easier for everyone) You may be mistaken than it’s a non-conformance 38 When Something Seems Wrong Is it really wrong? Do the lab personnel know its wrong? What is their explanation? Is it an isolated event, or a symptom of a deeper problem? Why didn't the quality system detect it? What lapse in the quality system allowed this to happen? 39 Compiling the Audit Report Writing the Non-Conformances (Deficiencies) Assembling the complete audit report Scope of Audit: What, How, Why Agenda of Audit: When, Where, How Long Identification of auditor(s) and auditee(s): Who Non-Conformances Completed checklists Time frame for corrective action responses Make sure you leave time for this! 40 What is a nonconformity? Definition: “non-fulfillment of a requirement” An equivalent term is “deficiency” “Non-compliance” should be avoided as it has a legal implication 41 What is an “audit finding?” Definition: “result of the evaluation of the collected audit evidence against audit criteria” Findings can be good or “not so good” Can indicate conformity, nonconformity or opportunities for improvement 42 Writing Deficiencies Clear Concise Non-confrontational Non-blaming Non-personal 43 Writing a Deficiency Structure of a deficiency statement The Requirement Description of the deficiency The objective evidence 44 Writing Deficiencies Directly related to the relevant quality system standard, technical procedures, laboratory policies and procedures, etc. And based on a clear statement of objective evidence recorded during the audit 45 Recommendations Auditors are free to make recommendations that result from audit findings or general observations of the laboratory Laboratories are free to ignore any recommendations! But recommendations can be a real help to the development and improvement of a QS (the lab is effectively getting free consultancy) and therefore should be considered 46 Closing Meeting Gather participants – everyone if possible Review the compliance areas – first Review the non-compliance issues Be careful to distinguish between non-compliances and recommendations Complete any records Thank people for their assistance and time 47 Corrective Actions A deficiency triggers the need for corrective action The auditee is responsible for corrective action Conduct a root cause investigation for each Identify potential corrective actions Determine the significance and resources Evaluate possible effects on other departments or other elements of the quality system Establish a time frame for implementation Reference each corrective action to one deficiency 48 Follow-Up on Deficiencies Auditee should first monitor that the corrective action brought desired results Verify that the agreed corrective action has been effective in the long term Additional audits may be warranted if the laboratory is still out of compliance 49 Audit Closure An audit is formally closed when… All reported nonconformities are addressed by corrective action plan Responsibility Corrective action is implemented Responsibility Corrective of auditee of auditee action is verified as effective Responsibility of audit program manager (CLA) 50 Audit summary Audits are a valuable tool for any process-based organization Audits are compliance based An auditor only writes deficiencies when objective evidence has been gathered All deficiencies must directly relate to the relevant standard, test method, etc. Remember the #1 Rule of Auditing 51 What does the CLA do? Develops criteria Accredits laboratories Carries out assessment visits Monitors performance Makes recommendations 52 Thank you for your attention 53