Program Evaluation Committees (PEC), Annual Evaluations, and the 10 year Self Study and Visit in the Next Accreditation System Cuc Mai MD FACP August 2014 Disclaimers • No part of my salary is supported by ACGME. • I am only a messenger and also a program director. • Information may change in the Next Best Accreditation System. Goals and Objectives • Review the goals of the accreditation process in the Next Accreditation System for Maintenance of Accreditation • Define current description of the Annual Program Evaluation, the 10 year self study, and 10 year self study visit • Define the requirements for the Program Evaluation Committee • Describe some actions programs should consider for the 10 year self study The Old Days… • Internal Reviews • Program Information Form • Duration between Site Visits based on Results of Accreditation Visit Maintenance of Accreditation in the Next Accreditation System Annual Program Evaluation (PR-V.C.) Resident performance Faculty development Graduate performance Program quality Documented improvement plan SelfStudy SelfStudy visit Ongoing Improvement Yr 0 Yr 1 Yr 2 AE AE Focused assessments Yr 7at anyYrtime Yr 9 8 based Yr possible 6 Yr 5 Yr 3 Yr 4 AEfrom AE. AE AE AE AEon data AE AE Full site visit at any time if data from AE eggregious. AE: Annual Program Evaluation Yr 10 AE Aims of the Next Accreditation System Enhance the ability of the peer-review system to prepare physicians for practice in the 21st Century To accelerate the movement of the ACGME toward accreditation on the basis of educational outcomes Reduce the burden associated with the current structure and process-based approach © 2013 Accreditation Council for Graduate Medical Education Information Current as of December 2, 2013 10 Self-Study and Self-Study Visit Self-Study Conducted by the program Annual Program Evaluation Review of program goals and improvement efforts Self-Study Visit Conducted by ACGME Field Staff members Annual Data Review Elements Policy 17.61 Review of Annual Data Continuous Data Collection/Review ADS Annual Update Resident Survey Faculty Survey Milestone data Certification examination performance Case Log data/Clinical experience Hospital accreditation data Faculty member and resident scholarly activity and productivity Other: ACGME complaints, Institutional Quality Metrics © 2013 Accreditation Council for Graduate Medical Education Information Current as of December 2, 2013 The Next Accreditation System Continuous Observations Assess Program Improvement(s) Promote Innovation Program Makes Improvement(s) Identify Opportunities for Improvement Next Accreditation System: Ten-Year Self-Study and Self-Study Visit SelfStudy visit Ongoing Improvement Yr 0 Yr 1 Yr 2 Yr 3 Yr 4 Yr 5 Yr 6 Yr 7 Yr 8 Yr 9 Yr 10 AE AE AE AE AE AE AE AE AE AE AE: Annual Program Evaluation Objectives of the Ten Year Self-Study Visit • A comprehensive review of the program – Using the Annual Program Evaluation and 10 year self study – Information on how the program creates an effective learning and working environment – How this leads to desired educational outcomes Objectives of the Ten Year Self Study Visit • Judge program success at Continuous Quality Improvement • Learn future goals of program • Will verify compliance with CORE requirements • Core and subspecialty programs will be reviewed together Core and Subspecialty Programs Reviewed Together • Needs of core residency taken into account • When fellowships are started • In decisions made regarding finite resources • Coordination of curriculum and program resources • Subspecialties can access to core resources • Local oversight of fellowships • Coordinated Self-Study of core & subspecialty programs: • Assess common strengths and areas for improvement • Action plans for areas for improvement • Efficient Self-Study Visit • Less time and resources spent, coordinated collection and review of data © 2014 Accreditation Council for Graduate Medical Education Next Accreditation System: Ten-Year Self-Study and Self-Study Visit SelfStudy Ongoing Improvement Yr 0 Yr 1 Yr 2 Yr 3 Yr 4 Yr 5 Yr 6 Yr 7 Yr 8 Yr 9 Yr 10 AE AE AE AE AE AE AE AE AE AE AE: Annual Program Evaluation Objectives of the 10 year Self Study • Conducted one year before self study visit • Assess ongoing compliance and improvement in all dimensions, specifically on outcomes • Consider program aims and strengths/weakness/opportunities/threats (SWOT) of program • Uses annual program evaluations and ACGME data to review program improvement activities, successes achieved, and areas still in need of improvement Self-Study: Key Processes • Data Gathering • Annual Program Evaluations, ACGME Annual Data • Focus on data gathering as a learning exercise • Evaluate strengths and areas for improvement • Explore opportunities and threats • Reflect stakeholder (residents, faculty, and relevant others) participation, input and perspective • Offer evidence to support conclusions • Data gathering begins in 2014 for programs with Self-Study Visit scheduled for 2015 © 2014 Accreditation Council for Graduate Medical Education The 10-year Self-Study: Timeline Time prior to Self Study Visit 11-12 months ACGME Actions 1. Sends summary of actions/follow-up from Annual Data Review 6-11 months 4 months Program Actions 1. Aggregates data from Annual Program Evaluations 1. Conducts Self Study 1. Sets FINAL SelfStudy Visit Date and informs program 10 days © 2014 Accreditation Council for Graduate Medical Education 1. Completes ADS data update 2. Uploads Self-Study summary to ADS New Areas: Program Aims and Opportunities and Threats • Program aims • Evaluate effectiveness in meeting aims • Assess relevant initiatives and outcomes achieved • Should be realistic • Opportunities and Threats • Assess how factors and contexts external to the program (eg, institutional, local, regional and national) that affect the program • Opportunities: Factors that favor the program, that the program may take advantage of • Threats: Factors that pose risks © 2014 Accreditation Council for Graduate Medical Education Defining Program Aims • Set aims as part of self-identified annual improvement process • Who are our residents/fellows? • What do we prepare them for? • Fellowship • Academic practice • Leadership and other roles • Who are the patients/populations we care for? © 2014 Accreditation Council for Graduate Medical Education Examples of Program Aims • Provide a comprehensive 3 year curriculum to enable residents to learn tertiary, secondary, and primary care skills in all settings. • Educate residents to be excellent practitioners of medically directed anesthesiology in an Anesthesia care team model. • Train individuals with expertise in population health and serving medically underserved. • Produce excellent, independent practitioners who will be local and national leaders, and for academic careers. © 2014 Accreditation Council for Graduate Medical Education Enhancing the Definition of “Areas for Improvement” • Negative aspects that detract from the value of the program or place it at a disadvantage. • You need to address these areas to compete with your best competitor. • What factors within your control detract from the ability to maintain a high-performing program? • What areas need improvement to accomplish objectives or enhance or supplement your existing strengths? • What does your program lack (expertise in a certain subspecialty, a type of technology, access to a particular patient population, faculty with interest and skill in research)? • Is there a lack of some types of resources; is the institution constrained in its capacity to provide support? © 2014 Accreditation Council for Graduate Medical Education DefiningOp “ po rtunities” • Opportunities are external attractive factors that, if acted upon, will contribute to the program flourishing. • What are your capabilities for further evolving the program; how can you capitalize on them? • Has there been recent change in your immediate context that that creates an opportunity for your program? • Are these opportunities ongoing, or is there a narrow window for them? How critical is the timing? © 2014 Accreditation Council for Graduate Medical Education DefiningThr “ e ats” • Threats include external factors that affect the program. • While the program cannot control them, beneficial to have plans to address them if they occur. • What factors beyond your control place your program e at risk? What are changes in rs idents’ p sec ialty choice, regulation, or other factors that may affect the future success of your program? • Are there challenges or unfavorable trends in your immediate context that may affect your program? • E.g. faculty burdened with heavy clinical load that prevents effective teaching and mentorship © 2014 Accreditation Council for Graduate Medical Education Who Should Organize and Conduct the Self-Study? • Not defined • Members of the Program Evaluation Committee? • Natural extension of improvement process through the Annual Program Evaluation • PEC requirements1 • The PEC must be composed of at least 2 faculty members and at least 1 resident(core); • must have a written description of responsibilities(core); 1 ACGME Common Program Requirements, Effective July 2013 © 2014 Accreditation Council for Graduate Medical Education Role of the PEC • The PEC should participate actively in: • planning, developing, implementing, and evaluating educational activities;(detail) • reviewing and making recommendations for revision of competency-based curriculum goals and objectives; • addressing areas of non-compliance with ACGME standards;(detail) • reviewing the program annually using evaluations of faculty, residents, and others, as specified below(detail) 1 ACGME Common Program Requirements, Effective July 2013 © 2014 Accreditation Council for Graduate Medical Education Role of the PEC (2) • PEC prepares a written plan of action • To document initiatives to improve performance in the outlined areas , (core) • To delineate how they will be measured and monitored . (core) • The action plan should be reviewed and approved by the teaching faculty and documented in meeting minutes . (detail) 1 ACGME Common Program Requirements, Effective July 2013 © 2014 Accreditation Council for Graduate Medical Education Next Accreditation System: Ten-Year Self-Study and Self-Study Visit Annual Program Evaluation (PR-V.C.) Resident performance Faculty development Graduate performance Program quality Documented improvement plan SelfStudy visit SelfStudy Ongoing Improvement Yr 0 Yr 1 Yr 2 Yr 3 Yr 4 Yr 5 Yr 6 Yr 7 Yr 8 Yr 9 Yr 10 AE AE AE AE AE AE AE AE AE AE AE: Annual Program Evaluation Annual Program Evaluation • Annual but may be more frequent • Done by Program Evaluation Committee • Document Review of Program Outcomes from Annual Data • Document Review of Program Achievement of Aims, Program Strengths, Program Threats, and Program Opportunities • Develop Continuous Program Improvement Projects and Track/Document Follow-up Sample Improvement Plan Area for Improvement Dissemination of Goals and Objectives Issue(s) Improvement Plan Posted on Educate residents Intranet (5 and faculty clicks to reach) More prominent Not placement on Intranet accessed (1-click) Not known how this is utilized by residents and faculty Make accessible/ viewable in every setting Integrate with resident formative evaluations © 2014 Accreditation Council for Graduate Medical Education Group Responsible 2 residents and 1 faculty member (names) – give them credit for work Target Completion Date Follow-up June XXXX for Quarterly implementation survey at start of new regarding academic year effectiveness of new approach Spot check Maintenance of Accreditation in the Next Accreditation System Annual Program Evaluation (PR-V.C.) Resident performance Faculty development Graduate performance Program quality Documented improvement plan SelfStudy SelfStudy visit Ongoing Improvement Yr 0 Yr 1 Yr 2 AE AE Focused assessments Yr 7at anyYrtime Yr 9 8 based Yr possible 6 Yr 5 Yr 3 Yr 4 AEfrom AE. AE AE AE AEon data AE AE Full site visit at any time if data from AE eggregious. AE: Annual Program Evaluation Yr 10 AE Program Considerations for the 10 Year Self Study Visit • How are you and your subspecialty programs working together? Program Considerations for the 10 year Self Study • Do you have defined program aims? • Have you ever analyzed your program with a SWOT analysis? Program Considerations for the Annual Evaluation? • Do you have a Program Evaluation Committee and policy? • Are you reviewing all the Program Outcomes evaluated by the ACGME and looking for areas of program improvements? • Do you create, follow through on, and document program improvement projects? The Next Accreditation System Continuous Observations Assess Program Improvement(s) Promote Innovation Program Makes Improvement(s) Identify Opportunities for Improvement ACGME Webinars and Other Resources ACGME webinars are available at: http://www.acgme.org/acgmeweb/tabid/431/ProgramandInstitutionalAccreditation/NextAc creditationSystem/Webinars.aspx CLER Overview of Next Accreditation System Milestones, Evaluation, CCCs Specialty-Specific Webinars (Phase I) Phase I Coordinator Webinars (surgical and non-surgical) Specialty-specific Webinars (Phase II): Nov 2013-Dec 2013 Slide presentations for distribution to the GME community: NAS, CCC, Milestones, Annual Program Evaluation/PEC, Updates on Policy– December 2013 Upcoming Specialty-specific Webinars (Phase II): Jan 2014-May 2014 CLER Self-Study (what programs do) Self-Study Visit (what ACGME site visitors do)