PIFmanship 101 (or how to survive writing/reviewing program information forms) Ann M. Dohn, MA Director, GME Designated Institutional Official (DIO) & Nancy Piro, PhD Program Manager/Education Specialist Stanford University Medical Center Department of Graduate Medical Education Disclosure No conflicts of interest to report Disclosure #2 We have made every attempt to protect the identities of the guilty. PIFmanship 101 “I see a Site Visit coming in your future…” PIF Writing 101 Gentle words of wisdom… – The first impression of your program for the site visitor will be your PIF. Be accurate Answer the questions (be terse…be tight) Have documentation to back up your answers Start early-it takes months to write a good PIF This is not something your mother, spouse, best friend, or admin asst. can do for you! General Information Glossary of ACGME terms : http://www.acgme.org/acWebsite/abo ut/ab_ACGMEglossary.pdf Examples of “common information” required in the PIFs we provide to our Program Directors & Coordinators: 1) Hospital Statistics: – 613 Licensed Beds at SHC – 272 Beds at LPCH 52 Obstetric 220 Pediatric 2. Major Participating Institutions (Affiliates) 1. 2. 3. 4. 5. 6. 7. Children's Health Council [058191] Kaiser Permanente Medical Center (Santa Clara) [050571] Kaiser Santa Teresa [058092] Lucile Salter Packard Children's Hospital at Stanford [050572] San Mateo Medical Center [050585] Santa Clara Valley Medical Center [050438] Veterans Affairs Palo Alto Health Care System [050273] 3. Library Resources http://lane.stanford.edu/biomedresources/index.html?laneNav=3 Background There are actually 2 PIFS to complete: – The Common PIF Addresses the program’s compliance with the Common Program Requirements common to all GME programs. The common PIF is electronically generated through the Web Accreditation Data System (WebADS) – The Specialty PIF Addresses compliance with the specialty specific program requirements Word document downloaded from the ACGME Site Common PIF Common Program Information Form www.acgme.org Use your ACGME assigned User ID & password Go to the PIF preparation section (left hand menu) Update the Common PIF data. Most of the data should be updated annually by your coordinator. The common PIF contains questions regarding participating sites, faculty CVs, evaluation, and duty hours. Some information is entered only at site visit time. Once all of the data is entered and VALIDATED Print Version Control Start by keeping track of which version of the PIF you are working on… You must hit “refresh” in WEBADS to save your changes “Refresh” before printing Common PIF –1 Common PIF Data feeds from ACGME WebADS TWO (or THREE) Original Signatures Required – Program Director – Core Program Director for Fellowships – DIO Oops Do you really need to have original signatures on the face page? – YES!!!!!! – (we were cited on this) Common PIF – 2 Common PIF Respond to previous citations – Data feeds from ACGME WebADS – If you had a progress report due, the data remains in WebADS Automatically populated into your next PIF Oops Common PIF Major Changes are: – Only since last site visit – Program Format, e.g., Have you gone from 3 to 4 years? – Change in Resident Complement? – Change in Program Leadership? – Change in Participating Sites? Ah Come On Now… Too Little (they had a new program director) Too Much (they had an old program director) List the outstanding or special feature of the program • • • • • • • Stanford’s fellowship program has a long history of training academicians and future leaders in XX and XX and is greatly enhanced by the NIH institutional grants and the tremendous resources available at Stanford University. The program has an NIH Institutional Research Training grant (T32) for the past 20+ years, and the division also has an NIH institutional Digestive Disease Center grant (P30) for the past 10+ years which is currently undergoing re-submission. The Stanford campus has great talents and excellent opportunities for both education and research. The key faculty members have strong patient care and research activities, and are recognized leaders in their fields both nationally and internationally. The institutional training grant provides both salary and tuition support for fellows to initiate and eventually pursue an academic career. In terms of research, one important feature of the fellowship program is the tremendous amount of expertise and resources available at Stanford University. Increasingly, we are encouraging fellows to be co-mentored by faculty members in other fields and be groomed to be the future leaders in those fields through an inter-disciplinary approach between the and other Divisions or Departments of the School of Medicine. We have a long track record of training fellows in health outcomes research, in collaboration with the Stanford Health Policy and Research Center. Fellows pursuing a career in this field usually also enroll in a formal Master’s degree program during their second and third year of their fellowship. The second area is Biodesign. This is a natural fit for XX since this is a procedure-oriented subspecialty with emerging areas of biodesign research activities, such as natural orifice translumenal endoscopic surgery (NOTES), that require the development of new equipment and techniques. Stanford has a world renowned biodesign group, that is training leaders in biomedical technology and innovation and has been offering a 1-year biodesign fellowship since 2001. We have developed a Biodesign tract in our fellowship program in which our fellows spend their second year as a biodesign fellow and spending the third year in completing a research project in biodesign. Another new program is modeled after the biodesign program but focuses on operational research is the Clinical Excellence Research center. One of our current second year fellows is being co-mentored by Dr. Arnold Milstein, Director of the Clinical Excellence Research Center. Further we are in the process of developing a research tract in in conjunction with the Global Health Program at Stanford and one of our 2012 fellows will be focusing on this. All these opportunities make the Stanford XX fellowship program unique in the country and the world, generating pioneers in new research areas and training future research and academic leaders. A major strength of our clinical program is its diverse patient population cared for by a diverse faculty of basic and … Common PIF – 3 Common PIF Participating Sites – Feeds from ACGME WebADS – Stanford, for example, is NOT a single limited site sponsor as we have 85 programs – Only required sites appear on the PIF Required means all residents rotate to that site – Length of rotation Make sure the length of rotations adds up to 12 months per year across all sites Common PIF - 4 Common PIF Participating Sites – Program Letters of Agreement (PLAs) We provide a PLA template on our GME Website for Programs to use Must be signed and dated The PLA may be between you and yourself… Common PIF - 5 Common PIF Program Director MUST be able to approve the teaching faculty The Program Director MUST evaluate the faculty and approve continuation as teaching faculty Program Directors MUST comply with the university’s or medical center’s policies on selection, evaluation, promotion, disciplinary action and supervision. – e.g., at Stanford we refer them to: The GME Housestaff Policies and Procedures – on the GME Website http://med.stanford.edu/gme/policy/ Common PIF The Program Director MUST be familiar with ACGME Policies and Procedures – Institutional Requirements – Common Requirements – Program Specific Requirements Common PIF Physician Faculty Roster – List Alphabetically and by Site – Faculty who spend at least 10 hrs per week in resident education – Make sure the data matches their CVs Common PIF Faculty Teaching Staff – Hours/week devoted to education Should be realistic – Board certification If double boarded include both boards If you are sub-specialty program director or faculty, ensure primary boards are included Oops Common PIF- 6 Common PIF Potential Area for a HUGE number of citations for Programs: – Must be totally completed (no BLANKS) – Ensure certification is valid – Ensure license is current and hasn’t lapsed – Is your program director board certified? – Are your faculty board certified? – How old are the publications? – How old are the appointments? – Etc.. Are We Confused? Concise Summary of Role in Program: Dr. Donald Duck is the Chief of the “Animation” Fellowship program. The associate director is Dr. Mickey Mouse the former director. In the capacity of CoDirector of the Fellowship, he serves to devise and structure the clinical training rotations for the fellows Common PIF – Concise Summary of Role in Program: <Program Director> oversees the operations of the entire program; supervises trainees during patient-care activities, mentors trainees’ research/projects, provides professional training, tutorials (lectures and seminars); monitors duty hour compliance; coordinates evaluation of courses, rotations, trainees and faculty. Common PIF Selected Bibliography – Strict limit of 10 – Strict limit only the last five years -“Publications from the last 5 years” excludes any before 2007… – Publications should not be “in press” if submitted many years ago…. – Submitted doesn’t equal a “Publication” If not ABMS certified, explain… Common PIF - 7 Common PIF - 8 Common PIF Number of ACGME Approved Positions – Note: If you have a resident making up a maternity leave, you must ask ACGME for an extra slot if you are over your cap Actively enrolled residents – Other than interns, everyone should have prior years of GME Common PIF - 9 Common PIF-10 Common PIF Transferred, Withdrawn or Dismissed Residents – Residents who resign are NOT dismissed – Must have summative evaluations on transferred trainees Evaluations – Yes residents are evaluated following each learning experience (ie. rotation) – Yes- evaluations are documented and state how Electronically Files Common PIF Methods of Evaluation – Assessment Method Direct Observation Simulation OSCE Etc – Evaluator Faculty Nurses Program Director 360 (required as of 2007) Common PIF -11 Common PIF Evaluation Describe how evaluators are educated to use the assessment methods listed above so that residents are evaluated fairly and consistently. – The Program Director meets with evaluators annually before new fellows start in the training program and reviews, answers questions and discusses the core competencies and core competency performance evaluations used in the program and assessment methods with the faculty. The electronic assessment system and the scales and their use is also reviewed and discussed in faculty meetings in order to ensure that faculty are fully educated and up to date with the assessment methods and processes. Common PIF Evaluation Describe how residents are informed of the performance criteria on which they will be evaluated. – At the beginning of each academic year, the Program Director conducts an orientation to address several key issues related to the residency including the performance criteria on which the residents will be evaluated. During this orientation session, the Program Director carefully details the specific evaluation methods to be used. Both the criteria and methods are also documented in the residency handbook. Additionally, at the beginning of each block rotation, the rotation director meets with the resident and carefully delineates the expectations and performance criteria on which the resident will be evaluated for that specific rotation. Common PIF Evaluation Describe the system to ensure that faculty complete written evaluations of residents in a timely manner following each rotation or educational experience. – At the end of each block rotation, our online resident data collection and tracking system, <our residency management system>, sends an automated reminder to the service attending(s) to evaluate the resident(s) on that particular rotation. Using <the system> the Program Coordinator tracks pending evaluations and follows up with faculty as needed to urge them to complete their evaluations on time. In the unlikely event that a faculty member is unresponsive to the coordinator's requests to complete evaluations, the Program Director contacts the faculty member and requests him/her to complete the evaluations. Common PIF Evaluation Describe the process used to complete and document written semiannual resident evaluations, including the mechanism for reviewing results (e.g., who meets with the residents and how the results are documented in resident files). – Residence performance is evaluated by the teaching faculty at the conclusion of each rotation using an electronic evaluation form <the system>. Additionally, each faculty mentor meets with their resident advisee quarterly and documents a summary of the meeting to be placed in the resident’s file. The Program Director meets with each resident on an individual basis semiannually to review the accumulated performance evaluations and mentor notes, provide feedback, and update the resident’s learning plan as appropriate. A summary of these meetings is documented and placed in the resident’s file. The residents are free to review the contents of their records at any time. Common PIF Evaluation Describe the system used by the residents to provide annual confidential written evaluations of the teaching faculty (have examples and forms available for review by site visitor). – Residents annually confidentially evaluate the teaching faculty using <system>. The electronically submitted <system>. evaluation forms are anonymous. Paper copies of completed evaluations without personal identifiers are printed by the Program Coordinator, reviewed by the Program Director, and presented to individual faculty members for review and consideration. – Written evaluations are completed electronically and anonymously by using a centralized system used throughout the hospital to provide annual aggregated data. Fellows are sent e-mail reminders about completing evaluations in a timely manner by the Fellowship Coordinator. The program coordinator follows up, aggregates the results and forwards them to the Program Director. The Program Director reviews the results and takes any actions appropriate with individual program faculty. Common PIF Evaluation Describe the program's (or Department's, if applicable) system for evaluating and providing feedback to the teaching faculty. – Residents annually evaluate the teaching faculty using the <system>. The electronically submitted evaluation forms are anonymous. Paper copies of completed evaluations without personal identifiers are printed by the Program Coordinator, reviewed by the Program Director, and presented to individual faculty members for review and consideration. If and when the evaluations reveal particular issues with the teaching faculty, the Program Director meets with the particular faculty more urgently to address those issues and concerns. Additionally, the Program Director meets with the all faculty on an annual basis to review resident feedback and to effect any necessary changes. Common PIF Program Evaluation Describe the approach used for program evaluation. – Residents and faculty annually evaluate the program using the <system>. The electronically submitted evaluation forms are anonymous. Residents and faculty are also encouraged to provide feedback to the Program Director whenever any issue arises or as they see opportunities for improvement. Additionally, residents and faculty participate in an Annual Program Improvement Meeting led by the Program Director. Aggregated data including the most recent ACGME survey results and the resident/faculty program evaluation results are reviewed and used at this meeting to improve the program. The Program Coordinator keeps minutes during this annual meeting and documents any plans to address areas requiring improvement. Action plan progress is monitored and documented by the Program Director. Common PIF Program Evaluation Describe one example how the program used the aggregated results of residents' performance and/or other program evaluation results to improve the program (have the written plan of action available for review by the site visitor). – Annual fellowship retreats including fellows and faculty are conducted each year in May or June (Whenever yours is held) ….the agenda for these meetings includes review of all the aforementioned aggregated data as well an open discussion about educational quality and compliance with the program requirements and suggestions for improving the program for incoming residents. (See GME Website for more information on this and examples of written action plans) – http://med.stanford.edu/gme/progra m_directors/eval_prog.html Common PIF -12 Common PIF Describe the improvement efforts currently undertaken based on feedback from the ACGME Resident Survey. – You MUST review your ACGME Annual Survey each year, bring the results to your Annual Program Retreat, address each area of concern (more than 20%) or any Duty Hour negative response Common PIF DUTY HOURS – Use the summary from your tracking system reporting function – Describe Common PIF Briefly describe how the faculty provides appropriate supervision of residents in patient care activities. – Faculty discusses all patient care history and physical findings with resident. Faculty then confirms both the history and physical findings of the patient. The patient is then discussed in detail, education given regarding the patient illness or complaint, education materials are provided to the resident and finally, a care plan is developed and provided by the resident and confirmed by the faculty. Common PIF Other Questions that may be included in you Common or Specialty PIF Other PIF Questions Skills and Competencies – Describe how residents are informed about their assignments and duties during the residency. This should be in writing and verbal. For example, all fellows are given the program training manual which describes their assignments and duties during orientation to the program. Each fellow signs an attestation that they have received and read the manual. The manual is discussed during orientation with the fellows as a group. Just Answer the Question! Describe how fellows are informed about their assignments and duties during the fellowship. Fellows are informed about their assignments at the beginning of the fellowship. Handouts are provided which describe the goals and objectives for each rotation, as well as a reading list of recommended articles. fellows are informed about their assignments at the beginning of the fellowship. Handouts are provided which describe the goals and objectives for each rotation, as well as a reading list of recommended articles. The Six Core competencies for each rotation include 1. Patient Care 2. Medical Knowledge 3. Practice-based learning & improvement, 4. Interpersonal & Communication Skills 5. Professionalism 6. Systems-based Practices Competencies differ for each rotation and will be provided. Other PIF Questions GENERALIZEGrievance Procedures – Describe how the program handles complaints or concerns the residents raise. The program follows the Dispute Resolution (grievance ) procedures described in the Stanford University Medical Center House Staff Policies & Procedures, pages 20-24, distributed to residents at the beginning of their residency and available on the GME website Other PIF Questions Describe how the program monitors for excessive service and modifies the program accordingly. – The residency has several methods to monitor for excessive service and modify the program if required: Formal rotation evaluations with alerts: Residents complete monthly evaluations using the system that are reviewed and communicated with the rotation directors. Committee on Residency Training and Clinical Services: Each class has resident representatives meet with the Program Director monthly for a formal agenda to review and address issues in the residency program. The rotation directors are sent the residents’ monthly feedback on a quarterly basis (to ensure confidentiality for the residents’ feedback) Yearly, the residents have a retreat in which each aspect of the program is reviewed. Area of concern are brought back to the Committee on Residency Training. The ACGME and GME resident survey questions with respect to excessive service are reviewed. Informal feedback: the residents are encouraged to let the Chief Residents, the Faculty, and the Program Director know immediately when either the service load or the educational activities are compromised. Weekly meetings of the program director and chief residents to discuss the residents and experiences on individual rotations. Computerized entry and monitoring of all hours is performed in <the system>. Duty Hours Duty Hours Duty Hours “Other” SPECIALTY PIF The specialty specific PIF is located on the ACGME website under your Residency Review Committee. – http://acgme.org/acWebsite/navPages/ nav_PDcoord.asp The specialty PIF contains the questions regarding the ACGME core competencies, block diagrams for your program, a narrative description of the program, scholarly activity, and often case logs. Specialty PIF-PracticeBased Learning & Improvement Describe one learning activity in which residents engage to identify strengths, deficiencies, and limits in their knowledge and expertise (self-reflection and selfassessment); set learning and improvement goals; identify and perform appropriate learning activities to achieve self-identified goals (life-long learning). – Programs may use a structured process for reflection in which a faculty advisor guides the resident in using feedback and evaluations to inform the self assessment process. Documentation of the semi-annual evaluation meetings in which this process is demonstrated would provide evidence that this requirement is being addressed. Specialty PIF-PracticeBased Learning & Improvement Describe one example of a learning activity in which residents engage to develop the skills needed to use information technology to locate, appraise, and assimilate evidence from scientific studies and apply it to their patients’ health problems. The description should include: a) locating information, b) using information technology, c) appraising information, d) assimilating evidence information (from scientific studies), e) applying information to patient care. – An appropriate learning activity could be a structured activity such as a journal club presentation, critical appraisal of a topic, or educational prescription with appropriate faculty oversight and formal assessment of skills. Additional documentation would be the written goals and objectives for this learning activity and how residents are assessed. Specialty PIFProfessionalism Describe at least one learning activity, other than lecture, by which residents develop a commitment to carrying out professional responsibilities and an adherence to ethical principles. – This activity should be structured, should demonstrate active faculty involvement (not just passive role modeling) and timely feedback to residents, and should include a mechanism for collecting evaluations (including routine multi-source assessment). Specialty SystemsBased Practice Describe an activity that fulfills the requirement for experiential learning in identifying system errors. – Important elements may include identified faculty to guide the activity, mechanism to ensure active engagement by each resident, and evidence of experiential learning (not just passive presence at conferences and meetings) in which residents participate in identifying a system problem or error and contribute to a potential solution. – Additional documentation could include written goals and objectives for this learning activity and how residents are assessed. Aggregated resident outcomes may be in the form of percentage of residents that completed a patient safety or other Systems-Based Practice project by the end of training, annual list of improvements that resulted from such projects, etc. PIF Briefly describe how the faculty provides appropriate supervision of residents in patient care activities. Remember the new 2011 new supervision guidelines as you answer this question. Table of Contents Common Errors Table of Contents Inaccurate Pages not numbered or not numbered correctly Document not spell-checked Question asked not answered Statistics not added correctly % of time for faculty inconsistent within the PIF Type font does not match Putting It All Together REVISE---One complete PIF (common and specialty) is due to the site visitor at least 14 days before the site visit. For example, Stanford’s GME needs the complete PIF 14-30 days before it is send to the site visitor. (the math means that the program should have the completed document ready 6 weeks BEFORE your site visit) Many large institutions have 3 site visits during a week. This translates to 3 PIFS to review. Help the poor folks out….and be EARLY!!!! The more time you give GME…the more time for editing, comments, and changes. You really cannot have too many eyes look at one PIF!! Save time in the PIF Site Visit/Internal Review Process Leverage Inexpensive Technology Applications e.g., – PDF – PIF Review Functionality – PDF – i-annotate or similar – Microsoft Word Track Changes Documents revisions that are required Electronic copies can easily be uploaded to internal web sites for easy access and documentation Enables team reviews – from “anywhere” Provides Program Directors easy review of recommendations Provides Program Coordinators a streamlined way to make many of the required changes The Land Mines As you write your PIF… remember to read each question carefully. – One example does NOT mean several – “How” does not mean “we do” – No “will’s” – No “see attached – “see below” And now we are going “PIF-less” A new process Somewhat scary – The unknown makes us anxious So this is the Stanford experience from an ACGME pilot of three programs that had PIF-less reviews Documents We Had to Have Ready for the PIF-Less Site Visit Common Program and Institutional Requirements – Policies and procedures for residents’ duty hours and work environment (institutional and program) – Moonlighting policy – Documentation of internal review (date, participants’ titles, type of date collected, and date of review by GMEC) Documents We Had to Have Ready for the PIF-Less Site Visit Common Program Requirements Overall educational goals for the program Competency-based goals and objectives for each experience at each educational level Current Program Letters of Agreement (PLAs) Files of current residents and most recent program graduates containing the following: – Evaluations of residents at the completion of each assignment – Evaluations showing use of multiple evaluators (e.g., faculty, peers, other professional staff, patients, and self-evaluation) – Documentation of residents’ semiannual evaluations of performance with feedback – Final (summative) evaluation of residents, documenting performance during the final period of education and verifying that the resident has demonstrated sufficient competence to enter practice without supervision …..cont’d Documents We Had to Have Ready for the PIF-less Site Visit Completed annual written confidential evaluations of faculty by the residents Completed annual written confidential evaluations of the program by the residents Completed annual written confidential evaluations of the program by the faculty Documentation of program evaluation and written improvement plan Documentation of resident duty hours If applicable, files of current residents who have transferred into the program including documentation of previous experiences and competency-based performance evaluations. If applicable, files of residents who have transferred out of this program into another program, including documentation of experiences and summative competency-based performance evaluations provided to the receiving program director Policy for recruitment, appointment, eligibility, selection, and promotion of residents Documents to Have Ready for the PIF-Less Site Visit New 2011 Common Standards Transfer protocol handover/transfer and sample educational materials related to handovers/ transfers Sample schedules that inform all members of the health care team of attending physicians and residents currently responsible for each patient’s care Document review of policies for supervision and lines of supervisory responsibility Protocols defining common circumstances requiring faculty involvement (care of a complex patient, ICU transfer, DNR or other end of life decision (by year/level of training) Protocol and (completed) sample documents for episodes when residents remain on duty beyond scheduled hours Policies to ensure that residents have adequate rest between daily duty periods and after in-house call (showing differences by level/year of training) Sample documents offering evidence of resident participation in QI and Safety Projects Documents to Have Ready for the PIF-less Site Visit New 2011 Common Standards Added documents for some programs (Site Visitor will clarify) – Review of Case/Procedure logs, if specified in the program requirements – Documentation of conference attendance, if required by the Review Committee – If available, examples of resident involvement in quality – Any additional materials relevant to the review in the specialty (Site Visitor will specify) And now we are going “PIF- less” Lessons learned from a pilot of the first “PIF-less” in August 2011 – Details! – Attention to organization – Team work – Time management And now we are going “PIF-less” We had all of the required elements loaded into our residency management system – Paperless – 3 programs – Site visitors “driving” the data collection by the 2nd day – And it was truly a great experience for ALL!!!!! PIFmanship 101 And that’s all folks… Stay tuned for PIF-lessmanship 102 next year!