Site Visit 102: The DIO’s
Role for Program Site
Visits
Ann M. Dohn, M.A.
DIO
Stanford
Disclaimer

No Conflicts of Interest to Report
DIO Competencies



Maintaining the Institution’s
Residency Program’s ACGME
accreditation
Improving the Institution’s
education program
Developing and supporting
residency program directors
DIO Vision

“Your job is not a powerful
one…your job is to empower
others.” i.e. program directors
–
Helen Rantz, 1984
Session Objectives

To understand the role of the
DIO in ACGME program site
visits
–
–
-
Using ACGME annual resident and
faculty surveys
Using internal reviews and annual
program evaluation in preparation
for site visits
Using other potential measures of
program quality
Session Objectives

A time of change…
–
Living in two worlds-starting
7/1/2013


“PIF-less Site visits / NAS
Site visits with PIFs
ACGME
OLD
NEW
Then and Now

1984 site visits vs. 2012 site
visits
Start with the New and
“unknown”

Unknown is scary
New Model: What Do We
Know?




Longer Cycles
Annual WebADS Updates
ACGME Resident and Faculty
Surveys
End of the PIF
End of the PIF

DIOs will have more recreational
reading time!
“New” Model

Effective 7/1/2013
–
“Early adaptors”= RRCs deciding to
go “PIF-less”







Emergency Medicine
Internal Medicine
Neurosurgery
Orthopedic Surgery
Pediatrics
Radiology (Diagnostic)
Urology
–
and their subs
New Accreditation
System (NAS)

The NAS requires categorization
of the ACGME’s Common
Program Requirements to clearly
identify:
–
–
–
Core Requirements
Detail Requirements
Outcome Requirements
New Accreditation
System (NAS)

Core Requirements
–
Statements that define structure,
resource, or process elements
essential to every graduate medical
educational program.




Duty Hours
Sponsoring Institution’s responsibility
for Program Accreditation
Change in program director
Qualifications of the program director
New Accreditation
System (NAS)

Detail Requirements:
–
Statements that describe a specific
structure, resource, or process for
achieving compliance with a Core
Requirement.
Examples of “Detail”
Requirements

Detail Requirements include
particular educational
approaches and learning
experiences


a given number of lectures or hours of
bedside teaching experiences
Ensuring compliance with grievance
and due process procedures as set
forth in the Institutional Requirements
and implemented by the sponsoring
institution
New Accreditation
System (NAS)

Outcome Requirements:
–
–
Statements that specify expected
measurable or observable
attributes (knowledge, abilities,
skills, or attitudes) of residents or
fellows at key stages of their
graduate medical education.
Programs in substantial compliance
with the Outcome Requirements
may utilize alternative or innovative
approaches to meet Core
Requirements.
Example of “Outcome”
Requirements

Outcome requirements include:
–
–
Milestones
Residents must be able to
competently perform all medical,
diagnostic, and surgical procedures
considered essential for the area of
practice.
Common Program
Requirements
Neurosurgery – new
Accreditation Example…
New Model-Tracer ModelInverted Model

Increases value on resident input
–
–

ACGME survey
Resident interviews
Focuses on PROGRAM rather
than a description of the program
–
The Tolstoy novel will no longer
be written!!
New Model-PIF
Questions


Small number of questions from
PIF now in annual WebADS
update
Remaining information assessed
by the field staff
Tracer Method

Allows for the selection of
particular elements to be
evaluated:
–
–
–
–
Citations
Complaints from residents/faculty
Non-compliance reported on the
ACGME resident survey
Duty hour and learning
environment standards
Tracer Method

Focuses on:
–
–
Program’s actual operations
Implementation of policies and
procedures
Tracer Method –
Interactive – A Key Change


Allows for discussion of
important topics related to the
new duty hour standards
Provides opportunities for site
visitors to educate program
leaders and share innovative
practices
Tracer Method

Longitudinal information on:
–
–
–

New standards
Prior citations/corrective actions
Resident survey identified areas of
non-compliance
Areas identified during
preparation for the Site Visit
Tracer Method


Will focus on relevant highpriority issues for the given
program
Planned for two-four areas/topics
per program
–
Done during the regular interview
(no walkarounds)
Tracer Method

Enhancing resident input
–
Residents to directly respond to the
site visitor with a single, confidential
“consensus list”



strengths
areas for improvements
starts with the trainees
PIF-Less Site Visit: The
Stanford Experience






August 2011
Three Programs
Visit done in lieu of mid-cycle
internal review
Data not used for accreditation
Focus was on program
requirements
Used comprehensive data
system (no paper)
PIF-Less Site Visit: The
Stanford Experience (2)




Individuals interviewed = regular
site visit
ACGME provided comprehensive
reports
Extensive debriefing with the site
visitors and program participants
Very positive experience for PDs.
PCs, and the DIO and GME staff
PIF-Less Site Visit:
Stanford Findings

PIF-Less (inverted) site visit
report finding
–
–
Comparable to standard internal
review
Note: our protocol always starts
with the residents/fellows
PIF-Less Site Visit:
ACGME Findings

Findings:
–
–
–
–
–
–
–
Residents feel more engaged
PDs love it
Early ID of Program Themes
Improves continuity and integration
of data
Seamless process for follow-up
Increased opportunity for
reconciliation of discrepant issues
Transformative Model - rather than
additive
PIF-Less Site Visit:
ACGME Findings (2)
Findings Continued
 Comparable to visit with PIF for
citation review
 Superior for review of program
improvements and positive
program elements
 Variability with other elements
–
Dependent on data system used by
the institution/program
Upcoming Changes in
the Site Visit

Broader implementation of the
inverted Site Visit
–
Team site visits (two site visitors)


Programs
Institutions
Yikes! ACGME is
coming….
ACGME Use of Survey
Data during the Site Visit




Use resident data to
validate/clarify program
information
Compare resident data to
program director and faculty
interviews
Validate resident data to
program documentation
Compare duty hours responses
to institutional data
DIO Use of Data for
Program Site Visits




ACGME resident and faculty
surveys
APRs
Internal Reviews
Other measures of program
quality
Annual ACGME Surveys
2012
Annual ACGME Surveys
2009
ACGME Concerns with
the Resident Survey

ACGME concerns with the
resident survey:
–
–
–
–
–
“Honesty” in reporting
? Residents being coached ?
Amount of time spent by residents
completing the survey = 20
seconds per question
Survey questions being sold to
companies who want to ‘help’
programs prepare
Survey fatigue
ACGME Annual Resident
Survey

Possible solutions to using the
survey
–
–
–
2012- Aggregation of individual
survey questions into groupings
2012-use of alternate forms of the
survey
Educating residents about the vital
nature of the survey
Annual Program
Reviews: Summary
Annual Program
Reviews: Data Elements
Internal Reviews
Monitoring Internal
Reviews
Other Possible Sources
of Data





GME Annual House Staff Survey
Duty Hours Reports
Evaluations of programs, faculty,
and house staff
Alumni surveys
Board and in-training exam
scores
GME Annual House Staff
Survey

Provides feedback on eight (8)
key areas of Graduate Medical
Education:
–
–
–
–
–
–
–
–
–
Overall Experience
Program Curriculum
Program Faculty
Evaluation and Feedback
Training Environment
Personal Wellness
Quality Improvements
Communication and Patient
Perceptions
Other areas….(anything we missed)
GME Annual House Staff
Survey

Provides a level of detail not
available from the ACGME
survey
–
–
Such as on call room issues,
fear/intimidation issues,
harassment issues, faculty
Allows for text answers, comments
Warning: Don’t do a survey unless you
are willing to accept and act on the input
GME Annual House Staff
Survey
GME Annual House Staff
Survey
Duty Hour Reports
DIO’s Role

Continuous monitoring of all
programs
–
–
Look at all of the data
Take to GMEC
Tools







ACGME surveys
GME house staff survey
APRs
Duty hour reports
ACGME letters
Internal reviews
Institutional report card
“The Report Card”
Site Visit (or Self Study)
Preparation





Ongoing
Best done continuously
Recommend not “last minute”
Takes time and resources
Needs your best “detail oriented”
staff
Where to start?
Internal
Scheduling/Tracking
Plan ahead






Look at the ACGME schedule
Monitor annually APRs
Monitor annually program
policies
Use the ACGME survey data!!
Use resident/program director
feedback
Update PIFs annually/monitor
annual updates
Truly A Juggling Act!
Teach….Teach…



Monthly program director &
coordinator sessions
Reach out to programs
Assign an education specialist
Faculty development for
program directors (&
coordinators)

Monthly 1 hour meetings
Site visit preparation
PIF writing
GME: Faculty
Development - AY 20112012
Faculty Development

Workshops
–
–
–
–
Fundamentals of curriculum writing
Evaluation
Policies
Feedback
How We Teach

Preparing for a site visit
Accreditation & Maintenance
of Program Quality:
A Continuous Process
Preparing for an ACGME
Site Visit
For Program Directors,
Assistant Directors and
Coordinators
TIMELINE: Preparing for
an ACGME Site Visit

Receive ACGME site visit letter
100-110 days
prior to visit

APR
Annually

Annual ACGME Resident Survey
Jan to May

Annual GME House Staff Survey
March to May
Mid-Cycle Internal Review
– Response to Mid-Cycle Review
between Site Visits
RRC Site visit prep:

Outline approach to PIF

Strategize with PC/GME

Midpoint
3-4 months prior
3-4 months prior

Submit draft to DIO
4-6 weeks prior

Submit final draft to GME/DIO
3-4 weeks prior

Final PIF to site visitor
2 weeks prior

Meet with faculty
Meet with residents/fellows
2-4 weeks prior
2-4 weeks prior

Site Visitor – ACGME
Field Staff

Two types: Field representatives and Specialist Site
Visitors. (Field reps do 1940 of the 2000 annual visits
e.g. 125 visits per year each – Specialist Site Visitors do
approximately 60 visits)
–
Field Representatives are employed by the ACGME, and are
expected to be knowledgeable in the Program Requirements
and on-site review of the medical specialty and subspecialty
areas accredited by the ACGME.

Bios on ACGME web site: 31 individuals: 24 MDs, 6
PhDs, one EdD

Role: produce report that verifies and clarifies the
information provided in the PIF and from the ACGME
resident survey

Will interview program director, faculty,
residents/fellows, and DIO

SITE VISITOR IS A FACT-FINDER for the RRC, NOT
THE DECISION-MAKER
Prerequisites for
Accreditation

I. Institutional Requirements

II. Common Program
Requirements

III. Specialty-Specific Program
Requirements
Read your
requirements…
Understanding
Requirements

Must: Present without fail

Should: Absence justified by
sound rationale

Desirable: Viewed as
enhancement to program
Understanding
Requirements

Must: Duty hours must be
limited to 80 hours per week,
averaged over a 4-week period.

2) Must/Should: Adequate time
for rest and personal activities
must be provided. This must
consist of an 8-hour time period
and between duty periods and
should consist of a 10-hour time
period between duty hour
periods.
Understanding
Requirements

Old common requirements:
–

Assessment should include ….
evaluation by faculty, patients,
peers, self, and other professional
staff.
New common requirements:
–
The program must… use multiple
evaluators (e.g., faculty, peers,
patients, self, and other
professional staff);
Know your
requirements…
Read your PIF…
Program Information
Forms (PIF)

Common PIF
–
–
–
–
–
–
Accreditation information
Sites
Faculty
Resident/Fellow appointments
Evaluation
Duty Hours
Program Information
Forms (PIF)

Specialty Specific PIF
–
–
–
–
–
–
–
–
–
Section for each competency
Duration and scope of education
General institutional information
Physician wellness
Faculty supervision and research
projects
Duty hours and work conditions
Best practices
Comments
Goals and objectives
PIF Strategy for Program
Directors




Download the PIF a year in
advance
Start working on one section at a
time
Do not expect your coordinator,
other faculty or your mother to
write the PIF
You know your program the best
of anyone!
Program Information
Form Preparation (PIF)

Review old PIF, internal review
report, ACGME letters and
responses (if any)

Cannot just update your old PIF
–

PIF format has changed!
Prepare PIF with current
program requirements open side
by side!
Who are we?
The Stanford University Programs
sponsored by Stanford Hospital
and Clinics
How do we explain
ourselves?

A master affiliation agreement
exists between Stanford
University School of Medicine and
Stanford Hospital and Clinics.
–
LPCH has also signed this
agreement but they do not have their
own programs
Master Affiliation
Agreements

GME Department renews
annually with major affiliates
–
–
–
–
VAH
SCVMC
UCSF
Kaiser
Program Letters of
Agreement (PLAs)





Formerly educational
agreements
Program responsibility
Needed between program and
affiliate site director
Must have program competencybased goals and objectives
Must be signed and dated
Single Site Sponsor

Does not apply to us!

We have 87+ programs!
PIF Preparation
Summary









Write the PIF
Clear, concise, in order, no missing
information
Beware of cut and paste errors – can not
always use cut and paste tool
Follow instructions!
Third parties to proof
Must be accurate and concordant with
resident feedback to visitor
Agreements are current, signed and
available
Faculty CVs current, especially scholarly
activity
Coordinate with your DIO (who must
sign) as to when (and where) to mail the
PIF
Common Errors








Document not spell-checked
Questions not answered
Blanks (Medical School, License
Expiration, etc.)
Statistics not added correctly
% of time for faculty inconsistent
within PIF
Pages not numbered
Type font does not match
CVs list more than requested
years of publications
The Questions!!!

Be sure to answer the question
not market your program
Remember




Answer the question concisely
Only provide documentation
requested
Check and double check the PIF
Have someone else who knows
your program read the PIF
Finally

A messy or incomplete PIF can
lead to a disaster

Some of the site visitors are nonMDs and some are MDs from
other specialties.
The PIF is Done, Now
What?




Preparing the Files, Forms,
Policies, and People
Preparation before the big day
The day of the visit
ORGANIZATION!!!
Preparation for the
Day
Site Visitor




Letter from ACGME
Make contact
immediately
Be very
accommodating
Site Visitor will make
specific scheduling
requests
–
–
–

Who should attend
Timeline for the day
Additional information
that should be provided
Offer:
–
–
–
–
Help with travel plans
Parking at SHC
Detailed directions and
location information
Breakfast/coffee/lunch
Preparation for the
Day
Schedule


Date is not usually flexible
All people requested by Site Visitor
must be present
–




Inform him/her immediately if someone is
unavailable
Clear, precise, organized – email to
Site Visitor
Once it’s set don’t change…especially
on the day of the visit
Schedule one conference room for
entire day
Arrange for tours of key facilities as
requested by the site visitor
Site Visit Preparation
Logistics



Schedule Chair, Faculty,
Residents/Fellows (peer selected),
Administrators (including DIO)
Review resident/fellow files (internal
audit): application, educational plans
(goals and objectives), rotation
schedule and call schedules,
conference attendance, evaluations
Group meetings:
–
–
Key faculty: review program
requirements and PIF
Residents/fellows:


Review program requirements and how
you are meeting them.
Review ACGME and GME resident
surveys if available
Preparing the People



Residents/fellows and participating
faculty
Distribute the PIF – should be read
several times prior to the visit
Review:
–
–
–
–
–
–

ACGME Resident Survey if available
(4 or >)
Policies
Evaluation process
ACGME competencies
Goals and objectives
Etc.
Prepare everyone, but don’t tell
them what to say!
Engaging Residents and
Fellows
•Site visitor will want to meet with 1012 (all if small program)
•Must be peer selected, with
representation from all years
•Review requirements, citations,
surveys; fatigue/sleep deprivation
education; duty hours guidelines;
competencies; rotation goals and
objectives; evaluation of residents,
faculty, and program, and core
competencies!
Engaging Residents and
Fellows




“What would you like to see
improved?”
Consider mock session with
other PD/DIO/GME Resident
Representative
Arrange coverage, exclude those
post call, hand off pagers
Site visit is not optimal time to
have new grievances aired for
first time!
Preparation for the
Day
Preparing the
Documentation

Two types:
–
–

Organized
–
–


Resident/Clinical Fellow personnel
files
Program Documentation
No loose paper (binders or folders)
Be able to find things quickly when
requested
Make sure all documentation
matches what was reported in
the PIF!
Prepare one location with all
required documents for review
Preparation for the
Day
Program Documentation







Binders and dividers
Documentation of
Internal Review (not
full report)
Program
educational goals
Competency-based
goals and objectives
(by rotation and
PGY)
Program Letters of
Agreement (PLAs)
Conference
schedules and
attendance
Duty hour reportsCompliance
Summary ONLY


PIF
Program policies,
including:
–
–
–
–

Evaluations
(templates)
–
–
–
–
–
–


Supervision
Duty hours
Moonlighting
Etc.
Of trainees for each
rotation
Semi-annual of trainee
by Program Director
Summative
evaluations of trainee
completing program
Of faculty by trainees
Of program by trainees
Of program by faculty
Rotation and call
schedules
Faculty CVs
Preparation for the
Day
Resident/Clinical Fellow
Files



Personnel files –
one file per trainee
Can organize by
year, type of
document, or both
Transfers: include
copies of
correspondence
between Program
Directors

Include:
–
–
–
Appointment paperwork
(each year)
Evaluations of trainee




–
Divided by competency
and complete set
Final evaluation for
program graduates
360 evaluations
6-month evaluations by
Program Director
Other educational
information, including:








Procedure logs
PowerPoint
presentations
In-service exam scores
Conference attendance
Rotation schedules
Published articles and
other scholarly activities
Records of disciplinary
actions
Moonlighting records
Day of Visit

Conference room
–
–
–
–
–
–

Page/email reminders for all participants
–



1 week prior, day before, and/or morning of Site
Visit
Program Coordinator should participate
–

Private and quiet
Make sure room is clean
Sign on door
Refreshments
Computer and phone access if possible
All binders and files should be in room, ready
upon request (back table)
Additional help for Coordinator to serve as a
“runner”
3 copies of PIF (exactly what was sent)
Stick to schedule – be on time
Again, be as accommodating as possible
1-2 days Prior




Give everyone a copy of the
schedule
Review files with coordinator
Ensure the residency office,
residency conference room,
sleep rooms, are clean and
orderly
Prepare the conference room
–
Do not disturb sign
Site Visit Day

Very early – Check conference room
and ensure all notebooks are in place

Text page everyone 1-2 hours prior

Additional staff standing by

Have everything at fingertips/credenza
behind table (15 second rule)
–
–
–
3 copies of PIF (identical to that sent)
Resident files
Other binders, other information
The RRC Decision



Letter sent within 90 days of the
RRC meeting where your PIF
and Site Visit Findings are
reviewed.
Citations/areas of
noncompliance delineated
Accreditation action
–
–
–
Provisional (new application)
Continued full accreditation
Adverse decisions (subject to
appeal):



Probation (proposed)
Withdraw (proposed)
Withhold (proposed)
Progress Reports and
Letters

RRC may request a response to major
concerns

Use tone of program improvement, not
defensive

Must be reviewed by GMEC and DIO,
sent in triplicate, sent on time

Only send responses to RRC when
requested

Written response to other concerns sent
to GMEC
What can Programs Do
Between Cycles to
Improve?





Annual program review and
follow-up plans for improvement
(a “must”)
Internal review and follow-up
recommendations (also a “must”)
GMEC meetings, program
coordinator meetings, workshops
Program Directors’ Annual
Retreat (Sponsored by GME)
National meetings
Summary



PLAN: Allocate enough time
Call DIO/GME with any questions
Learn (and have others learn):
–
–



The Institutional, Common and Programspecific requirements
The Core Competencies
Write the PIF side by side with the
requirements
Anticipate the unanticipated during the
site visit
Engage residents/fellows in the
process early
Lessons from a
Dinosaur….




The “typical” site visit
How to answer site visitor
questions…..
How to evaluate the site visitor
And the inevitable…..”horror”
stories
Gentle Words of Wisdom

“I’m only a good teacher
because I have made all of the
mistakes before…myself.”
Norman Shumway, M.D.
The Site Visitor

Reach out via email to the site
visitor to welcome
–
–
Ask if they have any special
requests
Announce your availability
The Site Visitor

Answer the questions
–
–
–
Be terse
Be tight
Avoid “we used to …” “we will”
Gentle Words of Wisdom

Keep notes
–
–
–
–
Likes and dislikes of site visitors
Who did what programs
Any correlation between site
visitors and accreditation cycle
Don’t hesitate to call ACGME with
problems!!!!
Horror Stories…..




Oops…sent the wrong version of
the PIF
Oops…forgot to download the
correct version of the PIF
Oops…the Golf Course
And the lavender dress vs. the
golden retriever
Questions

And that’s all folks…
Enjoy!
Questions?


Ann M. Dohn
adohn1@stanford.edu
DIOs Role in Teaching

PIFMANSHIP for
–
–
Program directors
Coordinators
How We Teach

PIF writing
PIFmanship 101
“I see a Site Visit
coming in your
future…”
PIF Writing 101

Gentle words of wisdom…
–
The first impression of your
program for the site visitor will be
your PIF.





Be accurate
Answer the questions (be terse…be
tight)
Have documentation to back up your
answers
Start early-it takes months to write a
good PIF
This is not something your mother,
spouse, best friend, or admin asst. can
do for you!
General Information




Glossary of ACGME terms :
http://www.acgme.org/acWebsite/ab
out/ab_ACGMEglossary.pdf
Examples of “common information”
required in the PIFs we provide to
our Program Directors &
Coordinators:
1) Hospital Statistics:
–
613 Licensed Beds at SHC
–
272 Beds at LPCH


52 Obstetric
220 Pediatric
2. Major Participating
Institutions (Affiliates)






Children's Health Council
[058191]
Kaiser Permanente Medical
Center (Santa Clara) [050571]
Kaiser Santa Teresa [058092]
Lucile Salter Packard Children's
Hospital at Stanford [050572]
San Mateo Medical Center
[050585]
Santa Clara Valley Medical
Center [050438]
3. Library Resources

http://lane.stanford.edu/services/
resourcecore.html
Background

There are actually 2 PIFS to
complete:
–
The Common PIF


–
Addresses the program’s compliance
with the Common Program
Requirements common to all GME
programs.
The common PIF is electronically
generated through the Web
Accreditation Data System (WebADS)
The Specialty PIF


Addresses compliance with the
specialty specific program
requirements
Word document downloaded from the
ACGME Site
Common PIF

Common Program Information Form









www.acgme.org
Use your ACGME assigned User ID &
password
Go to the PIF preparation section (left
hand menu)
Update the Common PIF data.
Most of the data should be updated
annually by your coordinator.
The common PIF contains questions
regarding participating sites, faculty CVs,
evaluation, and duty hours.
Some information is entered only at site
visit time.
Once all of the data is entered and
VALIDATED
Print
Common PIF –1
Common PIF


Data feeds from ACGME
WebADS
TWO Original Signatures
Required
–
–
Program Director
DIO
Common PIF – 2
Common PIF

Respond to previous citations
–
–
Data feeds from ACGME WebADS
If you had a progress report due,
the data remains in WebADS

Automatically populated into your next
PIF
Common PIF

Major Changes are:
–
–
Only since last site visit
Program Format, e.g.,

–
–
–
Have you gone from 3 to 4 years?
Change in resident complement?
Change in Program Leadership?
Change in participating sites?
Common PIF – 3
Common PIF

Participating Sites
–
–
–
Feeds from ACGME WebADS
Stanford is NOT a single limited
site sponsor
Only required sites appear on the
PIF

–
Required means all residents rotate to
that site
Length of rotation

Make sure the length of rotations adds
up to 12 months per year across all
sites
Common PIF - 4
Common PIF

Participating Sites
–
Program Letters of Agreement
(PLAs)


We provide a PLA template on our
GME Website for Programs to use and
educate that at Stanford, e.g., a
separate PLA is required for the
Children’s Hospital, because the
Faculty is Stanford University-i.e..,
The PLA may be between you and
yourself…
Common PIF - 5
Common PIF



Program Director MUST be able to
approve the teaching faculty
The Program Director MUST
evaluate the faculty and approve
continuation as teaching faculty
Program Directors MUST comply
with the university’s or medical
center’s policies on selection,
evaluation, promotion, disciplinary
action and supervision.
–
–
e.g., at Stanford we refer them to: The
GME House staff Policies and
Procedures – on the GME Website
http://med.stanford.edu/gme/policy/
Common PIF

The Program Director MUST be
familiar with ACGME Policies
and Procedures
–
–
–
Institutional Requirements
Common Requirements
Program Specific Requirements
Common PIF

Physician Faculty Roster
–
–
–
List Alphabetically and by Site
Faculty who spend at least 10 hrs.
per week in resident education
Make sure the data matches their
CVs
Common PIF

Faculty Teaching Staff
–
Hours/week devoted to education

–
Should be realistic
Board certification


If double boarded include both boards
If you are sub-specialty program
director or faculty, ensure primary
boards are included
Common PIF- 6
Common PIF

Potential Area for a HUGE
number of citations for
Programs:
–
–
–
Must be totally completed (no
BLANKS)
Ensure certification is valid
Ensure license is current and hasn’t
lapsed
Common PIF
–
Concise Summary of Role in
Program:

<Program Director> oversees the
operations of the entire program;
supervises trainees during patient-care
activities, mentors trainees’
research/projects, provides professional
training, tutorials (lectures and seminars);
monitors duty hour compliance;
coordinates evaluation of courses,
rotations, trainees and faculty.
Common PIF

Selected Bibliography
–
–
–

Strict limit of 10
Strict limit only the last five years “Publications from the last 5 years”
excludes any before 2006…
Publications should not be “in
press” if submitted many years
ago….
If not ABMS certified, explain…
Common PIF - 7
Common PIF - 8
Common PIF

Number of ACGME Approved
Positions
–

Note: If you have a resident making
up a maternity leave, you must ask
ACGME for an extra slot if you are
over your cap
Actively enrolled residents
–
Other than interns, everyone
should have prior years of GME
Common PIF - 9
Common PIF-10
Common PIF

Transferred, Withdrawn or
Dismissed Residents
–
–

Residents who resign are NOT
dismissed
Must have summative evaluation on
transferred trainees
Evaluations
–
–
Yes residents are evaluated following
each learning experience (i.e..
rotation)
Yes- evaluations are documented and
state how


Electronically
Files
Common PIF

Methods of Evaluation
–
Assessment Method




–
Direct Observation
Simulation
OSCE
Etc.
Evaluator




Faculty
Nurses
Program Director
360 (required as of 2007)
Common PIF -11
Common PIF
Evaluation

Describe how evaluators are
educated to use the assessment
methods listed above so that
residents are evaluated fairly and
consistently.
–
The Program Director meets with
evaluators annually before new fellows
start in the training program and reviews,
answers questions and discusses the
core competencies and core competency
performance evaluations used in the
program and assessment methods with
the faculty. The electronic assessment
system and the scales and their use is
also reviewed and discussed in faculty
meetings in order to ensure that faculty
are fully educated and up to date with the
assessment methods and processes.
Common PIF
Evaluation

Describe how residents are informed of
the performance criteria on which they will
be evaluated.
–
At the beginning of each academic year, the
Program Director conducts an orientation to
address several key issues related to the
residency including the performance criteria on
which the residents will be evaluated. During
this orientation session, the Program Director
carefully details the specific evaluation
methods to be used. Both the criteria and
methods are also documented in the residency
handbook. Additionally, at the beginning of
each block rotation, the rotation director meets
with the resident and carefully delineates the
expectations and performance criteria on
which the resident will be evaluated for that
specific rotation.
Common PIF
Evaluation

Describe the system to ensure that
faculty complete written evaluations of
residents in a timely manner following
each rotation or educational
experience.
–
At the end of each block rotation, our
online resident data collection and
tracking system, MedHub, sends an
automated reminder to the service
attending(s) to evaluate the resident(s) on
that particular rotation. Using MedHub, the
Program Coordinator tracks pending
evaluations and follows up with faculty as
needed to urge them to complete their
evaluations on time. In the unlikely event
that a faculty member is unresponsive to
the coordinator's requests to complete
evaluations, the Program Director
contacts the faculty member and requests
him/her to complete the evaluations.
Common PIF
Evaluation

Describe the process used to complete and
document written semiannual resident
evaluations, including the mechanism for
reviewing results (e.g., who meets with the
residents and how the results are documented in
resident files).
–
Residence performance is evaluated by the teaching
faculty at the conclusion of each rotation using an
electronic evaluation form (MedHub). Additionally,
each faculty mentor meets with their resident advisee
quarterly and documents a summary of the meeting
to be placed in the resident’s file. The Program
Director meets with each resident on an individual
basis semiannually to review the accumulated
performance evaluations and mentor notes, provide
feedback, and update the resident’s learning plan as
appropriate. A summary of these meetings is
documented and placed in the resident’s file. The
residents are free to review the contents of their
records at any time.
Common PIF
Evaluation

Describe the system used by the residents to
provide annual confidential written evaluations of
the teaching faculty (have examples and forms
available for review by site visitor).
–
–
Residents annually confidentially evaluate the
teaching faculty using <system>. The electronically
submitted <system>. evaluation forms are
anonymous. Paper copies of completed evaluations
without personal identifiers are printed by the
Program Coordinator, reviewed by the Program
Director, and presented to individual faculty members
for review and consideration.
Written evaluations are completed electronically and
anonymously by using a centralized system used
throughout the hospital to provide annual aggregated
data. Fellows are sent e-mail reminders about
completing evaluations in a timely manner by the
Fellowship Coordinator. The program coordinator
follows up, aggregates the results and forwards them
to the Program Director. The Program Director
reviews the results and takes any actions appropriate
with individual program faculty.
Common PIF
Evaluation

Describe the program's (or Department's, if
applicable) system for evaluating and
providing feedback to the teaching faculty.
–
Residents annually evaluate the teaching
faculty using the <system>. The electronically
submitted evaluation forms are anonymous.
Paper copies of completed evaluations without
personal identifiers are printed by the Program
Coordinator, reviewed by the Program Director,
and presented to individual faculty members
for review and consideration. If and when the
evaluations reveal particular issues with the
teaching faculty, the Program Director meets
with the particular faculty more urgently to
address those issues and concerns.
Additionally, the Program Director meets with
the all faculty on an annual basis to review
resident feedback and to effect any necessary
changes.
Common PIF
Program Evaluation

Describe the approach used for program
evaluation.
–
Residents and faculty annually evaluate the
program using the <system>. The
electronically submitted evaluation forms are
anonymous. Residents and faculty are also
encouraged to provide feedback to the
Program Director whenever any issue arises or
as they see opportunities for improvement.
Additionally, residents and faculty participate in
an Annual Program Improvement Meeting led
by the Program Director. Aggregated data
including the most recent ACGME survey
results and the resident/faculty program
evaluation results are reviewed and used at
this meeting to improve the program. The
Program Coordinator keeps minutes during this
annual meeting and documents any plans to
address areas requiring improvement. Action
plan progress is monitored and documented by
the Program Director.
Common PIF
Program Evaluation

Describe one example how the program
used the aggregated results of residents'
performance and/or other program
evaluation results to improve the
program (have the written plan of action
available for review by the site visitor).
–
–
Annual fellowship retreats including fellows
and faculty are conducted each year in May
or June (Whenever yours is held) ….the
agenda for these meetings includes review of
all the aforementioned aggregated data as
well an open discussion about educational
quality and compliance with the program
requirements and suggestions for improving
the program for incoming residents. (See
GME Website for more information on this
and examples of written action plans)
http://med.stanford.edu/gme/program_directo
rs/eval_prog.html
Common PIF -12
Common PIF

Describe the improvement
efforts currently undertaken
based on feedback from the
ACGME Resident Survey.
–
You MUST review your ACGME
Annual Survey each year, bring the
results to your Annual Program
Retreat, address each area of
concern (more than 20%) or any
Duty Hour negative response
Common PIF

DUTY HOURS
–
–
Use the summary from your
tracking system reporting function
Describe
Common PIF

Briefly describe how the faculty
provides appropriate supervision of
residents in patient care activities.
–
Faculty discusses all patient care history
and physical findings with resident.
Faculty then confirms both the history
and physical findings of the patient. The
patient is then discussed in detail,
education given regarding the patient
illness or complaint, education materials
are provided to the resident and finally, a
care plan is developed and provided by
the resident and confirmed by the
faculty.
Common PIF

Other Questions that may be
included in you Common or
Specialty PIF
Other PIF Questions

Skills and Competencies
–
Describe how residents are
informed about their assignments
and duties during the residency.

This should be in writing and verbal.
For example, all fellows are given the
program training manual which
describes their assignments and duties
during orientation to the program. Each
fellow signs an attestation that they
have received and read the manual.
The manual is discussed during
orientation with the fellows as a group.
Other PIF Questions


GENERALIZEGrievance Procedures
–
Describe how the program handles
complaints or concerns the
residents raise.

The program follows the Dispute
Resolution (grievance ) procedures
described in the Stanford University
Medical Center House Staff Policies &
Procedures, pages 20-24, distributed to
residents at the beginning of their
residency and available on the GME
website
Other PIF Questions

Describe how the program monitors for excessive
service and modifies the program accordingly.
–
The residency has several methods to monitor for
excessive service and modify the program if required:







Formal rotation evaluations with alerts: Residents complete
monthly evaluations using the system that are reviewed and
communicated with the rotation directors.
Committee on Residency Training and Clinical Services: Each
class has resident representatives meet with the Program
Director monthly for a formal agenda to review and address
issues in the residency program.
The rotation directors are sent the residents’ monthly feedback
on a quarterly basis (to ensure confidentiality for the residents’
feedback)
Yearly, the residents have a retreat in which each aspect of the
program is reviewed. Area of concern are brought back to the
Committee on Residency Training. The ACGME and GME
resident survey questions with respect to excessive service are
reviewed.
Informal feedback: the residents are encouraged to let the
Chief Residents, the Faculty, and the Program Director know
immediately when either the service load or the educational
activities are compromised.
Weekly meetings of the program director and chief residents to
discuss the residents and experiences on individual rotations.
Computerized entry and monitoring of all hours is performed in
MedHub.
SPECIALTY PIF

The specialty specific PIF is located
on the ACGME website under your
Residency Review Committee.
–

http://acgme.org/acWebsite/navPages/
nav_PDcoord.asp
The specialty PIF contains the
questions regarding the ACGME
core competencies, block diagrams
for your program, a narrative
description of the program,
scholarly activity, and often case
logs.
Specialty PIF-PracticeBased Learning &
Improvement

Describe one learning activity in
which residents engage to identify
strengths, deficiencies, and limits in
their knowledge and expertise (selfreflection and self-assessment); set
learning and improvement goals;
identify and perform appropriate
learning activities to achieve selfidentified goals (life-long learning).
–
Programs may use a structured process
for reflection in which a faculty advisor
guides the resident in using feedback
and evaluations to inform the self
assessment process. Documentation of
the semi-annual evaluation meetings in
which this process is demonstrated
would provide evidence that this
requirement is being addressed.
Specialty PIF-PracticeBased Learning &
Improvement

Describe one example of a learning
activity in which residents engage to
develop the skills needed to use
information technology to locate,
appraise, and assimilate evidence from
scientific studies and apply it to their
patients’ health problems. The description
should include: a) locating information, b)
using information technology, c)
appraising information, d) assimilating
evidence information (from scientific
studies), e) applying information to patient
care.
–
An appropriate learning activity could be a
structured activity such as a journal club
presentation, critical appraisal of a topic, or
educational prescription with appropriate
faculty oversight and formal assessment of
skills. Additional documentation would be the
written goals and objectives for this learning
activity and how residents are assessed.
Specialty PIFProfessionalism

Describe at least one learning
activity, other than lecture, by which
residents develop a commitment to
carrying out professional
responsibilities and an adherence
to ethical principles.
–
This activity should be structured,
should demonstrate active faculty
involvement (not just passive role
modeling) and timely feedback to
residents, and should include a
mechanism for collecting evaluations
(including routine multi-source
assessment).
Specialty Systems-Based
Practice

Describe an activity that fulfills the
requirement for experiential learning in
identifying system errors.
–
–
Important elements may include identified
faculty to guide the activity, mechanism to
ensure active engagement by each resident,
and evidence of experiential learning (not just
passive presence at conferences and
meetings) in which residents participate in
identifying a system problem or error and
contribute to a potential solution.
Additional documentation could include
written goals and objectives for this learning
activity and how residents are assessed.
Aggregated resident outcomes may be in the
form of percentage of residents that
completed a patient safety or other SystemsBased Practice project by the end of training,
annual list of improvements that resulted from
such projects, etc.
PIF

UPDATE-REVISE
Briefly describe how the faculty
provides appropriate supervision of
residents in patient care activities.
–
Faculty discusses all patient care history
and physical findings with resident.
Faculty then confirms both the history
and physical findings of the patient. The
patient is then discussed in detail,
education given regarding the patient
illness or complaint, education materials
are provided to the resident and finally, a
care plan is developed and provided by
the resident and confirmed by the
faculty. Face-to-face feedback
regarding patient care is given as
quarterly evaluation by all faculty.
Table of Contents
Common Errors







Table of Contents Inaccurate
Pages not numbered or not
numbered correctly
Document not spell-checked
Question asked not answered
Statistics not added correctly
% of time for faculty inconsistent
within the PIF
Type font does not match
Putting It All Together





REVISE---One complete PIF (common
and specialty) is due to the site visitor at
least 14 days before the site visit.
Stanford’s GME needs the complete PIF
14-30 days before it is send to the site
visitor. (the math means that you should
have the completed document ready 6
weeks BEFORE your site visit)
Stanford’s GME usually has 3 site visits
during a week. This translates to 3 PIFS
to review. Help the poor folks out….and
be EARLY!!!!
The more time you give GME…the more
time for editing, comments, and changes.
You really cannot have too many eyes
look at one PIF!!
Save time in the PIF Site
Visit/Internal Review
Process

Leverage Inexpensive
Technology Applications e.g.,
–
–
PDF – PIF Review Functionality
Microsoft Word Track Changes





Documents revisions that are required
Electronic copies can easily be
uploaded to internal web sites for easy
access and documentation
Enables team reviews – from
“anywhere”
Provides Program Directors easy
review of recommendations
Provides Program Coordinators a
streamlined way to make many of the
required changes
The Land Mines

As you write your PIF…
remember to read the question
carefully.
–
–
–
–
One example does NOT mean
several
“How” does not mean “we do”
No “will’s”
No “see attached – “see below”