Orientation, Milestones, and
Initial Assessment of New Interns
Webinar Presentation (June 11, 2014): Cindy Works MD, Brian Veauthier MD
Content: Brian Veauthier MD, Beth Robitaille MD
University of Wyoming Family Medicine Residency - Casper
Objectives
• Review benefits of evaluating new interns
including ACGME Requirements
• Describe a process we started to evaluate our
new interns based on the Family Medicine
Milestones
• Share our successes and pitfalls
• Present a case study of a current intern
• Open the floor for interactive discussion
Why Evaluate New Interns?
• Because the ACGME
says we need to
• Avoid the December
shock factor of a poorly
performing resident
• Variance in medical
school education assess strengths and
weaknesses
What does the ACGME require?
V.A.2.b) The program must:
V.A.2.b).(1) provide objective assessments of competence in patient care
and procedural skills, medical knowledge, practice-based learning and
improvement, interpersonal and communication skills, professionalism, and
systems-based practice based on the specialty-specific Milestones; (Core)
V.A.2.b).(1).(a) This assessment must involve direct observation of
resident-patient encounters. (Detail)
V.A.2.b).(1).(b) Each resident should be
assessed in each of the six competency
areas on entrance into the program.
(Detail)
Our Process
CCC Committee
met 2 weeks
after orientation
• Each new intern evaluated using Beta Testing
Milestone Form
• Gave recommendations for areas of concern
CCC Committee
shared results
with faculty
• Further input given for evaluations
• IEPs finalized
Faculty advisors
met individually
with interns
• Reviewed the Milestones in general
• Provided IEPs where appropriate
The Tools
General Interaction
During Orientation
USMLE/COMLEX Scores
Milestone
Evaluation
Mock In-training Exam
OSCE Cases
The Tools
General Interaction
During Orientation
• Useful to evaluate some of
the Milestones under
Professionalism and
Communication (Prof-2;
Com-3)
The Tools
USMLE/COMLEX Scores
• Another piece of data for
evaluating Medical Knowledge
• Poor test takers tend to be
poor test takers
The Tools
Mock In-training
Exam
• Give complete ITE exam
from previous year
• Score exam compared to
previous intern class
• Helps with MK-1
The Tools
OSCE Cases
• Use one afternoon and rotate
the interns through eight
OSCE cases - 30 minutes per
case
• Conducted by core faculty
The Tools
OSCE 1-CHEST PAIN CASE
• PATIENT CARE
• PC-1 Care of acutely ill
patient
• MEDICAL KNOWLEDGE
• MK-2 Applies critical thinking
skills
• SYSTEMS-BASED PRACTICE
• SBP-2 Emphasizes patient
safety
The Tools
OSCE 2-DIABETES CHRONIC
CARE
• PATIENT CARE
• PC-2 Care for patient with
chronic illness
• MEDICAL KNOWLEDGE
• MK-2 Applies critical thinking
skills
• SYSTEMS-BASED PRACTICE
• SBP-1 Provides cost conscious
care
• SBP- 4 Coordinates team based
care
The Tools
OSCE 3-DEPRESSED
PATIENT
• PATIENT CARE
• PC-1 Cares for acutely ill
• MEDICAL KNOWLEDGE
• MK-2 Applies critical
thinking skills
• SYSTEMS-BASED PRACTICE
• SBP-4 Coordinates team
based care
The Tools
OSCE 4-OB/INFANT
FEVER CASE
• PATIENT CARE
• PC-1 Care of acutely
ill patient
• MEDICAL KNOWLEDGE
• MK-2 Critical thinking
skills
The Tools
OSCE 5-WELL WOMAN CASE
• PATIENT CARE
• PC-3 Disease prevention and
health promotion
• MEDICAL KNOWLEDGE
• MK-2 Applies critical thinking
• SYSTEMS-BASED PRACTICE
• SBP-2 Emphasizes patient
safety
• SBP-4 Coordinates team
based care
The Tools
OSCE 6-SUTURING
CASE
• PATIENT CARE
• PC-5 Procedures
The Tools
OSCE 7-X-RAYS
AND ECGS
• MEDICAL KNOWLEDGE
• MK-2 Applies critical
thinking
The Tools
OSCE 8-MEDICAL
INFORMATION
MASTERY
• PRACTICE BASED
LEARNING AND
IMPROVEMENT
• PBLI-1 Locates,
appraises, and
assimilates evidence
from scientific studies
related to their patients’
health problems
Sample Evaluations
Intern A “The Rock Star"
Intern A
C3 colleagues
PROF2 conduct
PBLI 1 evidence
SBP4 team
SBP2 safety
SBP1 cost
MK2 critical
MK1 board
PC5 Proc
PC3 Prevent
PC2 chronic
PC1 acute
0
1
2
3
4
5
Intern AComments/Recommendations
“Excellent test scores 440. Mean from last year
386 with std 74”
“Professionally appropriate interaction”
“Faculty advisor feels like she won the lottery”
Intern B “Should be OK"
Intern B
C3 colleagues
PROF2 conduct
PBLI 1 evidence
SBP4 team
SBP2 safety
SBP1 cost
MK2 critical
MK1 board
PC5 Proc
PC3 Prevent
PC2 chronic
PC1 acute
0
1
2
3
4
5
Intern BComments/Recommendations
“Would benefit from showing more confidence”
“Needs significant work on suture skills”
“Performed lower than expected on the mock
ITE. Score 300 . Mean from 2012- 386 with std
74. consider early IEP”
“Professional interactions”
“Very quiet and not interactive during
orientation-is she fitting in ok?”
Intern C “Help"
Intern C
C3 colleagues
PROF2 conduct
PBLI 1 evidence
SBP4 team
SBP2 safety
SBP1 cost
MK2 critical
MK1 board
PC5 Proc
PC3 Prevent
PC2 chronic
PC1 acute
0
1
2
3
4
5
Intern CComments/Recommendations
“Give literature on common issues that is clinical”
“Give specific info on common issues”
“Needs organized approach to EKG- Wicks handout”
“Scored 220 on mock ITE- mean last year was 386
with STD 74. Strongly recommend early IEP”
“Has demonstrated appropriate professional
interaction so far”
“Some concerns with language barriers”
“Interacting well with peers”
INTERN C
MILESTONE IMPLEMENTATION INTO AN
INDIVIDUAL EDUCATION PLAN
BIG REVEAL: ORIENTATION
MILESTONE EVALUATION
• Significant deficits in her medical education leading to
obvious need for intervention
• Milestone deficiencies most obvious in:
• Medical Knowledge
• Patient Care
• Communication
JULY: ROUTINE ADVISOR MEET &
GREET
• Reviewed Milestone deficits
• Outlined her initial improvement plan/IEP
• Outlined close monitoring and follow-up
OCTOBER: ROUTINE ADVISOR
MEETING AND EVALUATION
• Some progress noted, but insufficient to become
commensurate with her peers
• Confirmed continued deficits and need for formal
intervention and remediation IEP
• General consensus: she was functioning at early MS-4
level
OCTOBER: ADVISOR & PROGRAM
DIRECTOR MEETING
• Intern C formally notified of deficiencies
• Remediation IEP developed based on the Milestones,
concentrating on the most relevant competencies:
• Medical Knowledge
• Patient Care
• Communication
• Practice-based Learning and Improvement
• Specific goals: to progress from Level 0 to Level 1 by
January in the identified sub-competencies
• Key away rotation of inpatient pediatrics in the interim
JANUARY: ADVISOR & PROGRAM
DIRECTOR MEETING
• Intern C reassessed on the Milestones
• Core program rotational evaluations
• Video evaluation
• Field notes
• ITE
• Verbal feedback from away pediatric rotation attending
• Senior preparedness OSCE
• Demonstrated improvement in several areas but did not
reach goal of Level 1 on all identified sub-competencies
• Key float rotation in February/March provided next
opportunity to achieve the progression required to
proceed on as a PGY-2: progress towards Level 2 on
identified sub-competencies
MARCH: ADVISOR & PROGRAM
DIRECTOR MEETING
• Intern C reassessed on the Milestones
• Field notes
• Evaluations
• Verbal feedback from faculty
• Written feedback from senior float resident
• Other IEP requirements (ITE and board preparation)
• Demonstrated enough continued progress in the identified
sub-competencies to continue at our residency
• Remains on remediation IEP
MEDICAL KNOWLEDGE
2
1.5
ORIENTATION
JANUARY
MARCH
1
0.5
0
MK-1 BREADTH/DEPTH
MK-2 APPLY TO PT
CARE
PATIENT CARE
2
1.5
1
0.5
0
ORIENTATION
JANUARY
MARCH
COMMUNICATION
2
1.5
1
0.5
0
ORIENTATION
JANUARY
MARCH
PRACTICE-BASED LEARNING AND
IMPROVEMENT
2
1.5
ORIENTATION
JANUARY
MARCH
1
0.5
0
PBL-1 EBP
PBLI-2 SELF LRNING
CONCLUSIONS
• Orientation Milestone evaluation via OSCE and mock ITE
helped to identify deficiencies early leading to early
implementation of improvement plan
• Monitoring Milestone progression throughout the year
helped to define specific and objective deficiencies
• The Milestone based IEP provided the appropriate detail
and language to set specific goals and plans of action
• The Milestones played an important role in this successful
(although ongoing) remediation
Positives
How did it work?
Met ACGME
requirement for
evaluating new
interns
Introduced the
new interns to
the Milestones
Unable to
evaluate certain
Sub-competency
and Milestones
Time consumingat least 30
minutes per
intern
Negatives
Helped identify a
weak intern and
start early
remediation
Only a snapshot
of the intern
How did it work?
Practical Points
• Evaluate all new interns one
question at a time
• Try to have all the data from
the tools easily accessible and
organized
How did it work?
Areas for Improvement/Changes
• Hopefully use New Innovations for
Milestone evals with OSCEs mapped
• Better organization of data from tools
• Continue to “learn” the Milestones
• Continue to “fine tune” OSCEs and
evaluation tools
Summary
Overall found the
process valuable and will
continue the process
• Meets ACGME requirement for
evaluating incoming interns
• Mechanism for introducing the
Milestones
• Helped us identify a very weak
resident early
Discussion