End of Shift Milestone Evaluation
Forms
• Michael S. Beeson, M.D., MBA
– Program Director, Akron General
Medical Center
• Amber Bradford, D.O.
– Associate Program Director, Akron
General Medical Center
• Steven Warrington, M.D.
• HAENOW Milestone Study
Group

• Hennepin- Danielle Hart, M.D.
• Akron General- Beeson, Bradford, Warrington
• Einstein- (Beth Israel Albert Einstein College of Medicine) Saadia Akhtar, M.D.
• Nevada- Michael Epter, D.O.
• Ohio State- Diane Gorgas, M.D.
• Washington- Fiona Gallahue, M.D.

• Introduce concept of End of Shift
Milestone Evaluation Forms
• Understand advantages/drawbacks of this method
• Introduce elements of reliability and validity of this method

• Milestone assessment basics
• Development of End of Shift Milestone Evaluation
Forms
• Description of End of Shift Milestone Evaluation
Forms
• Viewing of resident-patient interaction
• Scoring of 1 of 8 EM1 End of Shift Evaluation
Forms
• Discussion of this Methodology
• Concluding remarks

• Assessment
– Objective rather than subjective

• Formative vs Summative assessment

• Focus on Reliability and Validity of methods and tools
– Will different faculty rate the same?
– Is the setting valid?
– Is the assessment instrument valid?
– How is bias controlled?

• IM/Surgery rotations with same senior residents and rounding attending(s)
• EM shifts can be with a different attending
EVERY shift
• For EM, difficult to do an end of month
Milestone evaluation
– An attending may have worked with a specific resident sparingly
– Any assessment becomes a subjective global rating scale

General Milestone Assessment
Basics
• The farther from patient care (real or simulated) an assessment is made, the more it resembles a subjective global ratings scale
• The closer to direct patient care (real or simulated) assessment is made the more objective

Turning the EM Milestones into Global
Rating Scales
• Bias exists from faculty matching
PGY level with expected proficiency level

End of Shift Milestone Evaluation
Forms
• Struggled with how to make an efficient end of shift evaluation form
• SDOT useful, but could take 20-30 minutes to complete
• Desire to develop a brief end of shift assessment tool
– Novel concept- make it useful too!

• Procedure-based (PC9- PC14)
• Medical Knowledge (MK)
• Milestones left:
– Milestone Number by Proficiency Level
Level 1
23
2
26
3
40
4
31
Total
120

Separate Forms for EM1 and EM2-4
• EM1- Use Milestones from Proficiency levels 1-3
– 89 Milestones
• EM2-4 Use Milestones from Proficiency levels 2-4
– 97 Milestones
• 8 EM1 forms and 8 EM2-4 forms
• Proficiency levels of each Milestone are
NOT identified

Sample End of Shift Milestone Form

Sample End of Shift Milestone Form with Anchors

• Milestone evaluation training with individual faculty
• A separate form is used on a weekly basis
– Program Coordinator swaps them out weekly
• Weekly email of that week’s End of Shift Evaluation
Forms along with their anchors to faculty and residents
• Towards end of shift a form is completed by the attending with the resident
• Form is then faxed to Residency Office by ED Unit
Clerk

• View video

• Intern does not introduce himself to the patient
• Intern addresses patient by the wrong name
• Intern was 10 minutes late to his shift and was told to wear his white coat to work the day before
• Intern does not perform a head or neck exam but communicates a normal exam
• Intern does not communicate elevated blood pressure to his attending

• Intern initially prescribes Percocet to a patient with a known alcohol problem
• Intern initially wants to place the patient in an air cast splint for a distal fibular fracture
• Intern initially refers the patient to the medicine clinic for follow-up

Score the End of Shift Evaluation
Form

• Each form is faxed to the residency office
• Program Coordinator inputs form into online database
• Demonstration of Form data input

• Demonstration of Resident Report

• Better balancing of each form in terms of
Proficiency levels
• Which Milestones are just too difficult to evaluate using this method
• Could procedural subcompetencies be evaluated this way?
– Airway, Wound Care, US, Central Line Forms?
• Form completion compliance
• Buy-in by residents and faculty
• Unknown Inter-Rater Reliability

• Available now!
– www.cordtests.org
• Forms can be downloaded as Word document, put your logo on it, whatever…
• iPad app is nearly completed