ColleaguesPlease find the latest draft of my proposal for ESCape tomorrow. I am sorry for this being late but it has
changed significantly over the weekend after input from Pat O on Friday. I am looking forward to
presenting to you all tomorrow.
There are several areas I would most appreciate feedback (although any ideas are welcome).
1- What should be the balance of resident driven assessment and faculty driven assessment for
entrustment of trainees in GME.
2- How to best assess the implementation of this system (ie: process measurements- completion rates,
time until completion vs. outcomes measures- satisfaction, EPAs vs. milestones for assessment)
3- Amount of time needed to ask a data management consultant to be involved to help us prepare and
collect appropriate assessment data
Thanks again!
Jeff
Creation of a learner driven GME EPA assessment network for internal medicine residency and
fellowship programs.
Background
Assessment of graduate medical education trainees is a high priority of training programs. Training
program directors must be able to certify that graduates are able to practice independently in their
respective specialty/subspecialty. Various frameworks have arisen to guide how to assess and report
the competence of trainees. Increasingly, professional milestones are used by program directors to
ensure that trainees are ready for independent practice
In the case of internal medicine, both the body accrediting programs (ACGME) and the organization
certifying individuals (American Board of Internal Medicine (ABIM)) have had major roles in structuring
how program directors assess trainees in medicine specialties and subspecialties. Their philosophy has
evolved over time to include greater proscription of this assessment. This evolution began in 1999 when
the ACGME introduced the six core competencies (patient care, medical knowledge, systems-based
practice, practice-based learning, communication and professionalism). This began a transition of
philosophy of accreditation from process to training outcomes. The recent introduction of the Next
Accreditation System (NAS) evolved these concepts to include milestones as subsections of the core
competencies. The internal medicine community developed 142 educational milestones, now termed
curricular milestones, to further describe the knowledge, skills and attitudes of observable development
from a novice to a proficient trainee. The developmental milestones were recognized as being too
extensive to be effectively implemented or evaluated. A second effort produced the reporting
milestones, a narrative based set of 22 milestones. The 6 ACGME competency domains organize these
22 milestones such that each competency now has 2 to 5 subcompetencies. (2)
An alternative approach to assessment involves using entrustable professional activities (EPAs) as a
framework for assessing competence. EPAs reflect the degree of mastery of professional tasks that
together constitute the work of a profession. (3) Carrying EPAs requires abilities in multiple
competencies. This approach recognizes that professional tasks are complex and breaking those tasks
down to discreet, measurable steps, such as might occur with a milestone based approach, loses the
ability to provide a holistic view of the successful acquisition of professional skills. These EPAs are best
evaluated by observation of professional skills by experienced faculty members.
Current UCSF Department of Medicine Assessment Strategy:
As required by the ACGME and the ABIM, all Department of Medicine (DOM) ACGME training programs
assess the specialty/subspecialty specific milestones; the reporting milestones are recorded in the
Evalue platform system. Programs may deploy different tools (global assessments, procedure
observations, 360’s, chart stimulated recall, qualitative comments) to assess their program specific
milestones but most programs depend upon observational methods to record the initial milestone
assessment. Clinical competency committees (CCCs) then meet to determine the summative milestone
score that is reported to the ACGME and the ABIM. The CCCs are largely dependent upon the
milestone scores obtained from direct faculty observations.
Analysis of Current Milestone Assessment Strategy
Program CCCs are reporting issues with their newly implemented milestone assessment processes that
have been in effect since AY2013-14 for the internal medicine residency program and AY2014-15 for the
fellowship programs. First, compliance with completing the assessments is poor; across the DOM
training programs, the overall compliance of assigned evaluations is initially only 50%.. Much
administrative time is spent trying to get completed evaluations to the required 70% level and often
program leadership has to step in to improve faculty compliance. Reasons faculty cite for their
challenges in completing assessments is “assessment fatigue” caused by multiple end of the month
evaluations being due simultaneously, the need to assess as many as 22 milestones as well as overall
frustration with the Evalue platform.
Second, the quality of the assessments is also in question. CCCs are now often basing their judgments
too few numeric scores on each milestone category bringing their validity into question. This has led
CCCs to have concerns about whether they are fully able to assess and report trainee’s performances.
Similarly, it creates doubt that CCCs are able to effectively identify areas of improvement for advisors to
help trainees.
In response to these concerns about a milestone assessment based system and wanting to improve
trainee assessment, the internal medicine residency program assessment leadership team began
exploring using an EPA based assessment system in 2013.
In 2013-4 we utilized a Delphi method to implement a group of 8 EPAs to help make our assessments
more activity specific. (6) The EPAs are housed on the UCSFEPA.org website and contain the following
assessments:
•Serious Illness Communication EPA
•Rounds Observation EPA
•Discharge summary EPA
•4 habits clinic survey
•Clinic Post-Discharge EPA
•Acute Care EPA
•Code Leadership EPA
•Multidisciplinary Rounds Inpatient Feedback
These assessments are recorded utilizing Qualtrics through the UCSF MyAccess website and are
accessed by the faculty to generate evaluations. Several internal medicine fellowships (Palliative Care,
Geriatrics and Hematology/Oncology) have begun to use EPAS and the UCSFEPA.org platform for
assessment of their trainees. Each of these EPAs can be mapped to several milestones at the same time
as is shown in the following chart from proposed gynecology residency EPAs.
Benefits of using EPAs includes improved overall compliance with obtaining assessments due to trainee
initiation of the assessment, real time assessment by the faculty and ease with utilizing the Qualtrics
platform. Although not formally compared, there is a sense that the quality of the EPA assessments
may be better than the milestone based assessments.
Although these, and future EPAs, consist of a robust network of evaluations, we have been unable to
fully implement them due to several limitations with our assessment technology and expertise. The
GME office is now able to address these challenges by adding administrative leadership and data
management assistance. This help creates a firm infrastructure but to create a trainee centered,
interactive assessment network we will need additional help from Information Technology (IT) to create
a more seamless platform.
Past Assessment Efforts:
Over the last four years we have made a concerted effort to bring a coordinated IT effort to enable
utilizing our EPA network. Our initial efforts utilized the Mahara portfolio platform. Conceptually this
system offered several advantages including resident control over their assessments on their own
customizable webpage. Material could be collected and correlated for review with trusted advisors and
create a framework for individual assessment that could highlight a trainee’s skills. Unfortunately,
residents did not utilize this tool for several reasons. First, residents were too busy to log on to the
system to record evaluations. Second, they did not feel empowered to ask for completion of
evaluations. Third, since the Mahara system was completely resident driven there was no way to
reliably track resident completion of educational tasks.
Efforts to move Myfolio system through the Evalue platform were similarly unsuccessful due to data
management problems and the inability to design an appropriate single log-in access for faculty or
residents to review. Also, residents did not have ability to control their assessment information to pick
their best work to present to the CCC.
Last year we moved our EPAs to the Qualtrics survey platform and housed our EPAs on a website
created through consultation with Eric Widera MD, from the department of geriatrics and palliative
care. This system created a much easier system to collect EPA data, but currently our faculty completion
rates remain low as there is are no automatic reminder systems to help busy clinicians to remember to
complete these evaluations. The EPAs that are most successful have a faculty champion, such as
Heather Whelan MD, a hospitalist at the VAMC. Each month she circulates a meeting list to our ward
interns to schedule time to complete this EPA. This activity has created popular feedback for our interns
and is quite well received by our housestaff. This individual effort is time intensive, however, and
unlikely to be sustainable leaving our EPA system longevity in doubt.
During this time we began the process of seeking more information from our trainees about what their
goals are for their assessment process by doing focus groups with convenience samples of residents.
Focus Group Themes:
Eight focus groups in our internal medicine residency (and fellowship) programs over the last three years
involving approximately one-hundred trainees have demonstrated several themes regarding our
assessments. These themes were derived from our focus group notes and match concerns voiced by
resident trainees across the country when there is inadequate supervision or feedback. (4-5)
First, residency assessment that depends on summative feedback from end of rotation, milestone-based
assessments, lack specificity to enable residents to improve their professional practice. Residents would
prefer more timely, qualitative, in-person and practice based feedback to help them grow
professionally.
Second, currently used assessment tools create scores and comments that are difficult for residents to
understand or access to help them follow their progress.
Third, with the advent of the Clinical Competency Committee (CCC) mandated by the ACGME, residents
feel there is a lack of transparency or control over their own assessments leading to mistrust of the
evaluation system.
Finally, the current culture in our residency and fellowship programs does not facilitate constant, clear
and continuous feedback from faculty to trainees or from trainees to faculty. Residents feel this acutely
and do not feel they have permission to give constructive feedback to faculty and often feel criticized
when receiving anything less than superlative feedback from faculty.
Current Pilot Program:
With the help of Holly Nishimura BA, and Christy Boscardin PhD, fifty two of our internal medicine Prime
residents are starting a two month evaluation project where they actively solicit EPA activity based
feedback from faculty or other professional staff. These evaluations will be completely resident driven
and will be collected in Qualtrics. Each participating resident was assigned a unique code number by
Amy Forseth BA, our division administrator. She then emailed this individual code to each participating
resident. The email code is unique to each participant and is easily forwarded to observing faculty and
staff. The forwarded email automatically links the observing faculty member to the EPA survey menu
where they can easily choose the EPA they observed and quickly complete the evaluation via email. The
data are then captured for that individual resident. This “proof of concept pilot” enables us to examine
the rates of completion of EPAs, willingness of residents to participate in the process and their
satisfaction with the feedback they receive.
Although this pilot is an exciting advance, it only fulfills our goals of facilitating timely resident feedback,
it does not address our goals of creating an accessible resident dashboard where they are able to drive
their own assessments and highlight their best work for the CCC. It also only takes a small incremental
step towards moving our assessment and feedback culture forward to be more open and meaningful.
Proposal Goals:
The goal of our proposal is to address the four listed focus group themes by restructuring our
assessment system over the next five years to be primarily trainee driven and to improve our
information system so that all evaluations will be available to, and controlled by, the residents who
receive them. Our vision is that every trainee entering the internal medicine residency or subspecialty
fellowship programs would be given an assessment “dashboard” that would collect and store
information that demonstrates their progress towards independent practice throughout their training.
Data would be collected through a number of sources; Evalue, EPAs from Qualtrics, inservice exam
scores, clinic patient dashboards and resident portfolio data and will be mapped to milestones to
facilitate reporting to the ACGME. Trainees will be able to review all their data and will be able to
“release” their best work to demonstrate their entrustability on the milestones. This trainee assessment
dashboard will use the same Qlikview platform used by the medical school.
Therefore our project research question is: Is restructure of competency assessment to a trainee-driven
process associated with improved understanding of resident’s preparedness for independent practice
amongst UCSF GME trainees?
Process Proposal:
Create a resident-driven, EPA based evaluation network that will expand following the previously
published Delphi-derived EPAs for internal medicine and medicine subspecialty fellowships. The goals of
the project will use the focus group themes to create a learner driven evaluation network with a goal of
addressing each of the four themes.
Theme #1: To create more timely opportunities for feedback the residents will be asked to solicit
evaluations from their observing faculty. Trainees will send a uniquely coded email to the faculty
member that requests feedback on the EPA activity they want assessed. Residents will be required to
complete at least 3 EPAs per activity per year utilizing different faculty assessors, but they can choose
which of the scores and assessments they release to the CCC for final review.
Theme #2: Accessibility to their feedback will be ensured through creation of a resident assessment
home page for each resident in the program. The homepage will contain sections that organize and
display summative data from Evalue, EPA feedback evaluations, in service exam scores and a resident
portfolio that allow the resident to track and record their clinical and academic progress by uploading
documents that support mastery of individual milestones. Residents cannot edit this material but can
add annotations if they so desire. This homepage will be based on work done in the SOM after meeting
with Maxine Papadakis MD, Karen Hauer MD, PhD and Bonnie Helvig BA regarding strengths and
challenges with their current dashboard program. Importantly, this system must have the flexibility to
modify and grow as we fine tune assessments of trainee activities. Dashboard homepages will contain:
Each trainee’s original ERAS application, Evalue and Qualtrics data for evaluations, a schedule function for
viewing past/present schedules as well as vacations with an interactive timeline to enable residents to track
administrative tasks, links to online modules that the residents need to access/complete (and to be able to
record when they have completed them), self-assessment tools for faculty/trainee reviews, records for
communication for feedback from our CCC, individual trainee quality of care dashboard data from both
inpatient and clinic practice as well as instructional links and short videos instructing how decipher data as it
comes in. Please see sample homepage provided below.
Theme #3: Residents will control the data contained on their homepage. Residents can choose which
EPA evaluations are available for the CCC enabling them to show off their best work. Similarly, CCC
reports will become available to residents after each meeting so they can follow their progress on the
ACGME milestones. This is based on the theoretical work presented by Patricia O’Sullivan PhD in her
portfolio seminar regarding learning theory and adult learners.
Theme #4: Our hope is that along with our fledgling faculty education efforts, constant collection of
resident driven EPA based data will slowly help begin an overall culture change in assessment and
feedback in our department. Residents will become more comfortable asking for activity specific
feedback and faculty will become more comfortable giving it. This culture change will take time and
effort but the resident focus group data was clear that they want timelier, activity-driven feedback so
they should be motivated to ask for it.
Sample Resident Landing Page:
Specific Aims:
1- Create a resident and fellow assessment network that utilizes a resident driven, EPA based
evaluations.
2- Create an individual resident homepage that will become a landing page that can link residents
to all their assessment materials but also to UCSF based learning resources.
3- Ensure a more organized and efficient assessment data management system.
4- Change culture for assessment and feedback to become more transparent, ongoing and
meaningful system for our trainees.
Evaluation Plan:
Process measures
1234-
Percentage of responses to emails c/w Evalue
time to completion of epa evaluations c/w evalue
descriptive eval of epas completed
amount of time of resident use of dashboard
5- actual vs reported numbers of epas completed
Outcome measures
1- satisfaction (faculty/trainees) with system (focus groups)
2- comparison of entrustment on mapped epas c/w milestones
3- number of residents entrustable via epas c/w evalue milestones
what about how it influences CCC
Plan for Sustainability:
After the initial start-up effort and costs to construct the Dashboard system this assessment system
should be able to collect all the assessment data (Evalue, Qualtrics, Portfolio, In-service scores,
Qualitative comments, clinic dashboards, patient follow-up reviews), create summary milestone reports
for the CCC, and be updated annually for new residency and fellow trainees each year. The estimate
from the Information Services Unit (ISU) group is that it will take 6 hours, $500 annually, for trouble
shooting and updates to the system which will be made available from Departmental funds. Updates
and administrative work will mostly involve construction of the new trainee dashboards which can be
done by our program administrator, Amy Forseth.
Team Members:
Jeff Kohlwes MD, MPH. School of Medicine. Department of Internal Medicine. Clinical Professor of
Medicine. Director, PRIME Residency Program, Associate Program Director for Assessment and Feedback
Internal Medicine Residency Program.
Patricia Cornett MD. School of Medicine. Department of Internal Medicine. Clinical Professor of
Medicine. Associate Chair for Education, Department of Internal Medicine.
Vanessa Thompson MD. School of Medicine. Department of Internal Medicine. Assistant Clinical
Professor of Medicine. General Medicine Clinic Assistant Medical Director San Francisco General
Hospital, Internal Medicine Residency Director of Academic Development
Irina Kryzhanovskaya MD. School of Medicine. Department of Internal Medicine. Senior Resident in
Internal Medicine, Chief Resident-Elect.
Ray Chen MD. School of Medicine. Department of Internal Medicine. Junior Resident in Internal
Medicine
Total Budget
10. Consultations Completed:
Nina Jameson, Deputy Director, Program Management, Information Services Unit (ISU)- Consultation on
costs on IT solutions.
Maxine Papadakis MD, Bonnie Helvig BA, UME Dashboard and IT expert for demonstration on UME
dashboard and problem solving on creation of GME dashboard.
Karen Hauer MD, PhD consultation on assessment of EPAs and milestones.
Christy Boscardin PhD, Holly Nishimura BA, Department of Medical Education- Assessment expertise and
Qualtrics planning for the current Pilot Study.
References:
1- Green et al, J Grad Med Educ. 2009; 1:5-20.
2- http://www.acgme.org/acgmeweb/portals/0/pdfs/milestones/internalmedicinemilestones.pdf
Accessed 11/13/15
3- Ten Cate et al. Academic Medicine, Vol. 82, No. 6 / June 2007
4- Farnan et al. A systematic review: the effect of clinical supervision on patient and residency
education outcomes. Acad Med. 2012 Apr;87(4):428-42.
5- Foster et al. RAFT (Resident Assessment Facilitation Team): supporting resident well-being
through an integrated advising and assessment process. Fam Med. 2012 Nov-Dec;44(10):731-4
6- Hauer K, et al. Identifying entrustable professional activities in internal medicine training. J Grad
Med Educ. 2013 Mar;5(1):54-9.