Draft submitted for subcommittee approval 7.2313
Quick Start Manual
Authors: Gloria Kuhn (subcommittee chair), Mark Clark, Sam Luber, Gillian
Schmitz, Stephen Wolf (alphabetical order)
Contents:
I.
Information and components of the Next Accreditation System (NAS):
II.
Challenges and solutions to implementation of the milestones
III.
Glossary of terms: Appendix A
IV.
Websites for more information: Appendix B
NAS
Next Accreditation System (NAS): A new system of accrediting residency training
programs designed to foster innovation and improvement in the learning
environment, increase the accreditation emphasis on educational outcomes,
increase efficiency and reduce burden in accreditation, and improve communication
and collaboration with key internal and external stakeholders. Additionally, the
goals of the NAS are to free good programs to innovate, to assist poor programs to
improve, to reduce the burden of accreditation, and to provide accountability for
outcomes to the Public.
Components of NAS: 1) Resident assessment through milestones that were
envisioned to expand the description of the competencies into a progression of
observable behaviors, that would allow assessment of whether residents had
attained the knowledge, skills, attitudes and proficiency levels needed to practice
their chosen field of medicine. This process will include 1) formation of a clinical
competency committee to look at multiple sources of assessment data and report
their findings on each resident semi-annually to the program director 2) periodic
faculty surveys completed online regarding components of the training program,
3)periodic online resident surveys regarding components of their educational
experience, 4) evaluation of the Clinical Learning Environment (CLE) to assess how
the sponsoring institution fulfills its responsibility for the quality and safety of the
environment for learning and patient care and the clinical learning environment
review (CLER) that assesses the CLE.
Challenges and Solutions to Implementation of the Milestones
1. Lack of standardization in determining resident proficiency
 Solution: Develop anchors, or specific behaviors, that define each
proficiency level. Anchors provide benchmarks to define what would
constitute advancement to the next level.
 Solution: Create a standardized approach to daily shift evaluations that
incorporate the milestones
2. Minimizing Grade Inflation
1
Draft submitted for subcommittee approval 7.2313



Solution: Consider an evaluation form with unique anchor language that
clearly delineates behaviors needed to reach each level of competency.
This can improve reliability between faculty members and improve
feedback to residents.
Solution: Utilize multiple assessment techniques and evaluations from
different faculty
Solution: Faculty development, feedback, and tracking may demonstrate
to faculty how they grade in relation to their peers and may help reduce
grade inflation.
3. Elimination Bias in Scoring Methods
 Solution: Eliminate proficiency level identifiers. This identifier links the
milestones to certain headers, such as “novice” or “postgraduate year (PGY)1”. These introduce bias as evaluators may score to the level of expected,
rather than actual, performance.
 Solution: Consider binary scoring (ie yes/no) rather than traditional scaled
scoring (i.e. a 5 point scale). Binary scoring may eliminate bias by simply
assessing the presence or absence of a specific action. For example, “Did the
resident place oxygen on the patient, Yes or No?” Binary scoring allows more
accurate assessment of specific tasks but these behaviors may not be
applicable for every patient interaction.
 Solution: Consider using two different types of shift evaluation cards: one for
novice and one for more experienced residents. This may simplify the
evaluation by focusing on only the first two proficiency levels for less
experienced residents. Once he or she consistently meets requirements of
novice card, he or she is then evaluated with an advanced card, which focuses
on proficiency levels 3-5.
4. Incorporating Milestones into Evaluation Forms
 Solution: Some vendors are now linking evaluations with specific subcompetencies and milestones
 Solution: Develop shift evaluation cards that measure a few specific subcompetencies on every shift
5. Obtaining Sufficient Data within each Sub-competency
 Solution: Randomize the sub-competencies such that 4 or 5 are evaluated
after each shift. Alternate shift cards weekly, monthly. Etc.
 Solution: Create specific scenarios in simulation lab or standardized
patients to allow assessments of sub-competencies that are not
frequently encountered on a clinical shift
 Solution: Empower residents to be responsible for distributing their own
evaluations and ensuring they have adequate responses on each subcompetency
2
Draft submitted for subcommittee approval 7.2313

Solution: Creating a Performance Dashboard that highlights how many
evaluations each resident has for each of the 23 sub-competencies.
6. Working with Residency Software Management Systems to provide feedback
on future product development that better meet needs of emergency
medicine educators.. Encourage innovation, collaboration, and
communication.
 Solution: A Milestone “show and tell” will be highlighted at the Joint
Milestone Task Force at Scientific Assembly for vendors to participate in a
moderated panel discussion and demonstrate their products.
7. Development of Validity Evidence
 More research is needed to support validity evidence of current
performance evaluations and implementation of milestones.
Glossary of Terms
Appendix A
ABMS: American Board of Medical Specialties
ACGME: Accreditation Council for Graduate Medical Education
ADS: Accreditation Data System (also known as WebADS) is a Web-based system
that contains critical accreditation data for all sponsoring institutions and programs.
Serves as an ongoing communication tool with programs and sponsoring
institutions and incorporates several ACGME applications and functions. Programs
are required to enter program specific data (hospital ED census data, resident
procedure averages, etc) into WebADS at a minimum annually.
Clinical Competency Committee (CCC): A committee created by a residency
program designed to make consensus decisions on the progress of each resident;
CCC members should include core faculty members who have the opportunity to
observe and evaluate residents, but may include other members such as assessment
specialists and non-MD medical educators. The main task of the CCC will be to
determine, based on multiple types of assessment data, whether a resident has met
the milestones and will make recommendations to the PD regarding progression.
Clinical Learning Environment Review Program (CLER): CLER focuses on six
areas important to the safety and quality of care in teaching hospitals and the care
residents will provide in a lifetime of
practice after completion of education: 1) engagement of residents in patient safety;
2) engagement of residents in quality improvement (including opportunities for
reducing health disparities); 3) enhancing practice for care transitions; 4)
3
Draft submitted for subcommittee approval 7.2313
promoting appropriate resident supervision; 5) duty hour oversight and fatigue
management; and, 6) enhancing professionalism in the learning environment and in
reporting to the ACGME
Common Program Requirements: Defined program requirements as set forth by
the ACGME that programs must adhere to; these requirements are subcategorized
into:
Core Requirements: Statements that define structure, resource, or process
elements essential to every graduate medical educational program.
Detail Requirements: Statements that describe a specific structure, resource, or
process, for achieving compliance with a Core Requirement. Programs in substantial
compliance with the Outcome Requirements may utilize alternative or innovative
approaches to meet Core Requirements.
Outcome Requirements: Statements that specify expected measurable or
observable attributes (knowledge, abilities, skills, or attitudes) of residents or
fellows at key stages of their graduate medical education.
Core Competency: One of the 6 competencies defined by the ACGME on which
physicians in training should be assessed: patient care; medical knowledge;
practice-based learning and improvement; interpersonal and communication skills;
professionalism; and systems-based practice.
Educational Milestones: Observable developmental steps, organized under the six
competency areas, that describe a trajectory of progress on the competencies from
novice (entering resident) to proficient (graduating resident) and, ultimately, to
expert/master (expert/master level usually achieved after graduation).
Faculty Survey: an annual survey administered by the ACGME in which core faculty
are anonymously asked about the program in the areas of: faculty supervision and
teaching, educational content, resources, patient safety, and teamwork. A 60%
response rate is required by the ACGME.
Focused Site Visit: Assesses selected aspects of a program and may be used to
address potential problems identified during review of annually submitted data, to
diagnose factors underlying deterioration in a program’s performance, and to
evaluate a complaint against a program
GME: graduate medical education
Next Accreditation System (NAS): A new system of accrediting residency training
programs in designed to foster innovation and improvement in the learning
environment, increase the accreditation emphasis on educational outcomes,
increase efficiency and reduce burden in accreditation and improve communication
4
Draft submitted for subcommittee approval 7.2313
and collaboration with key internal and external stakeholders. Major components of
the NAS include milestone assessment, clinical competency committees and the
Clinical Learning Environment Review Program (CLER). Additionally, the goals of
the NAS are to begin the realization of the promise of Outcomes, to free good
programs to innovate, to assist poor programs to improve, to reduce the burden of
accreditation, and to provide accountability for outcomes to the Public.
Phase 1 Specialties: Specialties that will enter the NAS in July 2013: Diagnostic
Radiology, Emergency Medicine, Internal Medicine, Neurological Surgery,
Orthopaedic Surgery, Pediatrics, Urology
Program Information Form (PIF): The PIF was a standardized reporting tool for
residencies to demonstrate compliance with ACGME Core and Specialty-specific
program requirements. With the introduction of the NAS, PIFs will no longer be
used for continued accreditation.
Resident Survey: an annual survey administered by the ACGME in which residents
are anonymously asked about the program in the areas of: duty-hours, faculty,
evaluation, educational content, resources, patient safety and teamwork. A 70%
response rate is required by the ACGME.
RRC: Residency Review Committee (of the ACGME). RRCs are setup for each
specialty and are tasked with reviewing/approving applications for new residency
programs as well as approving continued accreditation of established programs
Self-Study Visit: A visit by the ACGME that will be scheduled every 10 years. It will
be conducted by a team of visitors who will interview residents, faculty and
leadership. The goal of the visit is to foster continuous improvement in the
program, learn future goals of the program and verify compliance with core
requirements. This visit will take the place of the site-surveys conducted in the past.
Subcompetency: Defined competencies under one of the 6 core competencies in
which educational milestones have been developed.
WEBSITES:
Appendix B
JMTF
“The Wiki” is a website created by members of CORD who formed the Joint
Milestone Task Force (JMTF). The purpose of the website is to help develop
resources for residency programs around the country. It is open access to all who
wish to obtain information. To actually join the group and place materials on the
wiki access must be requested. It has a wealth of information including articles,
5
Draft submitted for subcommittee approval 7.2313
answers to frequently asked questions, and checklists to evaluate procedural
competency. The site is well organized to enable materials to be easily found.
ACGME
www.Emmilestones.org
The ACGME has created multiple materials on their website relating to NAS. The
site has webinars explaining NAS which at this time are accessible only to program
directors, however the webinar slides are open access. The easiest way to gain
information is to go to www.acgme.org and search the site for information related to
either NAS or milestones.
Link to EM Milestones: http://www.acgmenas.org/assets/pdf/Milestones/EmergencyMedicineMilestones.pdf
ACGME/ NAS Website: http://www.acgme-nas.org
CORD
CORD website: www.cordem.org
End of Shift Milestone Evaluation Forms: www.cordtests.org
You can create a “test” shift evaluation using the following administrative ID and
password. Click on "For Residencies," and use 3530 as the Administrator ID, and
21528 as the administrative password. Then click on "Milestones" on the far
right. You are taken to a screen where you can download the Word documents,
enter in forms, or generate reports. The end of shift evaluation forms are easy to
use.
WEBINARS
ACGME Phase I Webinar Series (note: the webinars are not accessible to all faculty
at this time. Program directors can access by logging in.)
ACGME Phase I: Emergency Medicine
June 13, 2013
11:00-12:30 p.m. Central Time
Speakers:
Louis Ling, MD—Senior Vice President, Hospital-based
Accreditation, ACGME
Wallace Carter, MD—Chair, ACGME Review Committee for
Emergency Medicine
Michael Beeson, MD—Vice-Chair, ACGME Review Committee for
6
Draft submitted for subcommittee approval 7.2313
Emergency Medicine;
Chair, Milestone Working Group
This is a webinar to update the emergency medicine community about the NAS,
which goes into effect July 1, 2013. Drs. Carter, Beeson, and Ling will discuss recent
questions and clarifications about the NAS including: 1. using data as a screening
tool for the Review Committee; and, 2. the Program Requirements for the Clinical
Competency Committee (CCC) and the Program Evaluation Committee (PEC).
This session will assume that the audience has a basic background of the NAS.
7