Conflict of Interest



Kim Walker – No conflicts of interest to disclose
Ann Dohn – No conflicts of interest to disclose
Nancy Piro – No conflicts of interest to disclose
Department of Graduate Medical Education (GME)
PC002c
Coordinators and Clinical Competency
Committees: How to Streamline and Support
the Work of your Program’s CCC
Kim Walker, PhD
Program Manager/Education Specialist
Ann Dohn, MA
DIO & GME Director
Nancy Piro, PhD
Program Manager/Education Specialist
Session Outcomes
Participants will be able to:

Identify new aspects of the coordinators’ evolving role in
program administration.

Understand and use program requirements as a guide for
planning, organizing and implementing educational and
assessment tools.

Develop and utilize a newly developed comprehensive
resident performance profile tool to streamline the work of
the CCCs.
Department of Graduate Medical Education (GME)
New Role
Department of Graduate Medical Education (GME)
Evolving Role for Coordinators in
Evaluations
Education &
Evaluation
Coordinator/Manager
Administrator
Scheduler
Supreme
• Deliver evaluations
• Develop evaluation forms for PDs
to approve
• Schedule semi annual evaluations
• Ensure summative evaluations
completed and filed
Extraordinaire
• Constructing new milestone
evaluations to pilot/deliver
• Reviewing evaluation
completion data for accuracy
• Aggregating data for the
CCC from multiple sources
and forms
• Milestone data to ACGME
Department of Graduate Medical Education (GME)
Now I’m really confused!
Outcomes
Evaluations
CCCs
EPAs
Milestones
Goals and Objectives
Department of Graduate Medical Education (GME)
Where do I begin?
Department of Graduate Medical Education (GME)
Know the NAS Building Blocks:
Concepts defined


NAS
Core Competencies
–
–
–

Milestones
EPAs
Curriculum and Evaluations
Clinical Competency Committee (CCC)
Department of Graduate Medical Education (GME)
NAS – Next Accreditation System
What is NAS – in a nutshell:
“an outcomes-based accreditation process
through which the doctors of tomorrow will be
measured for their competency in performing the
essential tasks necessary for clinical practice in the
21st century.”
http://www.acgme.org/acgmeweb/tabid/435/ProgramandInstitutionalAccred
itation/NextAccreditationSystem.aspx
Department of Graduate Medical Education (GME)
Major Changes:
Accreditation based on…

Pre-NAS
–
–
–
–
–
–
Competencies
Site Visits – Up to 5+
year cycles
Internal Reviews
ADS Updates
PIFs
Resident Surveys

Current (New) NAS
–
–
–
–
–
Competencies with
Milestones
Self-Studies at ~ 8-10 year
intervals
Detailed ADS Updates
CLER Visits ~ 18 – 24
months (Institution)
Resident & Faculty
Surveys
Department of Graduate Medical Education (GME)
The New Accreditation System (NAS)…
Outcomes



Increased Annual reporting by Programs (online)
Reduced volume of accreditation demands …
but increased attention to accuracy and
completeness of information submitted
online
PIF-less Surveyor visits (unless new application)
–

Two Field Surveyors per visit
No Faculty CVs (only PD)….but Faculty &
Resident Scholarly Activity required.
Department of Graduate Medical Education (GME)
The Six AGME Core Competencies
Medical
Knowledge
Practicebased
Learning &
Improvement
Patient
Care
Six Core
Competencies
For Quality
Patient Care
Systems-based
Practice
Department of Graduate Medical Education (GME)
Interpersonal &
Communication
Skills
Professionalism
What Are Milestones?



High Level - Milestones are simply defined as
areas of competency/expectations for our
trainees
Linked to six core competencies
Defined as a continuum of progressive
growth/learning
Department of Graduate Medical Education (GME)
Dreyfus Model (1980):
Stages of developing expertise
* Rigid adherence to taught rules or plans
* Little situational perception
* No discretionary judgment
1 Novice
2 Advanced Beginner
* Guidelines for action based on attributes or aspects
* Situational perception still limited
* All attributes and aspects are treated appropriately and given equal
importance
4 Proficient
*
*
*
*
*
*
*
*
*
Coping with 'crowdedness'
Now sees actions at least partly in terms of longer-term goals
Conscious deliberate planning
Standardized and routinized procedures
Sees situations holistically rather than in terms of aspects
See what is most important in a situation
Perceives deviations from the normal patterns
Decision-making less labored
Uses maxims for guidance, whose meaning varies according to the situation
5 Expert
*
*
*
*
No longer relies on rules, guidelines or maxims
Intuitive grasp of situations based on deep tacit understanding
Analytic approaches used only in novel situations or when problems occur
Vision of what is possible
3 Competent
Department
of Graduate Medical Education (GME)
Source: Eraut, M. Developing Professional Knowledge and Competencies. (1994)
Milestone Level Definitions



Level 1: The resident is a graduating medical
student/experiencing first day of residency.
Level 2: The resident is advancing and
demonstrating additional milestones.
Level 3: The resident continues to advance and
demonstrate additional milestones; the resident
consistently demonstrates the majority of
milestones targeted for residency.
Department of Graduate Medical Education (GME)
Milestone Level Definitions (continued)

Level 4:

Level 5: The resident has advanced beyond
The resident has advanced so that he or
she now substantially demonstrates the milestones
targeted for residency. This level is designed as the
graduation target – not requirement.
performance targets set for residency and is
demonstrating “aspirational” goals which might
describe the performance of someone who has been
in practice for several years. It is expected that only
a few exceptional residents will reach this level.
Department of Graduate Medical Education (GME)
Reporting the Milestones
V.A.1.b).(1).(b) prepare and assure the
reporting of Milestones evaluations of each
(Core)
resident semi-annually to ACGME


Milestones are reported directly through ADS
Reporting windows are:
–
–
November 1- December 31
May 1- June 15
Department of Graduate Medical Education (GME)
Entrustable Professional Activities (EPA)

Professional life activities that define a medical
specialty:
–
–
–
Ground the competencies in a physician’s everyday
work
Activities lead to some outcome that can be observed
Complexity of the activities requires an integration of
knowledge, skills and attitudes across competency
domains
Department of Graduate Medical Education (GME)
Examples of EPAs



Facilitate handovers to another healthcare
provider either within or across settings
Contribute to the scholarly work of the
subspecialty
Co-manage patients with generalists and other
subspecialists
Source: https://www.abp.org/abpwebsite/taskforce/reslib/24.ppt
Department of Graduate Medical Education (GME)
Curriculum: Rotation-specific goals and
objectives & links to milestones
Department of Graduate Medical Education (GME)
Milestones Impact on Evaluations:
Linking questions to milestones

Step Two:
Ensure
specific
Advises the
referring health
care provider(s)evaluation
about the
appropriateness of a procedure in routine clinical situations
questions are linked to milestones
Department of Graduate Medical Education (GME)
Milestones Impact on Evaluation System



Allows for more objective methods of
assessment and provide better feedback
Provides a process for early identification of
residents that are having difficulties
All old and new evaluations and questions
should be aligned with and tracked to
milestones
Department of Graduate Medical Education (GME)
Clinical Competency Committee (CCC)
V.A.1. The program director must appoint the Clinical Competency
Committee.(Core)
V.A.1.a) At a minimum the Clinical Competency Committee
must be composed of three members of the program
faculty.(Core)
V.A.1.a).(1) Others eligible for appointment to the
committee include faculty from other programs and nonphysician members of the health care team.(Detail)
ACGME Common Program Requirements
Approved: February 7, 2012; Effective: July 1, 2013
Approved focused revision: June 9, 2013; Effective: July 1, 2013
Department of Graduate Medical Education (GME)
Clinical Competency Committee (CCC)
V.A.1.b).(1) The Clinical Competency Committee should:
V.A.1.b).(1).(a) review all resident evaluations semiannually; (Core)
V.A.1.b).(1).(b) prepare and assure the reporting of
Milestones evaluations of each resident semi-annually to
ACGME; and, (Core)
V.A.1.b).(1).(c) advise the program director regarding
resident progress, including promotion, remediation, and
dismissal.(Detail)
Department of Graduate Medical Education (GME)
Clinical Competency Committee (CCC)
V.A.1.b) There must be a written description of
the responsibilities of the Clinical Competency
(Core)
Committee.
ACGME Common Program Requirements
Approved: February 7, 2012; Effective: July 1, 2013
Approved focused revision: June 9, 2013; Effective: July 1, 2013
Department of Graduate Medical Education (GME)
Clinical Competency Committee (CCC):
Written description
Department of Graduate Medical Education (GME)
Clinical Competency Committee (CCC)

How the CCC does its work can be decided by
the Program Director
–
–
–
–
Subcommittees
Assigning residents to faculty members for pre-review
Pre-review work will vary
Scheduling and frequency of meetings
Department of Graduate Medical Education (GME)
What Should a CCC Do First?
Understand their specialty Milestones
(Posted on acgme.org)
 Decide how to assess the Milestones
– Program Evaluation Strategy
 If necessary, identify new evaluation
tools from program director
associations, societies, colleges

Department of Graduate Medical Education (GME)
NAS and Milestones and YOU

The program coordinator will play a crucial
role in developing, implementing, collecting
data on and reporting of milestone evaluation
tools.
Department of Graduate Medical Education (GME)
Managing it all…
Department of Graduate Medical Education (GME)
U - R - IT!
1.
2.
Understanding RRC program requirements
Requirements applied to evaluation
methods/process
3.
4.
Implementing new evaluation system
Tracking completion and accuracy (outliers)
for data aggregation
Department of Graduate Medical Education (GME)
1. Understand Your Program’s New
Requirements
Core
Outcomes
Details
Department of Graduate Medical Education (GME)
1. Understand Your Program’s New
Requirements
Each standard/requirement is categorized:
 Outcome - All programs must adhere
 Core - All programs must adhere
 Detail – Considered mandatory for new
programs and those that fail to meet core
requirements. Allows high-performing
programs to innovate.
Source: Implementing The Next Accreditation System ACGME Webinar
John R. Potts, III, M.D.: 4 November 2013
Department of Graduate Medical Education (GME)
1. Understand Your Program’s New
Requirements
Example Program Requirement:
VI.B. Transitions of Care
(Core)
(Core)
(Outcome)
Department of Graduate Medical Education (GME)
1. Understand Your Program’s New
Requirements
Example Program Requirement: VI.B. Transitions of Care
When core and outcome not in compliance, then:
Details
Department of Graduate Medical Education (GME)
U - R - IT!
1.
2.
Understanding RRC program requirements
Requirements applied to evaluation
methods/process
–
–
3.
4.
Linking milestones/EPAs and objectives to evaluation
questions
Utilizing milestone scales
Implementing new evaluation system
Tracking completion and accuracy (outliers) for
data aggregation
Department of Graduate Medical Education (GME)
2. Requirements Specific to CCC Review
of Trainee






Aggregating/compiling multiple evaluations of
individual trainees (V.A.1. Formative Evaluation)
Tracking trainee participation in conferences,
journal clubs, didactics (IV.A.3. Didactic Sessions)
Monitoring duty hour compliance (VI.G. Duty
Hours)
Reviewing involvement in quality improvement and
patient safety activities (IV.A.5.c. PBLI)
Reviewing scholarly work (IV.B. Scholarly Work)
Monitoring and reporting procedure logs (IV.A.5.a)
Department of Graduate Medical Education (GME)
U - R - IT!
1.
2.
Understanding RRC program requirements
Requirements applied to evaluation
methods/process
–
–
3.
4.
Linking milestones/EPAs and objectives to evaluation
questions
Utilizing milestone scales
Implementing new evaluation system
Tracking completion and accuracy (outliers) for
data aggregation
Department of Graduate Medical Education (GME)
3. Implementing

Evaluation systems
–
–
–
–
–
Milestone-based/EPAs
Rotation-specific
Patient handovers
Define evaluator groups (faculty, staff, patients)
Set up and timing of delivery systems
Department of Graduate Medical Education (GME)
3. Implementing

Documentation and reporting systems for:
–
–
–
–
–
Conference attendance
Scholarly work (Learning Portfolios)
Quality Improvement and Patient Safety
(Learning Portfolios / Safety reporting systems)
Duty Hours (recording, monitoring, reporting)
Case Logging (if applicable)
Department of Graduate Medical Education (GME)
U - R - IT!
1.
2.
Understanding RRC program requirements
Requirements applied to evaluation
methods/process
–
–
3.
4.
Linking milestones/EPAs and objectives to evaluation
questions
Utilizing milestone scales
Implementing new evaluation system
Tracking completion and accuracy (outliers) for
data aggregation
Department of Graduate Medical Education (GME)
4. Tracking and Reporting…

Start with the end in mind:
–
CCC biannual reporting windows to ACGME


November 1- December 31 / May 1 - June 15
Back track and set calendar events for:
–
–
–
Periodic monitoring of evaluation completion
Running aggregate reports and reviewing milestone
evaluation data
Reviewing case logs, learning portfolios, duty hours
Department of Graduate Medical Education (GME)
Pulling the Data Together
End-ofRotation
Evaluations
Clinical
Skills
Assessment
Nursing and
Staff / Techs
Evaluations
Quality
Improvement
Activities
Sim
Lab
Clinical
Competency
Committee
Progress on
Milestones
Department of Graduate Medical Education (GME)
In-service
training
exams
Safety
Incident
Reports
Case
Logs
Patient/
Family
Evaluations
Managing it all:
How will I pull this off?
Department of Graduate Medical Education (GME)
The Toolbox
Department of Graduate Medical Education (GME)
Creating a Resident Performance Profile
Goals to support your CCC
Resident performance data that is:
 Comprehensive
 Consolidated / Aggregated
 Easy for CCC to identify strengths, areas for
improvement, opportunities for advancement
Department of Graduate Medical Education (GME)
Creating a Resident Performance Profile:
Compiling and centralizing data
Department of Graduate Medical Education (GME)
Creating a Resident Performance Profile:
Apply visual formatting for trends
Department of Graduate Medical Education (GME)
Creating a Resident Performance Profile
Step 1 – Defining what to track
Evaluations
Milestone End of Rotation (Total)
Medical Knowledge
Patient Care
Interpersonal & Communication Skills
Professionalism
Systems-Based Practice
Practice-Based Learning & Improvement
Department of Graduate Medical Education (GME)
Creating a Resident Performance Profile
Step 1 – Defining what to track
In-service Assessments (MK; PC)
Routine procedure technical skills
assessment: Level 2-3
Complex procedure technical skills
assessment: Level 3-4
Medical Knowledge Assessments
Department of Graduate Medical Education (GME)
Creating a Resident Performance Profile
Step 1 – Defining what to track
Case Logs / Clinical Experience (PC)
VAGINAL DELIVERY
CAESAREAN SECTION
PEDIATRICS
PEDIATRICS UNDER 3
CARDIAC
ENDOVASCULAR
Department of Graduate Medical Education (GME)
Creating a Resident Performance Profile
Step 1 – Defining what to track
Quality Improvement/Patient Safety
QI Project
QI Committee Participation
QI Course Work (e.g., IHI)
SAFE Report / Adverse Event Review
Patient Handover Evaluations
Department of Graduate Medical Education (GME)
Creating a Resident Performance Profile
Step 1 – Defining what to track
Practice-Based Learning
Patient outcomes / Case Study Presentation
Scholarly Activity: Research study
Scholarly Activity: Publications
Presentation(s) at Internal/National Meetings
Department of Graduate Medical Education (GME)
Creating a Resident Performance Profile
Step 1 – Defining what to track
Interpersonal and Communication Skills
Patient Feedback
Staff Evaluation
Medical Student Feedback
Patient Handover Evaluations
Department of Graduate Medical Education (GME)
Creating a Resident Performance Profile
Step 2 - Link data sources to milestones
Evaluations
Milestone End of Rotation (Total)
Medical Knowledge
Patient Care
Communication
Professionalism
Systems-based Practice
Practice-based Learning
Quality Improvement/Patient Safety
QI Project
QI Committee Participation
QI Course Work (e.g., IHI)
SAFE Report / Adverse Event Review
Patient Handover Evaluations
Department of Graduate Medical Education (GME)
Milestones
24
MK A1
PC A1-10
ICS A1-2
PROF 1-5
SBP A1-2
PBLI A1-4
PBLI1-2
PBLI1-2
PBLI1-2
PROF1/2; SBP1
SBP1; ICS1
Creating a Resident Performance Profile
Step 3: CCC defines performance ranges
Example:
For all aggregate milestone evaluation scores for a PGY 3, the
CCC has defined these ranges by PGY level in advance of the
meeting:
At or Above Expectation: Below Expectation: Remediation:
Below 1.7
1.7 – 2.7
2.8 and higher
STRENGTH
WATCH
Department of Graduate Medical Education (GME)
AT RISK
Creating a Resident Performance Profile
Step 4 – Set conditional formatting
Department of Graduate Medical Education (GME)
Creating a Resident Performance Profile
Step 4 – Set conditional formatting
Example: Aggregate milestone evaluation data cells
Highlight cells to apply the
conditional formatting
Department of Graduate Medical Education (GME)
Creating a Resident Performance Profile
Step 4 – Set conditional formatting
Set Ranges: > , < , between
Department of Graduate Medical Education (GME)
Creating a Resident Performance Profile
Step 4 – Set conditional formatting
Select, “Greater Than” “Less Than” or “Between” to Set Value Ranges
Choose the corresponding fill color (e.g., red, yellow, green)
Department of Graduate Medical Education (GME)
Resident Performance Profile:
Step 5: Enter in data
Department of Graduate Medical Education (GME)
Creating a Resident Performance Profile
Visual trends and detailed data
Department of Graduate Medical Education (GME)
More tools…
Department of Graduate Medical Education (GME)
Leveraging Resident Management
System (RMS)Tools, if Available


RMSs – becoming more feature rich
Curriculum
–
–

Goals and Objectives and learning outcomes by
rotation
Teaching and Assessment methodologies
Evaluation tool development
–
Sharing between programs and institutions
Department of Graduate Medical Education (GME)
Leveraging Resident Management
System (RMS)Tools, if Available

Conference attendance statistics
–
–

Core competencies linked to specified
conferences
Attaching conference materials for later reference
Procedures and levels; linked procedure
evaluations
Department of Graduate Medical Education (GME)
Leveraging Resident Management
System (RMS)Tools, if Available
• Resident portfolio tools
QI participation and outcomes
Scholarly Activity logs
Department of Graduate Medical Education (GME)
Leveraging Resident Management
System (RMS)Tools, if Available

Aggregate reporting and graphic summaries

Peer or departmental average, individual average, minimum
and maximum scores, standard deviation or listing of all
scores
Department of Graduate Medical Education (GME)
Leveraging Calendaring and Task
Management Software



Set “data gathering and reporting” appointments
with yourself
Remember to start with the end in mind (e.g.,
CCC meeting dates)
Break down large tasks into smaller tasks to
keep it manageable
Department of Graduate Medical Education (GME)
Leveraging Calendaring and Task
Management Software
Department of Graduate Medical Education (GME)
Learning through Experience
Department of Graduate Medical Education (GME)
When a CCC Meeting…
Doesn’t go well

Data
–
–
–



not complete
not organized
not accurate
Does go well

Data
–
–
–
complete
organized
accurate
PD or faculty member
 Cooperative,
dominates meeting
collaborative decision
Prolonged inefficient
making
decision making with
 Efficient use of time
inability to gain consensus
 Sound valid conclusions
Unsubstantiated/unreliable
aligned with data
conclusions
Department of Graduate Medical Education (GME)
4
P
P
G
Y
3
P
G
Y
2
Y
P
G
Thetical, Hypo
G
Y
1
Successful Resident Ranking
PGY1- 4: Ready to graduate
6/30/08
6/30/09
6/30/10
6/30/11
MK A1-6
PC A1-10
1.5
1.3
1.4
2.3
1.9
2.0
3.1
3.1
2.9
3.9
3.6
4.0
ICS A1-2
1.1
2.3
2.9
3.9
PROF 1-3
2.1
2.8
3.5
4.2
Systems-based Practice SBP A1-2
1.3
2.0
Practice-based Learning
& Improvement
PBLI A1-2
1.5
2.8
Department of Graduate Medical Education (GME)
3.0
4.1
3.2
3.8
Date Completed
Evaluations
Milestone End of
Rotation (Total)
Medical Knowledge
Patient Care
Interpersonal
Communication Skills
Professionalism
Milestones
25
PGY 1 Ranked at Graduation Level
1
Milestone range for a
PGY1 should not be a
4.0, 4.5 or 5.0 …
Department of Graduate Medical Education (GME)
4
P
G
Y
3
P
G
Y
2
Y
P
Professionalism
P
Date Completed
Evaluations
Milestone End of
Rotation (Total)
Medical Knowledge
Patient Care
Interpersonal
Communication Skills
G
Thetical, Hypo
G
Y
1
Improvement evident
6/30/08
6/30/09
6/30/10
6/30/11
MK A1-6
PC A1-10
0.9
0.8
0.9
1.5
1.9
0.9
2.7
2.6
2.9
4.0
4.0
4.2
ICS A1-2
0.7
0.8
2.7
4.2
PROF 1-3
0.9
0.5
2.0
3.9
2.0
3.0
4.1
2.8
3.2
3.8
Milestones
25
Systems-based Practice SBP A1-2
1.0
Practice-based Learning
& Improvement
PBLI A1-2
1.1
Department of Graduate Medical Education (GME)
We Should not be at this Point
Department of Graduate Medical Education (GME)
Beyond data…Creating a climate
of CCC Success
Gentle Words of Wisdom






Tight efficient meetings
Ground Rules
Beware of Negative Group Think
Schedule firm standing meeting dates in advance
Reserve room of appropriate size with required
audio-visual tools if needed
…..and have snacks
Department of Graduate Medical Education (GME)
Use Technology to Your Advantage…
You can be a ‘Rock Star’
Department of Graduate Medical Education (GME)
Session Recap in a Nutshell…



Know your program requirements and follow them
unconditionally
Use simple spreadsheet, calendaring and task
organizational tools to manage, track and present
resident performance data to your CCC
Resident education is a cyclical process – revisit
and revise tools and processes each year
Department of Graduate Medical Education (GME)
Questions
Department of Graduate Medical Education (GME)
Contacts



Kim Walker - kwalker5@stanford.edu
Ann Dohn - adohn1@stanford.edu
Nancy Piro - npiro@stanford.edu
Department of Graduate Medical Education (GME)