 Nancy Piro, PhD – No conflicts of interest to disclose
 Kim Walker, PhD - No conflicts of interest to disclose
SES:096
Voice of the Director: Your experience on what
works and doesn’t work with CCC meetings
Nancy Piro, PhD
Education Specialist/Program Manager
Stanford University Medical Center
Kim Walker, PhD
Instructional Designer
School of Medicine, Stanford University
Session Overview
 Introduction and background on multi-institution survey
 Report on survey results from Program Directors at multiple institutions on the
CCC process and outcomes
− Emergent themes on directors’ experiences and their roles in the CCC
 Quantitative outcomes
 Qualitative outcomes
− What directors tell us was ineffective in the CCC process
− Best practices for coordinators supporting a successful CCC
 Action planning for implementation of best practices
Confidential – For Discussion Purposes Only
Objectives & Session Focus
 This session will focus on understanding and discussing
the results from a multi-institution survey on CCC practices
from the Program Directors’ perspective.
 At the conclusion, participants will be able to:
− Review their own practices in light of the survey outcomes
− Identify best practices that best suit their program or institution
needs
− Develop and take home an action plan enabled by these best
practices to better facilitate the work of their CCCs
Confidential – For Discussion Purposes Only
Background
Confidential – For Discussion Purposes Only
Milestones
Confidential – For Discussion Purposes Only
Clinical Competency Committee
How the CCC does its work is decided by the Program Director
Confidential – For Discussion Purposes Only
Confidential – For Discussion Purposes Only
Survey Idea – from Inspiration to Implementation
Confidential – For Discussion Purposes Only
9
Survey – Fun Facts
 What:
− A 14 Question* web-based survey with both quantitative and open ended questions
was developed to assess the program director and coordinator experience so far with
respect to CCCs.
 When and To Whom:
− Between November 12, 2014 and January 15, 2015, the survey link was distributed to
a wide sample of directors and coordinators (both within Stanford and across the
country to the 15 largest institutions, and to PC networks in OB/GYN, Anesthesia, and
Surgery)
 Analytic Methods Used – Both Qualitative and Quantitative Analyses were used.
 Response: 135 Program Directors and 325 Program Coordinators responded to
our survey by January 15, 2015
* Questions detailed on the following slide
Confidential – For Discussion Purposes Only
Survey Questions
1. What's your program?
2. Are you a voting member of your CCC?
3. During your CCC meeting, how much time (on the average) per resident
was spent on the milestone assessments?
4. What actions has your program taken to better align your training
program with your milestones (e.g., changed curriculum, added didactics,
linked current evals to milestones, used spreadsheets to aggregate
evaluation data, added new evaluation tools)?
5. What data was used in your CCC meeting for trainee assessment?
6. What data was more valuable in your overall milestone assessments?
7. Please describe any aspects of the milestone evaluation process that you
thought were particularly beneficial.
Confidential – For Discussion Purposes Only
Survey Questions (2)
8. Please describe any aspects of the CCC process that were particularly
cumbersome.
9. To what extent do milestone evaluations provide for a more thorough
semi-annual evaluation?
10. To what extent do milestone evaluations increase our documentation of
trainee strengths?
11. Milestone evaluations increase our documentation of trainee
weaknesses/areas for improvement.
12. The CCC process for trainee milestone evaluations is worth the
time/effort.
13. If not worth the time and effort, why not?
14. Lastly, what did we miss? Are there any other areas of concern or
suggestions?
Confidential – For Discussion Purposes Only
How to Apply to Your Institution
CURRENT PRACTICE
(Starting State)
CCC Membership and Size
Faculty Pre-work
Alignment of Milestones
Evaluation Tools
CCC Process
Challenges
Confidential – For Discussion Purposes Only
POSSIBLE PRACTICE
(End State)
Q1. Distribution of Programs Surveyed – 60 programs responded
Anesthesiology
3.31%
Cardiovascular disease
3.31%
Emergency medicine
Internal medicine
Neonatal-perinatal medicine
3.31%
4.96%
2.48%
Nephrology
3.31%
Obstetrics and gynecology
3.31%
Orthopaedic surgery
3.31%
Otolaryngology
Pathology-anatomic and clinical
2.48%
2.48%
Pediatric cardiology
3.31%
Pediatric infectious diseases
3.31%
Pediatric nephrology
Pediatrics
2.48%
3.31%
Physical medicine and rehabilitation
2.48%
Urology
2.48%
Other (1 response)
20.75%
Other (2 responses)
0.00%
Confidential – For Discussion Purposes Only
29.70%
10.00%
20.00%
30.00%
40.00%
Q2. PDs - voting member of your Clinical Competency Committee (CCC)?
100%
75%
50%
25%
0%
Confidential – For Discussion Purposes Only
83.97%
16.03%
Yes
No
PC Survey: Who is on your program's CCC?
Program Director
88.93%
Associate Program
Director
69.74%
Chief Resident(s)
7.01%
Rotation Director
25.83%
Program Faculty
Member
89.30%
Program Director from
other program
Attending who does
sign out
7.38%
21.03%
Nurse
8.86%
Case Manager
2.21%
0%
10%
Confidential – For Discussion Purposes Only
20%
30%
40%
50%
60%
70%
80%
90%
100%
Q3. How much time (on the average) per resident was spent on the
milestone assessments?
1-10 minutes
23.08%
11-20 minutes
41.54%
21-30 minutes
18.46%
31-40 minutes
9.23%
41-60 minutes
5.38%
>60 minutes
2.31%
0.00%
Confidential – For Discussion Purposes Only
10.00%
20.00%
30.00%
40.00%
50.00%
Q4. Alignment of milestones with training: Emergent Themes
New forms of evaluation (n=82)
New curriculum/didactics (n=26)
Confidential – For Discussion Purposes Only
18
Q4. Alignment of milestones with training: Emergent Themes
New direct observation tools
“Started surgical skill evaluations”
“Created a computer based tool to provide instant
feedback and document milestone progress.”
“…ramped up eval tools over time (mini CEX, tech
skills assessment)…”
Confidential – For Discussion Purposes Only
Q4. Alignment of milestones with training: Emergent Themes
Dashboard Spreadsheets
“Spread sheets, color coded
resident measures such as conference
attendance”
“created spreadsheet to make sure
our evaluation tools capture all the
milestones”
Confidential – For Discussion Purposes Only
20
Q5. Data used in CCC meetings for trainee assessment
Aggregate Direct
Milestone
Evaluations
65.38%
Aggregate Rotation
Evaluations
93.85%
Staff or patient
(360) Evaluations
76.92%
Technical Skills
Assessments/MiniCEX's
44.62%
Scholarly Activities
63.08%
Case Volumes
36.92%
0%
10%
Confidential – For Discussion Purposes Only
20%
30%
40%
50%
60%
70%
80%
90%
100%
Data included in the CCC Review: Overall Emergent Themes
 Aggregate milestone
evaluation scores
 Board scores
 Conference Attendance
 Continuity clinic chart review
 Curriculum Vitae
 Didactic attendance
 Duty Hour Summary
 Duty Hour reporting
compliance
 Evaluation completion rate
 In service exam scores
 Learning modules completion
 Medical records outstanding
OP reports
Confidential – For Discussion Purposes Only
 Medical student evaluations
 Mock oral results
 Narrative evaluation
comments
 Peer Evaluations
 Presentations
 Previous evaluation
 Procedure logs
 Research projects
 Self assessment scores
 Spider diagrams with
comparison data
 Continuity clinic attendance
Q6. What data was more valuable in your overall milestone assessments?
Quantitative
aggregate
evaluation scores
Qualitative
comments and
direct observations
by CCC members
12.40%
46.51%
Qualitative and
quantitative data
were equally
valuable
41.09%
0%
10%
Confidential – For Discussion Purposes Only
20%
30%
40%
50%
60%
70%
80%
90%
100%
Q7. Beneficial aspects of the milestone evaluation process
Committee-based review and discussion
“Group discussion to clarify our own
thinking and understanding of the
Milestones and their meaning”
Confidential – For Discussion Purposes Only
24
Q7. Beneficial aspects of the milestone evaluation process
Committee-based review and discussion
“…recognition of deficits in our teaching”
Confidential – For Discussion Purposes Only
25
Q7. Beneficial aspects of the milestone evaluation process
Committee-based review and discussion
“Multiple opinions strengthen
the rigor of the process.”
Confidential – For Discussion Purposes Only
26
Q7. Beneficial aspects of the milestone evaluation process
Committee-based review and discussion
“…provides a much more
comprehensive look at an
individual resident's27performance.”
Confidential – For Discussion Purposes Only
Q7. Beneficial aspects of the milestone evaluation process
More objective and focused reviews
“more detailed evaluation”
“common language”
“
“more objective evidence”
Confidential – For Discussion Purposes Only
Q7. Beneficial aspects of the milestone evaluation process
Identification of trainee strengths and areas
for improvement
Confidential – For Discussion Purposes Only
Q7. Beneficial aspects of the milestone evaluation process
Identification of gaps in education (curriculum)
Confidential – For Discussion Purposes Only
30
Q8. Cumbersome aspects: Emergent Themes
Milestone interpretation
“contrived”
“nebulous”
“uncalibrated”
"what are
they really
getting at
here?"
Confidential – For Discussion Purposes Only
Q8. Cumbersome aspects: Emergent Themes
Time Consuming
Confidential – For Discussion Purposes Only
Q8. Cumbersome aspects: Emergent Themes
Acquisition of sufficient data
Another 50 milestone
evals due tomorrow?!
I can’t take it anymore!!!
Confidential – For Discussion Purposes Only
33
Q9. Milestone evaluations provide for a more thorough semi-annual
evaluation.
Very Strongly
Agree
6.87%
Strong Agree
19.08%
Agree
41.22%
Disagree
16.79%
Strongly Disagree
9.92%
Very Strongly
Disagree
6.11%
0%
10%
Confidential – For Discussion Purposes Only
20%
30%
40%
50%
60%
70%
80%
90%
100%
Q10. Milestone evaluations increase our documentation of trainee
strengths.
Very Strongly
Agree
6.11%
Strong Agree
21.37%
Agree
38.93%
Disagree
20.61%
Strongly Disagree
7.63%
Very Strongly
Disagree
5.34%
0%
10%
Confidential – For Discussion Purposes Only
20%
30%
40%
50%
60%
70%
80%
90%
100%
Q11. Milestone evaluations increase our documentation of trainee
weaknesses/areas for improvement
Very Strongly
Agree
5.34%
Strong Agree
24.43%
Agree
40.46%
Disagree
18.32%
Strongly Disagree
9.16%
Very Strongly
Disagree
2.29%
0%
10%
Confidential – For Discussion Purposes Only
20%
30%
40%
50%
60%
70%
80%
90%
100%
Q12. The CCC process for trainee milestone evaluations is worth the
time/effort.
Very Strongly
Agree
6.92%
Strong Agree
16.92%
Agree
40.00%
Disagree
21.54%
Strongly Disagree
6.15%
Very Strongly
Disagree
8.46%
0%
10%
Confidential – For Discussion Purposes Only
20%
30%
40%
50%
60%
70%
80%
90%
100%
Q12. Why not worth the time: Emergent Themes
“already doing it in other ways”
“CCC does not tell me more about my
residents than I already know.”
Confidential – For Discussion Purposes Only
Q12. Why not worth the time: Emergent Themes
Cumbersome and Esoteric Milestones
Confidential – For Discussion Purposes Only
Q12. Why not worth the time: Emergent Themes
No prior research, testing for validity
Confidential – For Discussion Purposes Only
40
Q12. Why not worth the time: Emergent Themes
PDs need educational mission support.
Confidential – For Discussion Purposes Only
Other thoughts from Coordinators: Emergent themes
 Approaches to CCC faculty review process
 Recognizing time-consuming nature of work and need for support
 “Food for thought” for ACGME from program coordinators
Confidential – For Discussion Purposes Only
Overall Results Summary
 Overall Improved Evaluation and Documentation
− Semi-annual evaluations improved
− Documentation of trainee strengths and weaknesses
 Time Consuming
− Need leadership support
− Administrative Resources
− Ample planning time
 CCC Membership Size
− Share the burden!
− Value of diverse perspectives
 Faculty Roles
− Responsibilities
 Pairing faculty with trainees
 Assigning faculty to become “milestone” experts
Confidential – For Discussion Purposes Only
Lessons Learned / Best Practices
It’s your turn and your voice that counts!!!
How might you change your practice in light
of these findings?
Confidential – For Discussion Purposes Only
Questions?
Confidential – For Discussion Purposes Only
45
Contact Information:
 Nancy Piro, Program Manager/Education Specialist
− npiro@stanford.edu
 Kim Walker, Instructional Designer
− kwalker5@stanford.edu
Confidential – For Discussion Purposes Only