All Rights Reserved, Duke Medicine 2007
Clinical Competency Committees:
Transitioning our Learners in GME
2013 Graduate Medical Education Conference
Predicting Effective Transitions to the NAS
Palm Desert, CA – October 24, 2013
Kathryn Andolsek MD MPH
Kathryn.andolsek@duke.edu
All Rights Reserved, Duke Medicine 2007
An Institutional Question?
Are trainees at Loma Linda achieving the
competency required for unsupervised
practice that includes the delivery of safe,
timely, equitable, effective and patientcentered care?
All Rights Reserved, Duke Medicine 2007
“
All Rights Reserved, Duke Medicine 2007
So how do you
know?”
Objectives
1. Optimize your CCC for learners, your program and
institution
2. Use evidence from medical education research to
make best choices for CCC
3. Conduct a gap analysis and determine your Next
Steps to strengthen your CCC
All Rights Reserved, Duke Medicine 2007
All Rights Reserved, Duke Medicine 2007
Competency Based
Education
All Rights Reserved, Duke Medicine 2007
Dreyfus and Dreyfus Model
Expert/
Master
Proficient
Competent
Novice
Advanced Beginner
Time, Practice, Experience
Dreyfus SE and Dreyfus HL. A 1980
Carraccio CL et al. Acad Med 2008;83:761-7
All Rights Reserved, Duke Medicine 2007
Milestones and
Trajectories “D” “E”
Fully Competent
A’
A
D
B
C
E
Start
PGY1
All Rights Reserved, Duke Medicine 2007
Milestones
Finish
PGY3
Lucey and
Boote
Clinical Competency Committee (CCC)
• A group of faculty (and others) trained in knowing learner expectations
and the understanding of evaluations
• Uniformly monitor trainee progression through the continuum of
education using objective measures
• assist the PD with the assessment of competency, need for
remediation, and feedback
Page 10
All Rights Reserved, Duke Medicine 2007
All Rights Reserved, Duke Medicine 2007
Participation
CCC Quiz
All Rights Reserved, Duke Medicine 2007
Accreditation Council for Graduate Medical Education (ACGME)
Requirements/Guidance
Frequency Asked Questions
Published December 2012
Access at http://www.acgme-
Proposed Common Program
Requirements Section V
Comments accepted thru May
2013
nas.org/assets/pdf/NASFAQs.pdf
Access at
FAQ Addresses:
1.
2.
3.
4.
5.
Role of CCC in resident assessment
Qualifications for CCC membership
Preparation for CCC members
Milestones and the CCC
CCC and Remediation
ACGME NAS Webinar
All Rights Reserved, Duke Medicine 2007
http://www.acgme.org/acgmeweb/Portals/0/PFAss
ets/ProgramRequirements/CPR_Impact.pdf
New CPR 2016
http://www.acgme.org/acgm
eweb/Portals/0/PFAssets/Pr
ogramRequirements/CPRs
_07012016_TCC.pdf
1. The ACGME expects each program to
form a CCC and develop its’ members by:
A. 2013
B. 2014
C. When the specialty begins to require milestones
D. The time of the first C.L.E.R. visit
E. Before the next Program Site visit
F. Before the next “Internal Review”
G. None of the above
All Rights Reserved, Duke Medicine 2007
1. The ACGME expects each program to
form a CCC and develop its’ members by:
A. 2013
B. 2014
C. When the specialty begins to require milestones
D. The time of the first C.L.E.R. visit
E. Before the next Program Site visit
F. Before the next “Internal Review”
G. None of the above
All Rights Reserved, Duke Medicine 2007
CCC implementation required by…..
Proposed CPR Impact Statement:
“The proposed effective date for these revisions is July 1, 2013. However, programs in
Phase II of NAS will not be expected to be in full compliance with new requirement until July
1, 2014.”
FAQ:
ACGME Programs expected to form a CCC by June 2013
FAQ Addresses:
1. Role of CCC in resident assessment
2. Qualifications for CCC membership
3. Preparation for CCC members
4. Milestones and the CCC
5. CCC and Remediation
All Rights Reserved, Duke Medicine 2007
Accessed February 20th 2013
at http://www.acgmenas.org/assets/pdf/NASFAQs.
pdf
2. The minimum number of CCC
members should be
A.
B.
C.
D.
2
3
4
5
All Rights Reserved, Duke Medicine 2007
2. The minimum number of CCC
members should be
A.
B.
C.
D.
2
3
4
5
All Rights Reserved, Duke Medicine 2007
Common Program Requirements 7-1-2013
All Rights Reserved, Duke Medicine 2007
http://www.acgme.org/acgmeweb/Portals/0/PDFs/FAQ/CCC_PEC_FAQs.pdf
All Rights Reserved, Duke Medicine 2007
3. The CCC members….
A.
B.
C.
D.
E.
Vote on each resident’s performance
Provide a consensus on each resident
Only consider residents who need remediation plans
Only review residents in their last year of training
None of the above
All Rights Reserved, Duke Medicine 2007
3. The CCC members….
A. Vote on each resident’s performance
B. Provide a consensus on each resident
C. Only consider residents who needs remediation
plans
D. Only review residents in their last year of training
E. None of the above
All Rights Reserved, Duke Medicine 2007
Common Program Requirements 7-1-2013
All Rights Reserved, Duke Medicine 2007
Common Program Requirements 7-1-2016
All Rights Reserved, Duke Medicine 2007
Role of CCC in resident assessment
• Members make consensus decision on the progress of
each resident
• CCC utilizes:
– Existing resident assessment data
– Faculty observations
– Data from milestone assessment (beginning July 2013)
Accessed February 20th 2013 at http://www.acgmenas.org/assets/pdf/NASFAQs.pdf
All Rights Reserved, Duke Medicine 2007
4. The CCC must include:
A.
B.
C.
D.
E.
F.
G.
All Rights Reserved, Duke Medicine 2007
Patients
Nurses
Peer selected residents
Core faculty members
Program director
All of the above
C, D, and E
4. The CCC must include:
A.
B.
C.
D.
E.
F.
G.
All Rights Reserved, Duke Medicine 2007
Patients
Nurses
Peer selected residents
Core faculty members
Program director
All of the above
C, D, and E
Common Program Requirements 7-1-2013
All Rights Reserved, Duke Medicine 2007
Qualifications for CCC members
• faculty members
• Those who observe and evaluate residents; willing to
make hard decisions
• Others such as assessment specialists, non-MD
educators, resident (?)
• Webinar: Advisors/mentors excluded from discussion
All Rights Reserved, Duke Medicine 2007
http://www.acgme.org/acgmeweb/Portals/0/PDFs/FAQ/CCC_PEC_FAQs.pdf
7-19-2013
All Rights Reserved, Duke Medicine 2007
5. According to the ACGME, faculty development
needed by the CCC members includes:
A. Knowing their potential legal liability
B. Giving “bad news” to the resident after the milestone determination has
been reached
C. Reaching a common agreement of milestones narrative meaning
D. Determining how many assessments are needed for any given
milestones
E. Applying QI improvement principles to the evaluation process
F. Knowing the best remediation strategies for certain milestone
suboptimal performance
G. All of the above
H. None of the above
All Rights Reserved, Duke Medicine 2007
5. According to the ACGME, faculty development
needed by the CCC members includes:
A. Knowing their potential legal liability
B. Giving “bad news” to the resident after the milestone determination has
been reached
C. Reaching a common agreement of milestones narrative meaning
D. Determining how many assessments are needed for any given
milestones
E. Applying QI improvement principles to the evaluation process
F. Knowing the best remediation strategies for certain milestone
suboptimal performance
G. All of the above
H. None of the above
All Rights Reserved, Duke Medicine 2007
Faculty Development for CCC members
(…plans to develop training resources for CCCs)
• Evaluation is a core faculty competency but most will need
additional training in evaluation process
• Discuss milestone narratives & reach common agreement of meaning
• Evaluation process training:
– How to aggregate data
– How to interpret evaluation data
• Venue for faculty member discussion about resident evaluation:
– How many assessment needed for each milestone
– Data quality
– Application of QI principles as part of evaluation process
Accessed February 20th, 2013 at http://www.acgmenas.org/assets/pdf/NASFAQs.pdf
All Rights Reserved, Duke Medicine 2007
6. A resident rotates on another specialty
service. That specialist evaluates them as
performing poorly. The CCC should….
A. Use the grade provided by the specialist as their
grade in assessing resident performance against
the milestones
B. Not consider the evaluation as it came from a
different specialty than the program
C. Take the evaluation and apply it with other data to
the resident’s program milestones
D. Vote whether the evaluation seems accurate and
should be included in the overall review of the
resident’s performance
All Rights Reserved, Duke Medicine 2007
6. A resident rotates on another specialty
service. That specialist evaluates them as
performing poorly. The CCC should….
A. Use the grade provided by the specialist as their
grade in assessing resident performance against
the milestones
B. Not consider the evaluation as it came from a
different specialty than the program
C. Take the evaluation and apply it with other data to
the resident’s program milestones
D. Vote whether the evaluation seems accurate and
should be included in the overall review of the
resident’s performance
All Rights Reserved, Duke Medicine 2007
Milestones and the CCC
• CCC will take data from evaluations and apply them to
the milestones to mark progress of residents.
• The CCC will have the advantage of knowing how each
of the specialists evaluated residents and can apply that
knowledge as it marks the residents’ progress on the
milestones.
Accessed February 20th 2013 at http://www.acgmenas.org/assets/pdf/NASFAQs.pdf
All Rights Reserved, Duke Medicine 2007
7. Pilot assessments on the milestones have
found that the first time the evaluation is done, it
takes approximately _____for each resident.
A.
B.
C.
D.
E.
F.
5 minutes
10 minutes
20 minutes
30 minutes
45 minutes
Up to an hour
All Rights Reserved, Duke Medicine 2007
7. Pilot assessments on the milestones have
found that the first time the evaluation is done, it
takes approximately ____ for each resident.
A.
B.
C.
D.
E.
F.
5 minutes
10 minutes
20 minutes
30 minutes
45 minutes
Up to an hour
All Rights Reserved, Duke Medicine 2007
Time required: milestones and the CCC
• It takes a significant amount of time (up to an hour) to
conduct the milestones assessment for each resident the
first time the evaluation is done.
• Subsequent evaluations take less time.
Accessed February 20th 2013 at http://www.acgmenas.org/assets/pdf/NASFAQs.pdf
All Rights Reserved, Duke Medicine 2007
Remediation and the CCC
• Goal: identify residents who are not progressing
• CCC sets performance thresholds (until national benchmark data
available)
• For early years - compare resident performance on milestones to peers
in program
• When deficiency identified, interventions considered (may or may not
be role of CCC?)
–
–
–
–
–
–
Mentor
Additional readings
Skill lab experiences
Added rotations
Accessed February 20th 2013 at http://www.acgmenas.org/assets/pdf/NASFAQs.pdf
Extend education
Counseling to consider another specialty/profession
All Rights Reserved, Duke Medicine 2007
Benefits of CCC
•
For the resident
– Offers insight and perspectives of a GROUP of faculty
– Serves as an early warning system if resident fails to progress
(opportunity for remediation)
•
For the program director
– CCC outcomes shared with resident so Program Director can be
advocate for resident learning
•
For the faculty/program
– Utilizes objective assessment measures making it easier for
evaluators
– Chance to simplify assessment tools; evaluation system
– Opportunities to improve programs - curriculum, rotation schedules,
supervision, mentorship (educational QI)
All Rights Reserved, Duke Medicine 2007
Evidence based: What the literature says….
•
Group conversations more likely to uncover deficiencies in professionalism for students (Hemmer
et al 2000)
•
Group assessments improve inter-rater reliability and reduced range restriction in multiple
domains (Thomas et al 2011)
•
CCC helps ensure decisions are not arbitrary or capricious AND serves as checks and balances
(especially identifying marginal resident) (Wu et al 2010, Schwind et al 2004, Williams et al 2005)
•
No evidence (that) individuals dominate discussions or gang up (Williams et al 2005)
•
18% of resident deficiencies requiring remediation only became apparent through group
discussion; average discussion 5 minutes per resident, range 1-30 minutes (Schwind et al 2001)
•
Amount of time spent discussing a resident is frequently a measure of the severity of the
problem/concern (Sanfrey et al 2012)
All Rights Reserved, Duke Medicine 2007
CCC: The Legal & Practical
Legal precedent suggests Courts will support educational institutions decisions
about resident progression and graduation when:
•
•
•
Decisions are reasonable (e.g. made at routine faculty meetings)
There is a shared understanding of expectations
Residents with performance issues are given Notice (informed of deficiency)
and Opportunity to Cure (improve)
(Missouri v. Horowitz 1978)
•
•
Decisions are made “conscientiously and with careful deliberation”
Taking into account learners entire record
(Univ. of Michigan v. Ewing 1985)
Are your CCC proceedings and documentation “peer review protected”?
All Rights Reserved, Duke Medicine 2007
CCC Components, Practical Processes & Procedures

Membership: How many members? Who is eligible/optimal? For how long?

Who will serve as the CCC Chair and for how long?

When, how often and how long should the CCC meet?

What is the role and responsibility for each member?

How will learner expectations be determined? What are the shared set of standards/thresholds?

What are objective (& varied) measures of performance? What assessments (existing or future) could
be used?

How are findings determined?

How will findings be conveyed to Program Director? (who makes ultimate decision)

How/by whom will findings be conveyed to learners?

What happens when CCC determines learner is performing sub-optimally?

What faculty development is required for CCC members?

How will you describe this process to your program?
All Rights Reserved, Duke Medicine 2007
All Rights Reserved, Duke Medicine 2007
Pre-Conference Survey Results
• Electronic
survey link sent
on February 19th
• 123/270
responded
(46%)
• THANK YOU!
All Rights Reserved, Duke Medicine 2007
All Rights Reserved, Duke Medicine 2007
Think about…CCC Components, Processes & Procedures
 Membership: How many members? Who is eligible/optimal? For how long?
All Rights Reserved, Duke Medicine 2007
All Rights Reserved, Duke Medicine 2007
All Rights Reserved, Duke Medicine 2007
All Rights Reserved, Duke Medicine 2007
Think about…CCC Components, Processes & Procedures
 Membership: How many members? Who is eligible/optimal? For how long?
Evidence Based: What the literature says…..
No single individual should make judgments about the competence of a
trainee in isolation (Swing et al 2010)
assessment in competency based medical education benefits from the “wisdom of
a group process in making judgment about trainee progress”
(Holmboe et al 2010)
Group conversations more likely to uncover deficiencies in professionalism
(Hemmer 2001)
Group assessment improved inter-rater reliability and reduced range restriction in
multiple domains in an internal medicine residency (Thomas 2011)
5-7 members most effective – “balances out idiosyncrasies in judges
(Williams et al 2012)
Maybe at least 3….
All Rights Reserved, Duke Medicine 2007
ratings”
Think about…CCC Components, Processes & Procedures

Membership: How many members? Who is eligible/optimal? For how long?

Who will serve as the CCC Chair and for how long?
All Rights Reserved, Duke Medicine 2007
All Rights Reserved, Duke Medicine 2007
Think about…CCC Components, Processes & Procedures

Membership: How many members? Who is eligible/optimal? For how long?

Who will serve as the CCC Chair and for how long?
Consider:
 Maybe NOT the program director ???
 Possibly the length of the training program
All Rights Reserved, Duke Medicine 2007
Think about…CCC Components, Processes & Procedures

Membership: How many members? Who is eligible/optimal? For how long?

Who will serve as the CCC Chair and for how long?

When, how often and how long should the CCC meet?
All Rights Reserved, Duke Medicine 2007
All Rights Reserved, Duke Medicine 2007
Think about…CCC Components, Processes & Procedures

Membership: How many members? Who is eligible/optimal? For how long?

Who will serve as the CCC Chair and for how long?

When, how often and how long should the CCC meet?
Evidence Based: What the literature says…..
The role of assessment in competency based medical education benefits from
processes that are more continuous and frequent
(Holmboe et al 2010)
So, what does “that mean?”
All Rights Reserved, Duke Medicine 2007
Think about…CCC Components, Processes & Procedures

Membership: How many members? Who is eligible/optimal? For how long?

Who will serve as the CCC Chair and for how long?

When, how often and how long should the CCC meet?

What is the role and responsibility for each member?
All Rights Reserved, Duke Medicine 2007
Think about…CCC Components, Processes & Procedures

Membership: How many members? Who is eligible/optimal? For how long?

Who will serve as the CCC Chair and for how long?

When, how often and how long should the CCC meet?

What is the role and responsibility for each member?
Consider:
 Complete confidentiality – nothing discussed outside of CCC
 No delusions of power
 Committed to objective assessment of each learner
All Rights Reserved, Duke Medicine 2007
Think about…CCC Components, Processes & Procedures

Membership: How many members? Who is eligible/optimal? For how long?

Who will serve as the CCC Chair and for how long?

When, how often and how long should the CCC meet?

What is the role and responsibility for each member?

How will learner expectations be determined? What are the shared set of standards/thresholds?
All Rights Reserved, Duke Medicine 2007
Think about…CCC Components, Processes & Procedures

Membership: How many members? Who is eligible/optimal? For how long?

Who will serve as the CCC Chair and for how long?

When, how often and how long should the CCC meet?

What is the role and responsibility for each member?

What are the shared set of standards/thresholds? How will learner expectations be determined?
Evidence Based: What the literature says….
• Competency based medical education requires developmental criteria = milestones,
learning objectives or benchmarks
• Faculty should acquire shared mental models and understanding on what
competence should look like at various developmental stages
(Holmboe et al 2011)
All Rights Reserved, Duke Medicine 2007
Think about…CCC Components, Processes & Procedures

Membership: How many members? Who is eligible/optimal? For how long?

Who will serve as the CCC Chair and for how long?

When, how often and how long should the CCC meet?

What is the role and responsibility for each member?

How will learner expectations be determined? What are the shared set of standards/thresholds?
 What are objective (& varied) measures of performance? What assessments
(existing or future) could be used?
All Rights Reserved, Duke Medicine 2007
Think about…… CCC Components, Processes & Procedures

Membership: How many members? Who is eligible/optimal? For how long?

Who will serve as the CCC Chair and for how long?

When, how often and how long should the CCC meet?

What is the role and responsibility for each member?

How will learner expectations be determined? What are the shared set of standards/thresholds?
 What are objective (& varied) measures of performance? What assessments
(existing or future) could be used?
Evidence Based: What the literature says….
Assessment must:
• Be based on authentic encounters and frequent direct observation
(Holmboe et al 2010; Carraccio et al 2002; Williams et al 2003; Govaerts et al 2007)
• Move away from just numbers; greater reliance on qualitative assessment
(Govaerts et al 2007)
All Rights Reserved, Duke Medicine 2007
Potential Assessment Measures
• Use a variety
• Include direct observation
• Possibly 10? What “10” do you use?
All Rights Reserved, Duke Medicine 2007
Potential Assessment Measures
•
•
Use a variety
Possibly 10?

















Rotation evaluations (by attending - direct observation for adequate time)
Multi-rater (360) peer
Multi-rater (360) nurse
Multi-rater (360) patient
Research project project
Teaching skills evaluation
Self-assessment
Procedure/patient log summary
Patient care outcomes (more difficult - maybe chart audit of outcomes or performance services
data)
projecct
Handoff evaluation
Standardized patient
video
Simulation observation
Milestones achievement
Performance improvement project
Compliance items
All Rights Reserved, Duke Medicine 2007
All Rights Reserved, Duke Medicine 2007
All Rights Reserved, Duke Medicine 2007
Consider a Dashboard
All Rights Reserved, Duke Medicine 2007
All Rights Reserved, Duke Medicine 2007
Think about…CCC Components, Processes & Procedures

Membership: How many members? Who is eligible/optimal? For how long?

Who will serve as the CCC Chair and for how long?

When, how often and how long should the CCC meet?

What is the role and responsibility for each member?

How will learner expectations be determined? What are the shared set of standards/thresholds?

What are objective (& varied) measures of performance? What assessments (existing or future)
could be used?
 How are findings determined?
All Rights Reserved, Duke Medicine 2007
Think about…CCC Components, Processes & Procedures

Membership: How many members? Who is eligible/optimal? For how long?

Who will serve as the CCC Chair and for how long?

When, how often and how long should the CCC meet?

What is the role and responsibility for each member?

How will learner expectations be determined? What are the shared set of standards/thresholds?

What are objective (& varied) measures of performance? What assessments (existing or future) could
be used?
 How are findings determined?
Consider:
 It’s a consensus - not a vote
 It’s a RECOMMENDATION to the program director
All Rights Reserved, Duke Medicine 2007
All Rights Reserved, Duke Medicine 2007
Think about…CCC Components, Processes & Procedures

Membership: How many members? Who is eligible/optimal? For how long?

Who will serve as the CCC Chair and for how long?

When, how often and how long should the CCC meet?

What is the role and responsibility for each member?

How will learner expectations be determined? What are the shared set of standards/thresholds?

What are objective (& varied) measures of performance? What assessments (existing or future) could
be used?

How are findings determined?
 How will findings be conveyed to PROGRAM DIRECTOR? (who makes the
ultimate decision)
 How are findings conveyed to ACGME?
 How are findings “retained” by the Program (or should they?)
 How will findings be conveyed to the LEARNERS?
All Rights Reserved, Duke Medicine 2007
Think about…CCC Components, Processes & Procedures

Membership: How many members? Who is eligible/optimal? For how long?

Who will serve as the CCC Chair and for how long?

When, how often and how long should the CCC meet?

What is the role and responsibility for each member?

How will learner expectations be determined? What are the shared set of standards/thresholds?

What are objective (& varied) measures of performance? What assessments (existing or future) could be
used?

How are findings determined?
 How will findings be conveyed to Program Director? (who makes ultimate decision)
 How will findings be conveyed to the resident(s)?
Evidence Based: What the literature says…..
Formative and summative assessment is critical to resident success
(Carraccio et al 2002)
Faculty must become less fearful of providing meaningful (truthful) performance data
(Holmboe et al 2011)
All Rights Reserved, Duke Medicine 2007
Think about…CCC Components, Processes & Procedures

Membership: How many members? Who is eligible/optimal? For how long?

How will learner expectations be determined? What are the shared set of standards/thresholds?

What are objective (& varied) measures of performance? What assessments (existing or future) could
be used?

When, how often and how long should the CCC meet?

Who will serve as the CCC Chair and for how long?

What is the role and responsibility for each member?

How are findings determined?

How will findings be conveyed to Program Director? (who makes ultimate decision)

How/by whom will findings be conveyed to learners?
 What happens when CCC determines resident is performing sub-optimally?
All Rights Reserved, Duke Medicine 2007
Think about…CCC Components, Processes & Procedures

Membership: How many members? Who is eligible/optimal? For how long?

How will learner expectations be determined? What are the shared set of standards/thresholds?

What are objective (& varied) measures of performance? What assessments (existing or future) could be
used?

When, how often and how long should the CCC meet?

Who will serve as the CCC Chair and for how long?

What is the role and responsibility for each member?

How are findings determined?

How will findings be conveyed to Program Director? (who makes ultimate decision)

How/by whom will findings be conveyed to learners?
 What happens when CCC determines resident is performing sub-optimally?
Consider:
Collection of performance data for high-stakes use could adversely affect the
accuracy of resident assessment and the usefulness of the results for formative
purposes within a program (Swing et al 2009)
CCC may NOT be the venue for remediation and corrective action plans
All Rights Reserved, Duke Medicine 2007
Think about…CCC Components, Processes & Procedures

Membership: How many members? Who is eligible/optimal? For how long?

How will learner expectations be determined? What are the shared set of standards/thresholds?

What are objective (& varied) measures of performance? What assessments (existing or future) could
be used?

When, how often and how long should the CCC meet?

Who will serve as the CCC Chair and for how long?

What is the role and responsibility for each member?

How are findings determined?

How will findings be conveyed to Program Director? (who makes ultimate decision)

How/by whom will findings be conveyed to learners?

What happens when CCC determines learner is performing sub-optimally?
 What faculty development is required for CCC members?
All Rights Reserved, Duke Medicine 2007
Think about…CCC Components, Processes & Procedures
 What faculty development is required for CCC members?
Evidence Based: Literature says…..
• “assessor training is a key component of high quality assessment in residency
programs…evidence to suggest that faculty can be trained to improve the
quality of their assessments”
(Dudak et al 2012)
• Faculty must build competency in teaching and assessment
(Holmboe et al 2011)
•
Faculty need:
 Feedback on range of scores compared with co-evaluators
 Basic knowledge of core psychometric concepts
 Time to learn and practice new teaching/assessment skills
(Holmboe et al 2011)
All Rights Reserved, Duke Medicine 2007
Think about…CCC Components, Processes & Procedures

Membership: How many members? Who is eligible/optimal? For how long?

When, how often and how long should the CCC meet?

Who will serve as the CCC Chair and for how long?

What is the role and responsibility for each member?

How will learner expectations be determined? What are the shared set of standards/thresholds?

What are objective (& varied) measures of performance? What assessments (existing or future)
could be used?

How are findings determined?

How will findings be conveyed to Program Director? (who makes ultimate decision)

How/by whom will findings be conveyed to learners?

What happens when CCC determines learner is performing sub-optimally?

What faculty development is required for CCC members?
 How will you describe this process to your program?
All Rights Reserved, Duke Medicine 2007
Think about…CCC Components, Processes & Procedures

Membership: How many members? Who is eligible/optimal? For how long?

When, how often and how long should the CCC meet?

Who will serve as the CCC Chair and for how long?

What is the role and responsibility for each member?

How will learner expectations be determined? What are the shared set of standards/thresholds?

What are objective (& varied) measures of performance? What assessments (existing or future)
could be used?

How are findings determined?

How will findings be conveyed to Program Director? (who makes ultimate decision)

How/by whom will findings be conveyed to learners?

What happens when CCC determines learner is performing sub-optimally?

What faculty development is required for CCC members?
 How will you describe this process to your program?
 Description in program manual (for faculty and learners)
 Discussed at GMEC, new resident/fellow orientation, faculty meetings
All Rights Reserved, Duke Medicine 2007
1.
CCC:
Next
Steps
Select members of CCC
2. Develop common expectations of CCC Role
3. Learn, develop or borrow milestones (resident expectations)
4. Define/refine (simplify!) assessment process
– Evaluation tools (attempt to identify at least 10 assessments for each learner)
– Who are evaluators (ample opportunity to observe learner performance)
– Guidance for evaluators
5. Identify if anything missing and fill in the gaps
6. Write a CCC description so transparent to all (who, what, when, why)
7. CCC Members train/learn together (faculty development)
8. Use experience to improve the process (It’s educational CQI!)
All Rights Reserved, Duke Medicine 2007
Survey Results:
Reported Best Practices
•
CCC "assigned" to specific PGY, so they become "experts" at that level
•
Everyone on committee works with residents with first hand knowledge of
resident abilities
•
generous amount of qualitative and quantitative multi-sourced evaluations for all
fellows to allow us to compare across training level & by the program as a whole
•
Each member of the committee has expertise in separate subspecialties and
recognizes skill or deficiencies in their respective fields of interest
•
Each resident knows the outcome within a day of the meeting and the advisers
and PD are aware of any problems or concerns
•
Combine small programs with CCC of the core
•
Make a faculty development opportunity
•
Use RMT
All Rights Reserved, Duke Medicine 2007
Potential Enhancements To Your CCC
•
Use CCC forum as opportunity for faculty development (content in addition to
resident evaluation). Read and discuss medical education journal article, latest
ACGME Common Program Requirements, GME Program Goals.
•
Identify thresholds for acceptable performance in areas such as compliance.
(e.g. < 10% duty hour violation; < 95% hospital web module completion, < 85%
conference attendance)
•
For high performing residents with no questionable evaluations, the CCC can
simply confirm acceptable performance and instead spend most of its’ time
reviewing and discussing those residents with questionable performance who
are potentially not meeting expectations. (but include all)
•
Identify critical times for CCC to meet. e.g. Jan-February if contract issues and
scheduling must be determined for the next academic year; prior to Program
Director’s semi-annual review with residents.
All Rights Reserved, Duke Medicine 2007
Potential Enhancements To Your CCC
•
Frequency? Can the CCC can meet more frequently for a brief review of all residents
and review in detail the performance of one PGY level cohort each time.
•
Combined programs (e.g. med-psych and med-peds) consider having their own CCC
and use information from each Core CCC on resident performance.
•
Include a description of CCC in the resident manual (and distribute to faculty)
•
Include a professional staff healthcare team member who works closely with
residents on CCC (e.g. social worker, nurse practitioner)
•
Use CCC members to revise and simplify evaluation system (every evaluation form
does not need to cover every competency, every rotation does not need to cover
every milestone, etc)
•
Have resident advisors (where applicable) prepare a presentation for the CCC on
their advisee’s performance utilizing a variety of assessment tools.
•
Evaluate it!
All Rights Reserved, Duke Medicine 2007
Other Best Practices?
All Rights Reserved, Duke Medicine 2007
From a list serve…
“I am beginning the process of negotiating the budget for our CCC
committees I’m trying to get a sense of what programs have
received….
Our ask to the hospital so far We have asked for 0.006 Faculty FTE’s and
0.003 coordinator FTE’s per resident plus a TBD startup bolus of hours.
These numbers were arrived at based on the NIH type model we plan to
use and a best-guess at time for faculty to review data.”
XXXXX
Program Director
All Rights Reserved, Duke Medicine 2007
Participation
Given what you know now, what is the first thing you
will do to enhance your CCC?
All Rights Reserved, Duke Medicine 2007
References
• Albanese MA. Challenges in using rater judgments in medical education. J Eval Clin Pract. 2000;6:305–19.
• Baker K. Determining resident clinical performance: Getting beyond the noise. Anesthesiology.
2011;115(4):862-878.
• Carr SJ. Assessing clinical competency in medical houseofficers: how and why should we do it? Postgrad
Med J. 2004;80(940):63-6.
• Carraccio C, Wolfsthal SD, Englander R, Ferentz K, Martin C. Shifting paradigms: From Flexner to
competencies. Acad Med. 2002; 77(5):361-367.
• David DA, Mazmanian PE, Fordis M, Van Harrison R, Thorpe KE, Perrier L. Accuracy of physician selfassessment compared with observed measures of competence: A systematic review. JAMA.
2006;296(9):1094-102.
• Downing SM. Threats to the validity of clinical teaching assessments: What about rater error? Med Educ.
2005;39:353–5
• Dudek NL, Marks MB, Wood TJ, Dojeiji S, Bandiera G, Hatala R, Cooke L, Sadownik L. Quality evaluation
reports: Can a faculty development program make a difference? Med Teach. 2012;34:e725-e731.
• Ginsburg S, McIlroy J, Oulanova O, Eva K, Regehr G. Toward authentic clinical evaluation: Pitfalls in the
pursuit of competency. Acad Med. 2010;85:780–6.
• Greaves JD, Grant J: Watching anesthetists work: Using the professional judgment of consultants to assess
the developing clinical competence of trainees. Br J Anaesth. 2000;84:525–33
• Govaerts MJ, van der Vleuten CP, Schuwirth LW, Muijtiens AM. Broadening perspectives on clinical
performance assessment: Rethinking the nature of in-training assessment. Adv Health Sc Educ Theory
Pract. 2007;12(2):239-260.
• Hamdy H, Prasad K, Anderson MB, Scherpbier A, Williams R, Zwierstra R, Cuddihy H. BEME systematic
review: Predictive values of measurements obtained in medical schools and future performance in medical
practice. Med Teach. 2006;28:103–16
• May 16.
All Rights Reserved, Duke Medicine 2007
References
• Hattie J, Timperley H. The power of feedback. Rev Educ Res. 2007;77(1):81-112.
• Haur KE, Mazotti L, O’Brien B, Hemmer PA, Tong L. Faculty verbal evaluations reveal strategies used to
promote medical student performance. Med Educ Online. 2011; 10.3402/meo.v16i0.6354. Epub 2011
Hemmer PA, Hawkins R, Jackson JL. Assessing how well three evaluation methods detect deficiencies in
medical students' professionalism in two settings of an internal medicine clerkship. Acad Med.
2000;75(2):167-73
• Herbers JE Jr, Noel GL, Cooper GS, Harvey J, Pangaro LN, Weaver MJ. How accurate are faculty
evaluations of clinical competence? J Gen Intern Med. 1989;4:202–8
• Holmboe ES: Faculty and the observation of trainees' clinical skills: Problems and opportunities. Acad
Med. 2004;79:16–22
• Holmboe ES, Hawkins RE. Methods for revaluating the clinical competence of residents in internal
medicine: a review. Ann Intern Med. 1998;129(1):42-8.
• Holmboe ES, Sherbino J, Long DM, Swing SR, Frank JR. The role of assessing in competency-based
medical education. Med Teach. 2010;32:676-682.
• Holmboe ES, Ward DS, Reznick RK, Katsufrakis PJ, Leslie KM, Patel VL, Ray DD, Nelson EA. Faculty
Development in Assessment: Noel GL, The missing link in competency-based medical education. Acad
Med. 2011;86(4):460-467.
• Herbers JE Jr, Caplow MP, Cooper GS, Pangaro LN, Harvey J. How well do internal medicine faculty
members evaluate the clinical skills of residents? Ann Intern Med. 1992;117:757–65.
• Issenberg SB, McGaghie WC, Waugh RA. Computers and evaluation of clinical competence. Ann Intern
Med. 1999;130(3):244-5.
• Kogan JR, Holmboe ES, Hauer KE. Tools for direct observation and assessment of clinical skills of medical
trainees: A systematic review. JAMA. 2009;302:1316–26..
All Rights Reserved, Duke Medicine 2007
References
• Langsley DG. Medical competence and performance assessment. A new era. JAMA. 1991;266(7):977-80.
• Lurie SJ, Mooney CJ, Lyness JM. Measurement of the general competencies of the accreditation council for
graduate medical education: A systematic review. Acad Med. 2009;84:301–9Miller A, Archer J. Impact of
workplace based assessment on doctors’ education and performance: a systematic review. BMJ. 2010;
341:c5064.
• Nasca TJ, Philibert I, Brigham T, Flynn TC. The Next Accreditation System – Rationale and Benefits. N Engl
J Med. 2012;366(11):1051-1056.
• Pangaro L. A new volcabulary and other innovations for improving descriptive in-training evaluations. Acad
Med.1999;74(11):1203-7.
• Scavone BM, Sproviero MT, McCarthy RJ, Wong CA, Sullivan JT, Siddall VJ, Wade LD. Development of an
objective scoring system for measurement of resident performance on the human patient simulator.
Anesthesiology. 2006;105:260–6.
• Stillman PL, Swanson DB, Smee S, Stillman AE, Ebert TH, Emmel VS, Caslowitz J, Greene HL, Hamolsky
M, Hatem C, Levenson DJ, Levin R, Levinson G, Ley B, Morgan GJ, Parrino T, Robinson S, Willms J.
Assessing clinical skills of residents with standardized patients. Ann Intern Med. 1986;105:762–71.
• Swing SR, Clyman SG, Holmboe ES, Williams RG. Advancing Resident Assessment in Graduate Medical
Education. J Grad Med Educ. 2009;1(2):278-86.
• Swing SR; International CBME Collaborators. Perspectives on competency-based medical education from
the learning sciences. Med Teach. 2010;32(8):663-8.
• Wilkinson JR, Crossley JG, Wragg A, Mills P, Cowan G, Wade W. Implementing workplace-based
assessment across the medical specialties in the United Kingdom. Med Educ. 2008;42(4):364-73.
• Williams RG, Klamen DA, McGaghie WC. Cognitive, social and environmental sources of bias in clinical
performance ratings. Learn Med. 2003;15(4):270-292.
• Williams RG, Sanfey H, Chen X, Dunnington GL. A controlled study to determine measurement conditions
necessary for a reliable and valid operative performance assessment. Annals of Surg. 2012;256(1):177-187.
All Rights Reserved, Duke Medicine 2007