NOTES FROM
SEPTEMBER 19, 2013 NETWORK WORKSHOP
Lake Washington Rowing Club
Wading Knee-Deep into the Next Accreditation System (NAS): Tackling the
Clinical Competency Committees (CCC’s) and Milestones at Your Program
GOAL
Teams from Network Programs will leave the workshop informed about managing the
Family Medicine Milestones and CCC’s with an action plan for your program
OBJECTIVES
1. Discuss approaches to form a functional CCC
2. Identify curricular areas your program needs to augment to teach and evaluate
Milestones which are not currently addressed
3. Identify mechanisms for an efficient system to organize data for resident Milestones
assessment (such as New Innovations)
4. Review Faculty Development strategies for a functioning CCC and Milestones
assessment
5.
Leave with an action plan for your program’s team which includes assessment of
administrative resources needed for implementation by July 2014
MILESTONES – KEY TOPICS
Clinical Competency Committees – Ardis Davis







Billings, Montana – Lessons learned from participation as a national Beta-test
site
o 4 FP’s on the CCC allows for one can be absent
o Meet ½ day (longer too tiring)
o Average is 1 ½ hour per resident
o Prep time after meeting
o Time to meet together is major issue
Old evaluation tools are often inadequate
Old semi-annual eval feeds into the ccc
o Rolling evals vs two whole – program events (ACGME reposting?
Requirement)
Unique challenges with multi-site programs – need to schedule time for site
directors to participate in CCC with the main program
--------------------RTTs may need to use volunteer faculty for CCC; resources? Pay? Overwork?
Consider representative experiences to generalize to milestones
Faculty development is essential
---------------------



Advisor vs advisory committee
o Affects CCC committee makeup
o 3 core steady CCC members throughout year? Can it fluctuate throughout
the year? can only change one core per year
Other beta test site lessons
o core fac (x3) + guest
o smaller more frequent groups  rolling
o better  every month
 ACGME reporting windows may not be conducive to that
one process in place – advisory committee does pre-work of each resident then
presents to CCC – speeds up process
dedicated, blocked time
---------------------

Acute resident problem. Is this CCC business?
o Resident in difficulty is out of the purview of the CCC
 This is a Due process issue
---------------------




Consider rolling sessions (doing it all in 1 session is exhausting)
Okay to include PD & chief resident(s)
Working with 1 class at a time streamlines reviews
Advisors can be used to review/present each resident to CCC, prep time can
streamline CCC meetings
May not be worth prolonged discussions over 5 points
Best practice  have resident meet with coordinator prior to review to review
completeness of evaluation portfolios


Procedural competency & Milestones – Lisa Johnson and Bill Kriegsman
PICS # 15-16


Origin  BSQ: from an STFM meeting presentation by Tufts years ago
New procedure requirements on website (NOTE: the approved 9/2013 ACGMEFM Special Requirements do NOT include a list of required procedures. They
say each program must make their own list and ensure all residents achieve
competency in these by graduation.)
o Drove competency on individual procedures
 Required list of procedures being set by RC-FM
o Preset number of procedures before able to do each BSQ eval.
 For most BSQ’s residents decide when to be “tested” –when they
feel competent






Based on specific BSQ, simple vs complex
Some procedures may never be performed without direct
supervision as a resident
 BSQ evaluator does not give pointers or teach- they only observe
 Passing the BSQ allows PD to state resident is competent/
independent for that procedure in their final graduation letter.
o Can use videos as prep ( Pfeninger book as resource)
 Online learning tools available at UW Healthlinks website
Issue #1  getting evaluators to evaluate each procedure independently instead
of lumping procedure competency together
o number of evaluators (core faculty and community attending)
o electronic evals not as useful for real time feedback
 Felt that it was better to do this on paper than on-line
 On-line would reduce data entry but also delay feedback
o Can we publish a generic paper BSQ -helpful as a template?
Issue #2 – Some faculty are out of practice with rarely used procedures
Issue #3 - Getting certain required numbers for graduation
o Can be difficult to get some procedures (such as anoscopy, a required
procedure)
 Simulations
 Workshops
 Pre-procedure opportunities
o Programs need to be aware of the required procedures
o Hard and fast required number (for credentialing) difficult to get at times
 This is set by the program, not by the RC-FM
 Minimum exposure number used in the past
 Increased number does not = competence
 Residents tend to focus on number
Issue #4 BSQ = cultural change
o Took several years, but now the Olympia residents have adopted it well
Programs are encouraged to create new BSQs
 Looking for a way to make them widely available (they are on the
UW NDRL)
 Olympia is freely sharing their work with WWAMI, but due credit
should be given when reproducing their effort
Use of Technology: Mapping evaluation with Milestones – Rae Wright








If evaluation questions are written in milestones language w/level 1-5 grading
scale that matches, it’s much easier for the CCC to interpret, but requires a lot of
work to rewrite evals.
You can keep evaluations the same and tie/map each question to the milestones,
but CCC would have to interpret the level. New Innovations now has built in
mapping function up and running.
--------------------------Ideally we could get “tech” to do the work for us by building this into programs
such as NI, vs programs creating their own new evaluations using milestones.
There are multiple styles of milestones evaluations being developed by different
programs
Consider 2 sets of evals, some that are mapped, some that are rewritten to
match milestone language more directly.
----------------------------Recommendation: Create menu of evaluation styles with advantages and
disadvantages of each. Programs can then choose, share those tools for the
style that works best for them.
Faculty development for each would be needed for grading to be consistent
across evaluators.
Core faculty best suited for using milestones evaluations, if properly instructed to
prevent grade inflation and give best info to CCC for use in the semi-annual
assessments.
Faculty Development - facilitated by Jennifer Hoock and Judy Pauwels
Flip Chart Notes edited by Ardis Davis

Common Anxieties
o common understanding of:
 Core faculty
 Other faculty
o manpower/competing demands
o decentralization
 rural TTs – FacDev
 Community preceptors
o Faculty turnover
o Volunteers  motivation/support
o Efficient training
o Getting a common language
o Time!!!! including Protected Time
o M-clinic evals – longitudinal formative
o
o
o
o
o
Training of faculty
Faculty engagement or not?
who? Task/competence
resistance  paradigm change
Expertise/goals
 Organized approach – CLARITY
 Is this evidenced based?
o Resources
o Infrastructure support
 NI
 Staff

What’s been tried:
o dedicated “go-to” person – local expert
o import “export”
o committing FD time to this
 little bites
o develop shared lexicon (paradigm)
o mapping current evals (rotations, clinic/observe) to milestones
o select milestones appropriate for learning setting
o trialed with existing interns
o identified gaps in both teaching & evaluation
o self-evaluation
o formative feedback – longitudinal
 summative feedback

Faculty Development Needs:
o Effective local development plan
o Include others – specialists
o Individuals vs. group
o Develop own faculty to do direct obs feedback

Evaluation of faculty & formative feedback
o Solutions:
 Videotape faculty/give feedback
 Go to facdev – UW & Madigan
 Needs learner assessment and respond to that
 Webinars + 3-4 FacDev sessions after needs assess with
presentations by faculty
 Use existing structure for this
 Post webinar discussion
 Residents help push faculty
 Focus on longitudinal evaluation
 Structure teaching

Interactive FD session
o Look at data
o ID holes in teaching
 Shared understanding
 Resident engagement
 “AHA’s”
o Iterative process/ongoing conversation
o Formative feedback/capturing it
o Mapping teaching/evaluation