Conflict of Interest
 Ann Dohn – No conflicts of interest to disclose
 Nancy Piro – No conflicts of interest to disclose
Confidential – For Discussion Purposes Only
SES072
Mission NOT Impossible – Using Data to make
You and Your Program Shine!
Ann Dohn, MA
DIO & GME Director
Nancy Piro, PhD
Program Manager/Education Specialist
Session Outcomes
Participants will be able to:
 Understand the many varied sources of data elements
and how they can be organized and used in both
resident (CCC) and program (APE) evaluations
 Develop and utilize dashboard/report card structured
tools to track resident performance and program
performance across all aspects of their training
requirements, both current and longitudinally
 Identify ways to customize and adapt to individual
program and institution needs
 Take home tools to facilitate the implementation of
these objectives
Confidential – For Discussion Purposes Only
Overwhelmed by Data?
Data elements can be organized and used in both
resident (CCC) and program (PEC / APE)
evaluations
Confidential – For Discussion Purposes Only
NAS Data and YOU !
 The program coordinator plays a crucial role in developing,
implementing, collecting data, reporting milestone data,
program evaluation data…
Confidential – For Discussion Purposes Only
Let’s Break it Down…Different data elements for different purposes:
Confidential – For Discussion Purposes Only
Do you provide the CCC members any evaluation data to review
before the meeting?
100%
75%
50%
25%
82.65%
17.35%
Yes
No
0%
Confidential – For Discussion Purposes Only
7
Focusing on the data for your CCC (Clinical
Competency Committee)
Confidential – For Discussion Purposes Only
Pulling the Data Together
End-ofRotation
Evaluations
Clinical
Skills
Assessment
Nursing and
Staff /
Techs
Evaluations
Confidential – For Discussion Purposes Only
Quality
Improvement
Activities
Clinical
Competency
Committee
Progress on
Milestones
Sim
Lab
In-service
training
exams
Safety
Incident
Reports
Case
Logs
Patient/
Family
Evaluations
Data used in CCC meetings for trainee assessment
Aggregate Direct
Milestone
Evaluations
65.38%
Aggregate Rotation
Evaluations
93.85%
Staff or patient
(360) Evaluations
76.92%
Technical Skills
Assessments/MiniCEX's
44.62%
Scholarly Activities
63.08%
Case Volumes
36.92%
0%
10%
Confidential – For Discussion Purposes Only
20%
30%
40%
10
50%
60%
70%
80%
90%
100%
Creating a Resident Performance Profile
Step 1 – Defining what to track
Evaluations
Milestone End of Rotation (Total)
Medical Knowledge
Patient Care
Interpersonal & Communication Skills
Professionalism
Systems-Based Practice
Practice-Based Learning & Improvement
Confidential – For Discussion Purposes Only
Creating a Resident Performance Profile
Step 1 – Defining what to track
In-service Assessments (MK; PC)
Routine procedure technical skills
assessment: Level 2-3
Complex procedure technical skills
assessment: Level 3-4
Medical Knowledge Assessments
Confidential – For Discussion Purposes Only
Creating a Resident Performance Profile
Step 1 – Defining what to track
Case Logs / Clinical Experience (PC)
VAGINAL DELIVERY
CAESAREAN SECTION
PEDIATRICS
PEDIATRICS UNDER 3
CARDIAC
ENDOVASCULAR
Confidential – For Discussion Purposes Only
Creating a Resident Performance Profile
Step 1 – Defining what to track
Quality Improvement/Patient Safety
QI Project
QI Committee Participation
QI Course Work (e.g., IHI)
SAFE Report / Adverse Event Review
Patient Handover Evaluations
Confidential – For Discussion Purposes Only
Creating a Resident Performance Profile
Step 1 – Defining what to track
Practice-Based Learning
Patient outcomes / Case Study Presentation
Scholarly Activity: Research study
Scholarly Activity: Publications
Presentation at Internal and National Meeting
Confidential – For Discussion Purposes Only
Creating a Resident Performance Profile
Step 1 – Defining what to track
Communication
Patient Feedback
Staff Evaluation
Medical Student Feedback
Patient Handover Evaluations
Confidential – For Discussion Purposes Only
Creating a Resident Performance Profile
Step 2 - Link data sources to milestones
Evaluations
Milestone End of Rotation (Total)
Medical Knowledge
Patient Care
Communication
Professionalism
Systems-Based Practice
Practice-Based Learning
Quality Improvement/Patient Safety
QI Project
QI Committee Participation
QI Course Work (e.g., IHI)
SAFE Report / Adverse Event Review
Patient Handover Evaluations
Confidential – For Discussion Purposes Only
Milestones
24
MK A1
PC A1-10
ICS A1-2
PROF 1-5
SBP A1-2
PBLI A1-4
PBLI1-2
PBLI1-2
PBLI1-2
PROF1/2; SBP1
SBP1; ICS1
Milestone Data Translation to Numerical Data
Confidential – For Discussion Purposes Only
Creating a Resident Performance Profile
Step 3: CCC defines performance ranges
Example:
For all aggregate milestone evaluation scores for a PGY
3, the CCC defines these ranges:
At or Above Expectation:
2.8 and higher
Confidential – For Discussion Purposes Only
Below Expectation:
1.7 – 2.7
Remediation:
Below 1.7
Creating a Resident Performance Profile
Step 4 – Set conditional formatting
Confidential – For Discussion Purposes Only
Creating a Resident Performance Profile
Step 4 – Set conditional formatting
Example: Aggregate milestone evaluation data cells
Highlight cells to apply
the conditional
formatting
Confidential – For Discussion Purposes Only
Creating a Resident Performance Profile
Step 4 – Set conditional formatting
Click on “Conditional Formatting”
“Highlight Cells Rules”
Confidential – For Discussion Purposes Only
Creating a Resident Performance Profile
Step 4 – Set conditional formatting
Select, “Greater Than” “Less Than” or “Between” to Set Value
Ranges
Choose the corresponding fill color (e.g., red, yellow, green)
Confidential – For Discussion Purposes Only
Creating a Resident Performance Profile
Compiling and centralizing data
Confidential – For Discussion Purposes Only
Resident Performance Profile:
Step 5: Enter in data
Confidential – For Discussion Purposes Only
Creating a Resident Performance Profile
Visual Trends and Detailed Data
Confidential – For Discussion Purposes Only
There was no way we could have had an effective CCC meeting
without completing pre-work.
TRUE
90.58%
FALSE
8.07%
Not
applicable
1.35%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Confidential – For Discussion Purposes Only
27
Other Resources to make you shine!
Confidential – For Discussion Purposes Only
Leveraging Resident Management System
(RMS)Tools, if Available
 RMSs – becoming more feature rich
− CCC Online Functionality – Assign Files, Upload all required
data for Preliminary Review, etc
− Curriculum
 Goals and Objectives and learning outcomes by
rotation
 Teaching and Assessment methodologies
− Evaluation tool development
 Sharing between programs and institutions
Confidential – For Discussion Purposes Only
Leveraging Resident Management System
(RMS)Tools, if Available
 Conference attendance statistics
− Core competencies linked to specified conferences
− Attaching conference materials for later reference
 Procedures and levels; linked procedure evaluations
Confidential – For Discussion Purposes Only
Leveraging Resident Management System
(RMS)Tools, if Available
• Resident portfolio tools
QI participation and outcomes
Scholarly Activity logs
Confidential – For Discussion Purposes Only
Leveraging Resident Management System
(RMS)Tools, if Available
 Aggregate reporting and graphic summaries

Peer or departmental average, individual average, minimum and maximum scores,
standard deviation or listing of all scores
Confidential – For Discussion Purposes Only
Leveraging Calendaring and Task Management
Software
 Set “data gathering and reporting” appointments with yourself
 Remember to start your timeline with the end in mind (e.g., CCC meeting
dates)
 Break down large tasks into smaller tasks to keep it manageable
Confidential – For Discussion Purposes Only
Leveraging Calendaring and Task Management
Software
Confidential – For Discussion Purposes Only
PGY 1 Ranked at Graduation Level
1
Confidential – For Discussion Purposes Only
When Remediation is Evident
Confidential – For Discussion Purposes Only
Switching gears to APEs
APEs
CCCs
Confidential – For Discussion Purposes Only
Where do I begin?
Confidential – For Discussion Purposes Only
Program Evaluation Committee (PEC) Must Monitor and Track (V.C.2):
1. RESIDENT PERFORMANCE
2. FACULTY DEVELOPMENT
3. GRADUATE PERFORMANCE
4. PROGRAM QUALITY
5. PROGRESS ON THE PREVIOUS YEAR’S ACTION PLAN
Confidential – For Discussion Purposes Only
RESIDENT PERFORMANCE
 The most recent aggregated written evaluations of the
residents submitted by faculty and other evaluators
 In-training/In-service exam scores
 Procedure logs (if applicable)
 Scholarly activity (publications, presentations, grant
awards, etc.)
 Learning portfolios: documented quality improvement
activities
Confidential – For Discussion Purposes Only
FACULTY DEVELOPMENT
 ABMS certification status for all faculty
 Updated faculty CVs
 Documentation (faculty survey; attendance logs) of faculty
participation in:
− CME-type activities directed toward acquisition of clinical
knowledge and skills and also activities directed toward
developing teaching abilities, professionalism, and abilities for
incorporating the core competencies into practice
− Teaching (conferences, grand rounds, journal clubs, lecturebased CME events, workshops, directed QI projects, practiceimprovement self study).
 Faculty actively involved in mentor relationships with
residents/fellows.
Confidential – For Discussion Purposes Only
GRADUATE PERFORMANCE
 Aggregated board exam pass rates
 Aggregated alumni survey results (typically, such surveys
target physicians one year and five years after graduation –
survey questions may inquire about such items as current
professional activities of graduates and perceptions on how
well prepared they are as a result of the program)
 Other outcome measures
− Practice location (underserved areas)
− Academic Affiliations
− Scholarly Activity
Confidential – For Discussion Purposes Only
Alumni Surveys …
Confidential – For Discussion Purposes Only
PROGRAM QUALITY
 The most recent aggregated written evaluations of the program
submitted by faculty
 The most recent aggregated written evaluations of the program (and/or
specific rotations) submitted by residents
 The most recent aggregated written evaluations of the faculty submitted
by residents
 Faculty’s recent scholarly activity (publications, presentations, grant
awards, etc.)
Confidential – For Discussion Purposes Only
PROGRAM QUALITY - Continued
 The most recent ACGME survey results
 The most recent GME House Staff survey results
 The most recent GMEC Internal Review Report
 Any recent communications from the ACGME or RRC
 Program Report Card/Scorecard
− Trend Analyses
Confidential – For Discussion Purposes Only
PROGRAM QUALITY - Continued
 Curriculum
 Overall and rotation-specific goals and objectives (Are they
appropriate? Do they align with the core competencies?)
 Didactic curriculum (Is there at least one regular conference
targeted to the residents’ level?)
 Opportunities for scholarly activity
 Compliance with any new standards established by the ACGME,
RRC, ABMS, etc.
− Assessment Methods (Are evaluation tools appropriate? Do they align with
the core competencies?)
− Resources: Personnel (PD, PC, faculty), Affiliated Training Sites,
Patient/Procedure Volume,
 Learning Environment (space, call rooms, books,
computers, etc.)
Confidential – For Discussion Purposes Only
PROGRESS ON THE PREVIOUS YEAR’S ACTION PLAN
Review progress / (attempts to resolve problems) with respect to
last year’s Annual Review delineating identified areas of weakness.
Confidential – For Discussion Purposes Only
Diamond in the Rough…
Confidential – For Discussion Purposes Only
Making Your Program Shine
 Use your data !!
− ADS Updates
 Major Changes
− Applicants
− Annual Reports
− Scorecards
− Monitor Program Improvement
− Monitor Trends
Confidential – For Discussion Purposes Only
Report Card Trending Example
Confidential – For Discussion Purposes Only
How Can You Build a Scorecard? Easier than it looks!
Confidential – For Discussion Purposes Only
External Measures
Confidential – For Discussion Purposes Only
Internal Measures
Confidential – For Discussion Purposes Only
Web ADS
Confidential – For Discussion Purposes Only
Annual Reports
PEDIATRIC GASTROENTEROLOGY
ANNUAL REPORT
Wednesday, June 11, 2014
Managing it all: How will I pull this off?
Confidential – For Discussion Purposes Only
Use Technology to Your Advantage…RECAP



Know your program requirements and
follow them unconditionally
Use simple spreadsheet, calendaring and
task organizational tools to manage,
track and present resident performance
data to your CCC
Resident education is a cyclical process –
revisit and revise tools and processes
each year
Confidential – For Discussion Purposes Only
Asking Why??
 Continue to ask: “Why am I doing this??
 Is there a better way…”Five Why’s”
 Then streamline – no use collecting data that won’t be used i.e., “useless”
Confidential – For Discussion Purposes Only
The Toolbox
Confidential – For Discussion Purposes Only
Tools Can Be Downloaded @ www.gme.stanford.edu
GME Community
Confidential – For Discussion Purposes Only
Templates – A3
Tools
Confidential – For Discussion Purposes Only
Questions
Confidential – For Discussion Purposes Only
Contact Information:
 Ann Dohn: adohn1@stanford.edu
 Nancy Piro: npiro@stanford.edu
Confidential – For Discussion Purposes Only