Now That We Have an Institutional
Report Card---How Do We Use It?
Conference Session: SES010
2012 ACGME Annual Education Conference
Ann Dohn, MA, DIO,
and
Nancy Piro, PhD,
Program Manager/Education Specialist
Department of Graduate Medical Education
Stanford Hospital & Clinics
Department of Graduate Medical Education (GME)
Disclosure
• No conflicts of interest to report
Department of Graduate Medical Education (GME)
Session Objectives
At the end of this session, participants will be able to:
1) Effectively use objective data in Balanced Report
Cards to make Institutional and Programmatic
decisions.
2) Understand strategies for gaining Institutional
Administration (C-Suite) support for DIO/GMEC
Decisions based on balanced report cards
3) Predict future ACGME review outcomes based on
empirical data from balanced report cards
Department of Graduate Medical Education (GME)
Setting the Stage
• How does GME administration function in
2012?
• What are the barriers?
• What are the challenges?
• How can we be heard by the C-Suite?
• What language should we speak?
• How do we optimize success?
Department of Graduate Medical Education (GME)
How does GME Administration
Function?
• How do we know?
• What do we measure?
• What management strategies do we use?
Department of Graduate Medical Education (GME)
Historically…..
• GME has not used objective data in making
many decisions
• Instead, decisions were often made based on:
–
–
–
–
–
–
Politics
Financial considerations
Special deals
Hunches
Ability to scream loudly and frequently
Fear
Department of Graduate Medical Education (GME)
But what is going on in the environment?
• Evidenced based
– Balanced score cards
– Increased accountability for resources
– Transparency
Department of Graduate Medical Education (GME)
Fitting GME within the overall business
culture
• Remember…your “C-Suite” colleagues often
have that magical MBA …..
• They think differently
• They use a different language
Department of Graduate Medical Education (GME)
Fitting GME within the overall business
culture
• We need to understand how the C-Suite thinks
• AND learn to think the same way…
AND speak their language…
To gain their support!!!
Department of Graduate Medical Education (GME)
What Considerations does the C-Suite
have with Respect to Data?
• Data must have:
– An Organizational focus
– Regular Means of Data Collection
– Include External Assessments as well as Internal
Assessments
• Data used must be
– Standardized (comparable) / lean
– Capable of transformation
– Aligned with mission statement
Department of Graduate Medical Education (GME)
DIOs need to be able to make decisions
based on comparative data
• Data must:
– Be Organizationally Focused
– Have Regular Means of Data Collection
– Include External Assessments as well as Internal
Assessments
– Standardized
– Capable of transformation
– Aligned with mission statement
Department of Graduate Medical Education (GME)
Conclusion
• CEOs and DIOs have the same
requirements with respect to their data
needs
Department of Graduate Medical Education (GME)
But what models for data do we have?
• Few data models exist for GME
• The concept of Institutional Accountability is
relatively new.
– Until the ACGME Outcome Project, there was no
centralized curriculum oversight in GME, unlike
medical schools / UME.
Department of Graduate Medical Education (GME)
The “Report Card” Vision
• In 2005, Stanford hired
its first PhD in GME
– The vision was to
develop tools to
construct evidence based
decision-making for
Graduate Medical
Education consistent
with our mission
– “We needed a Report
Card”…
Department of Graduate Medical Education (GME)
SIGH….
• It wasn’t as easy as first thought!
Department of Graduate Medical Education (GME)
Our First Attempt …
Department of Graduate Medical Education (GME)
Background on Institutional
Report Cards
• Government and Industry Models
– Multiple models exist and can be used as per specific
purpose:
•
•
•
•
•
•
•
•
•
GRPA (Government Performance and Results Act)
Organizational Report Cards
Balanced Scorecard
Benchmarking
Program Evaluations
Social Indicators
No Child Left Behind
C BEST
College Performance Testing
Department of Graduate Medical Education (GME)
Which Model Should We Use?
• We needed a model that:
– was organizationally focused and managed
– had a track record of effective use
– fit our existing structure with multiple programs and
organizations
– was flexible enough to be adapted for use on an
annual basis – not an accreditation cycle-Regular
Data Collection
– was “Easily Digestible”
– had internal and external measurement dimensions
Department of Graduate Medical Education (GME)
Our Choice
Balanced Scorecard
Framework
in an Organizational
and/or
Report Card
Tool
Best
of
Both
Worlds
Department of Graduate Medical Education (GME)
The Balanced Scorecard Approach
• The Balanced Scorecard is a performance
measurement and performance management
system developed by Robert Kaplan and David
Norton (1992, 1996)
– adopted by a wide range of leading edge
organizations, both public and private.
(“The Balanced Scorecard--Measures That Drive Performance,” Harvard Business Review, JanFeb 1992; and “The Balanced Scorecard-Translating Strategy into Action,” Harvard Business
School Press, 1996)
Department of Graduate Medical Education (GME)
Stanford’s Report Card – Three Programs
Department of Graduate Medical Education (GME)
Key to the Report Card
Department of Graduate Medical Education (GME)
Balanced Scorecard
Strategic Perspectives
How do our
residents see
us?
Institutional /
Financial Growth
Are we putting
our resources in
the right
places?
Resident
Are our
programs
excelling?
Mission
Vision
Strategy
Learning
Program
Processes
Do we continue
to improve
(outcomes)?
Department of Graduate Medical Education (GME)
Selection of Balanced Report Card Measures
INTERNAL PERI
PRE
RESIDENT
PERCEPTIONS
“Voice of the
Resident”
• GME Internal HS Survey
Recommendation of Program
• GME Internal HS Survey
Overall Satisfaction
• Resident Evaluation of Program
PROGRAM
PROCESSES
• Faculty Eval of program
• Duty Hr Violations
• # Internal Citations
LEARNING
OUTCOMES
FINANCIAL /
GROWTH
• Training Exams
• Resources
POST EXTERNAL
• Match Depth
• ACGME Survey Responses
• Teaching
• Supervision
• Scholarship
• Non Hostile/Intimidating Environment
• Address Concerns Confidentially
• ACGME Cycle Length
• # ACGME Citations
• Board Pass Rates
• Career Placement
• Grants
Department of Graduate Medical Education (GME)
Balanced ScoreCards “Report Cards”
Had
Were
– Organizational focus
– Regular Means of Data
Collection
– Include External
Assessments as well as
Internal Assessments
– Standardized
– Capable of
transformation
– Aligned with the
Mission of Stanford
Graduate Medical
Education (GME)
Department of Graduate Medical Education (GME)
Stanford Hospital & Clinics
Use of the “Balanced Report Card”
• The Stanford Report Card is built on the
Balanced Scorecard conceptual framework for
translating an organization’s vision into a set
of performance indicators distributed among
four perspectives adapted for GME:
1. Resident Perception Measurements
2. Program Processes
3. Learning Outcomes
4. Financial/Growth
Department of Graduate Medical Education (GME)
Stanford Background
Stanford University Medical Center currently
sponsors 85 ACGME-accredited training programs
with over 1100 enrolled residents and fellows.
Department of Graduate Medical Education (GME)
Stanford University Medical
Center Mission
• Dedication to pursuing the highest quality of patient care and
graduate medical education, recognizing that one of its major
responsibilities is the provision of organized educational
programs.
– Support of quality graduate medical education programs and
excellence in residency training and research.
– Guidance and supervision of the resident while facilitating the
resident’s professional and personal development and ensuring
safe and appropriate care for patients.
– Provision of adequate funding of graduate medical education to
ensure support of its faculty, residents, ancillary staff, facilities, and
educational resources to achieve this important mission.
– Ensuring that all of its graduate medical education programs meet
or exceed the Institutional and Program Requirements
promulgated by the Accreditation Council for Graduate
Medical Education.
Department of Graduate Medical Education (GME)
Translating the Stanford Hospital & Clinics’
GME Mission to the Balanced Report Card
1. Resident Perception Measurements
“Guidance and supervision of the resident while
facilitating the resident’s professional and personal
development and ensuring safe and appropriate care for
patients.”
2. Program Processes
“Ensuring that all of its graduate medical education
programs meet or exceed the Institutional and Program
Requirements promulgated by the Accreditation Council
for Graduate Medical Education.”
Department of Graduate Medical Education (GME)
Stanford Hospital & Clinics
Report Card
3. Learning Outcomes
“Support of quality graduate medical education
programs and excellence in residency training and
research.”
4. Financial/Growth
“Provision of adequate funding of graduate medical
education to ensure support of its faculty, residents,
ancillary staff, facilities, and educational resources
to achieve this important mission.”
Department of Graduate Medical Education (GME)
Stanford Hospital & Clinics Balanced Report
Card
• Indicators are designed to measure SHC’s progress
toward achieving its vision; other indicators are
designed to measure the long term drivers of success.
• Through the balanced report card SHC :
– Monitors its current performance (finances, resident
satisfaction, learning outcomes and program process
results)
– Monitors its efforts to improve processes, educate residents
– Enhances its ability to grow, learn and improve the quality
of its fellowship and residency educational programs.
Department of Graduate Medical Education (GME)
Why GME Thinks We Need This
• ACGME and Institutions are increasingly
holding DIOs and GME Committees accountable
for their utilization of institutional resources.
• Actions / decisions must be based on
documented real-time analyses of needs.
Department of Graduate Medical Education (GME)
Selection of Balanced Report Card Measures
INTERNAL PERI
PRE
RESIDENT
PERCEPTIONS
“Voice of the
Resident”
• GME Internal HS Survey
Recommendation of Program
• GME Internal HS Survey
Overall Satisfaction
• Resident Evaluation of Program
PROGRAM
PROCESSES
• Faculty Eval of program
• Duty Hr Violations
• # Internal Citations
LEARNING
OUTCOMES
FINANCIAL /
GROWTH
• Training Exams
• Resources
POST EXTERNAL
• Match Depth
• ACGME Survey Responses
• Teaching
• Supervision
• Scholarship
• Non Hostile/Intimidating Environment
• Address Concerns Confidentially
• ACGME Cycle Length
• # ACGME Citations
• Board Pass Rates
• Career Placement
• Grants
Department of Graduate Medical Education (GME)
Findings
• After several years of data collection….
– Some data is more valuable
– Some data drives change
Department of Graduate Medical Education (GME)
VOICE OF THE RESIDENT
Internal Measures
External Measures
• Match Depth
• Resident Overall
Evaluation of the Program • % Top Medical Schools
• Duty Hour Violations
• ACGME Survey - %
Compliant Responses
• GME/HS Survey - Overall
•Teaching
Satisfaction ?
•Supervision
• GME/HS Survey •Scholarship
Recommend Program ?
•Non Hostile/Intimidating
Environment
•Address Concerns
Confidentially
Department of Graduate Medical Education (GME)
PROGRAM PROCESSES
Internal
• Faculty Overall
Evaluation of the Program
• # Internal Citations
External
• ACGME Cycle Length
• # ACGME Citations
Department of Graduate Medical Education (GME)
Case Study - Stanford
• How the DIO uses the Balanced Report
Card:
– A Tale of Three Programs…
– Starting from the bottom….
– Assuming that being at the “bottom” may mean
“needs more help”
Department of Graduate Medical Education (GME)
How Do We Use this Data?
• Look at Indicators that are Resident Driven
– “Voice of the Resident”
– Is there a discrepancy between the voice of the
resident and the other indicators?
• Would the majority of the residents not choose the
program again yet the program receives a five year
accreditation cycle with one or no citations?
Department of Graduate Medical Education (GME)
How Do We Use this Data?
• How Do the Programs Compare Against
Each Other?
• How do they compare against their
ACGME Cycles?
Department of Graduate Medical Education (GME)
Stanford’s Report Card – Three Programs
Department of Graduate Medical Education (GME)
What does this tell the DIO?
(Program A)
• Setting the stage:
– 2008-ACGME gave a 5 year cycle with 1
citation
– 2009-ACGME resident survey shows
“tanking program”
– 2009-GME House Staff survey shows
“program in trouble”
– Duty Hour violations are > 15 times greater than
the Institutional Average
» Over a three year period
Department of Graduate Medical Education (GME)
Stanford’s Program A
Department of Graduate Medical Education (GME)
Stanford’s Program A
Department of Graduate Medical Education (GME)
What does this tell the DIO?
– Educational Milieu
• Intimidation and Fear of Retaliation - Increasing
• Lack of confidence in the program to confidentially
deal with problems or concerns - Increasing
• Overall resident satisfaction with the program –
decreasing
• Would recommend: extremely low in 2008/09
(lowest of any SHC program)
Definitely an early warning….
Department of Graduate Medical Education (GME)
How Did We Use the Data
• Validated data with resident interviews
(single and group)
You know you have a problem when the residents call and ask to
meet you at Starbucks…
Department of Graduate Medical Education (GME)
Setting up an “Action Plan”
• Met with program leadership
– Shared data
– Discussed “their” interpretation of the data
– “Brain-stormed” with program leadership
• “How can GME help you?”
Department of Graduate Medical Education (GME)
Program View on Situation
• Need more help…more residents…more
MD extenders….
• Residents are the problem…”not tough” and
they “whine”….
Department of Graduate Medical Education (GME)
GME Evaluation of the Situations
• Qualitative analysis of every comment on
the internal house staff survey
• Trend analysis of surveys
• DIO used her training in conflict resolution
• Report developed to define the problems
– Shared with C-Suite
Department of Graduate Medical Education (GME)
Action Plan
• Used data regarding duty hours with the CSuite to:
– justify hiring additional physician extenders
– reconfigure rotations
• Used resident program satisfaction data
with the Dean to:
– leverage program change and “motivate”
program leadership
Department of Graduate Medical Education (GME)
Results - Success ?
• Duty Hours have improved
• Waiting on resident sat from 2012 survey
(May)
Department of Graduate Medical Education (GME)
Program B
• Setting the stage:
– Internal Review in 2008-09 found a program
substantially out of compliance
» Problems were largely identified as structural ie:
formal policies/curriculum
» Acting program director
» Small, internationally known fellowship
» Very competitive
» Trainees pleased with program
» Trainees interviewed 2 months after start of the
program
Department of Graduate Medical Education (GME)
Program B
Department of Graduate Medical Education (GME)
Stanford’s Program B
Department of Graduate Medical Education (GME)
Setting up an “Action Plan”
• Assigned a GME staff professional to work
with the program director on the missing
“structural elements” in the program.
• DIO monitored progress
Department of Graduate Medical Education (GME)
Progress…
• Internal review citations corrected
• 2010 ACGME review gave 5 years and a
single citation
Department of Graduate Medical Education (GME)
However….the voices of the residents
• ACGME survey begins to show concerns over
training environment
• DIO starts receiving individual concerns from
trainees
• Program becomes large enough to qualify for
its own internal survey (smaller programs are
grouped)
– Internal survey shows major concern areas on
Report Card…a sea of “RED”
Department of Graduate Medical Education (GME)
Power of the Data
• Concerning data is shared with program
leadership
Department of Graduate Medical Education (GME)
Report Card Speaks
• Changes are implemented
–
–
–
–
–
Service is reduced
Associate program director appointed
Protected time for education enforced
Department leadership engaged
Administration support in place
Department of Graduate Medical Education (GME)
Disconnect
• Report card allowed GME to implement
intervention which successfully addressed
the areas of concern
• BUT…..ACGME (not having access to the
most recent data) called for an early site
visit.
Department of Graduate Medical Education (GME)
Program “C”: Setting the Stage
• The program was already experiencing
difficulties before their most recent
ACGME site visit
– Tough transition to a new program director
when former director abruptly left
• Problems identified on both the ACGME
and internal GME house staff surveys
Department of Graduate Medical Education (GME)
Program “C”
Department of Graduate Medical Education (GME)
Stanford’s Program C
Department of Graduate Medical Education (GME)
Action Items
• GME assigned a staff member to work
closely with program
– Note: program director readily accepted any
and all assistance-knew he had “inherited” a
“train wreck”.
Department of Graduate Medical Education (GME)
Progress?
• Program continued to spiral down……
– Lack of C-Suite “buy in”
Department of Graduate Medical Education (GME)
ACGME Outcomes
• ACGME site visit
– Proposed probation which resulted in:
• HUGE wake-up call for the C-Suite, residents, and
four major affiliates
• Data invaluable in presenting problems
Department of Graduate Medical Education (GME)
Action Items
• Data presented to
–
–
–
–
–
Department chair
Dean
DIOs at major affiliates
CEOs
RESIDENTS
Department of Graduate Medical Education (GME)
Consensus
• Decision to “save the program”
– C-Suite
– Program leadership
– GME
Department of Graduate Medical Education (GME)
Results
• Appealed the probation
– 2 year cycle
– Program continues to improve
Department of Graduate Medical Education (GME)
What does this tell the DIO?
Report card allows for easy monitoring of
multiple factors
With longer ACGME accreditation cycles in
our futures…imperative to be able to monitor
programs frequently
Department of Graduate Medical Education (GME)
Lessons Learned…
• Three very different programs
– But data drives change in all
– In all three cases the final outcome was data
driven
– In all three cases the C-Suite was needed to
resolve issues-and was “moved” to help ONLY
after being presented with data
Department of Graduate Medical Education (GME)
Lessons Learned (continued)
• As good as we think our “hunches” are…
– The C-Suite
• wants facts, figures, stats, etc
• You need to “prove” your case
• The C-Suite is VERY sensitive to the “norm”,
average, scores of other programs, benchmarks, etc.
• Your argument needs to be evidenced-based
• Don’t forget the outcomes!!
Department of Graduate Medical Education (GME)
A Tale of Three Programs
Department of Graduate Medical Education (GME)
Questions
Department of Graduate Medical Education (GME)