Federation of State
Physician Health Program
2012 Annual Meeting
Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV
FSPHP April 23-26, 2012
1
THE DISRUPTIVE PHYSICIAN:
THE DEVELOPMENT AND USE OF A 360
INSTRUMENT AS A MONITORING TOOL
Betsy White Williams PhD MPH
Assistant Professor
Director of Outcomes and Research
Office of Continuing Medical Education
Rush University Medical Center
Clinical Program Director
Professional Renewal Center
Lawrence, KS
William H. Swiggart, MS, LPC/MHSP
Assistant in Medicine
Vanderbilt Department of Medicine
Co-Director
Center for Professional Health
Vanderbilt University School of Medicine
Nashville, TN
Marine V. Ghulyan, MA
Research Analyst
The Center for Professional Health
Vanderbilt University School of Medicine
Nashville, TN
Kayci Vickers
Research Assistant
Professional Renewal Center
Lawrence, KS
Michael V. Williams, Ph.D.
Wales Behavioral Assessment
Principal
Wales Behavioral Assessment
Lawrence , KS
Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV
FSPHP April 23-26, 2012
2
Learning Objectives
 Understand the results of 360 degree assessment in
monitoring of intervention effectiveness.
 Understanding the likely changes in the results from
360 degree evaluations over time.
 Understanding the interpretation of outliers in
utilizing a 360 degree evaluation to determine
intervention efficacy
Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV
FSPHP April 23-26, 2012
3
Presentation - Context
 Physicians identified as disruptive practitioners are
increasing being referred to Physician Health
Programs. Heretofore it has been difficult to monitor
the progress of these physicians post intervention.
 This presentation discusses the development of a
360 survey instrument.
 While we were interested in evaluating and comparing the
data between cases and a comparison sample,
 we were particularly interested in the use of the 360 for
both identification and monitoring.
 The 360° survey was recently developed based on
input from experts and a review of the literature.
Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV
FSPHP April 23-26, 2012
4
3 Core Competency Survey (3CC)
 It is not enough to have good motives;
others respond to our behavior.
 Physicians are often not given essential
feedback about their behavior.
 The Three Core Competency Survey
(3CC) is designed to provide feedback
from those we work with.
Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV
FSPHP April 23-26, 2012
5
Disruptive behavior
 “Intimidating and disruptive behaviors can
foster medical errors, contribute to poor
patient satisfaction and to preventable
adverse outcomes, increase the cost of care,
and cause qualified clinicians, administrators
and managers to seek new positions in more
professional environments.”
 Issue 40: Behaviors that undermine a culture of safety | Joint
Commissionhttp://www.jointcommission.org/assets/1/18/SEA_4
0.PDF
Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV
FSPHP April 23-26, 2012
6
Spectrum of Disruptive Behaviors
Aggressive
Anger Outbursts
Profane/Disrespectful
Language
Throwing Objects
Demeaning Behavior
Physical Aggression
Sexual Comments or
Harassment
Passive
Aggressive
Passive
Chronically late
Derogatory
comments about
institution,
hospital, group,
etc.
Refusing to do
tasks
Not responding
to call
Inappropriate or
inadequate chart
notes
Racial/Ethnic Jokes
Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV
FSPHP April 23-26, 2012
7
Disruptive behavior
 “The term “disruptive behavior” is changed
in the standards
 The term “disruptive behavior” in two elements
of performance (LD.03.01.01, EPs 4 and 5)
has been revised to “behavior or behaviors
that undermine a culture of safety.” ”
 Joint Commission online November 11, 2011
 http://www.jointcommission.org/assets/1/18/jconline_Nov_9
_11.pdf
Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV
FSPHP April 23-26, 2012
8
Disruptive Behavior Leads to
Communication Problems…Communication
Problems Lead To Adverse Events1
 Communication breakdown factored in OR errors 50% of
the time2
 Communication mishaps were associated with 30% of
adverse events in OBGYN3
 Communication failures contributed to 91% of adverse
events involving residents4
Gerald B. Hickson, MD
James W. Pichert, PhD
Center for Patient & Professional Advocacy
Vanderbilt University School of Medicine
1. Dayton et al, J Qual & Patient Saf 2007; 33:34-44.
3. White et al, Obstet Gynecol 2005; 105(5 Pt1):1031-1038.
2. Gewande et al, Surgery 2003; 133: 614-621.
4. Lingard et al, Qual Saf Health Care 2004; 13: 330-334
Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV
FSPHP April 23-26, 2012
9
Failure to Address Disruptive
Conduct Leads To
 Team members may adopt disruptive
person’s negative mood/anger (Dimberg &
Ohman, 1996)
 Lessened trust among team members can
lead to lessened task performance (always
monitoring disruptive person)... effects
quality and patient safety (Lewicki & Bunker,
1995; Wageman, 2000)
Felps, W et al. 2006. How, when, and why bad apples spoil the barrel: negative group members and
dysfunctional groups. Research and Organizational Behavior, Volume 27, 175-222.
Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV
FSPHP April 23-26, 2012
10
Disruptive Behavior Creates
 fear
 ignorance (expectations,
 confusion or uncertainty
 vengeance vs. those who





oppose/oppress them
hurt ego/pride
grief (denial, anger,
bargaining)
apathy
burnout
unhealthy peer pressures




behavior standards, rules,
protocols, chain of
command, standards of care)
distrust of leaders
dropout: early retirement or
relocation
errors
disruptive behavior begets
disruptive behavior
Vanderbilt University and Medical Center Policy #HR-027
Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV
FSPHP April 23-26, 2012
11
Two systems interact
The internal system
The external system
Functional &
nurturing
Hospital/Clinic
Physician
Dysfunctional
Good skills
Poor skills
“The Perfect Storm”
Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV
FSPHP April 23-26, 2012
12
Etiologies
 Institutional Factors
 Scapegoats
 System Reinforces Behavior
 Individual Pathology may over-shadow institutional
pathology
Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV
FSPHP April 23-26, 2012
13
Etiologies
 Personal Factors
 Individual pathology
 Life Stressors
 Lack of knowledge and skills
Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV
FSPHP April 23-26, 2012
14
2004 AAMC Council of Deans
“Physicians are often poorly socialized
and enter medical school with inadequate
social skills for practice.”
“There is a growing body of literature
documenting residency programs do not
prepare resident physicians adequately for
the practice of medicine.”
Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV
FSPHP April 23-26, 2012
15
Potential Resources for Healthy
Coping
 Courses
 Coaches, counselors
 Comprehensive




Evaluation
360° Evaluations
Risk Managers
Physician Wellness
Treatment Centers
 Office of General Counsel
 State BME
 Professional Societies
 QI Officers
 EAP
 Others
 State Physician Health
Program
16
Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV
FSPHP April 23-26, 2012
16
CME Remediation of “disruptive behavior”
 Development of insight
 Development of Skills
 Development of implementation strategy
 Feedback and monitoring
Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV
FSPHP April 23-26, 2012
17
General Trends
 At 3 months, significant improvements in 20 of
the 22 physicians
 Increased motivating behaviors and motivating
impact
 Decreased disruptive behaviors and disruptive
impact
 Changes in behavior reported by “others”
Samenow CP, Swiggart W, Blackford J, Fishel T, Dodd D, Neufeld R, Spickard A. A CME Course Aimed at
Addressing Disruptive Behavior. Physician Executive; 34 (1) Jan/Feb 2008: 32-40.
18
Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV
FSPHP April 23-26, 2012
18
360 Survey to Provide Feedback and
Monitor Behavior
 BASED on CORE COMPETENCY AREAS
 Interpersonal and Communication Skills
 Professionalism
 System based practice
Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV
FSPHP April 23-26, 2012
19
Survey Development
 Review of 300+ publications of the
evidence focused articles on physician
professional behavior;
 Abstraction of assessment items with a
expert based Delphi process to yield
candidates for the final scale.
Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV
FSPHP April 23-26, 2012
20
Survey Development
 Interpersonal and Communication Skills
 Uses verbal communication to provide
appropriate feedback to others.
 Makes others feel comfortable
approaching to ask questions or make
suggestions.
 Communicates effectively with patients.
© Williams, Swiggart, and Williams
Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV
FSPHP April 23-26, 2012
21
Survey Development
 Professionalism
 Willingly performs all tasks, functions, or
responsibilities that are typically expected
of him/her.
 Responds promptly to telephone and
pages.
 Reports timely to hospital/clinical duties.
 Responds quickly and appropriately to
administrative communications.
© Williams, Swiggart, and Williams
Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV
FSPHP April 23-26, 2012
22
Survey Development
 System-based practice
 Creates a sense of teamwork and valued
contribution by team members.
 His/her behavior makes others
comfortable in their work.
 Creates an accepting work environment.
© Williams, Swiggart, and Williams
Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV
FSPHP April 23-26, 2012
23
Importance of Monitoring
 Necessity of ensuring the behavior does not
recur,
 Anecdotal evidence of a significant level of
recidivism,
 Prior behavioral issues are a significant risk
factor for later disruption (Papadakis and
colleagues, see for example, Papadakis, Arnold et al.
2008)
 Facilitates earlier identification
Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV
FSPHP April 23-26, 2012
24
Importance of Feedback
 Form of guided self-assessment,
 Reinforces behavioral changes,
 Provides a standard by which to assess
gains.
Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV
FSPHP April 23-26, 2012
25
Methods-Subject Group
 Participants in the The Course for Distressed
Physicians, a remedial CME course developed
at the Center for Professional Health at
Vanderbilt University.
 Cases: referred for workplace difficulties that
relate to team behavior
 Comparison: Physicians of similar specialties as
the cases .
Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV
FSPHP April 23-26, 2012
26
Methods-Data Analysis
 Specificity and sensitivity using the two
classes of participants were analyzed.
 The means and distributions were analyzed
for consistency with other measures of
performance more consistent with process
measurement.
 The outcome of an analysis of outliers is
reported.
Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV
FSPHP April 23-26, 2012
27
Reliability and Validity
 Neither reliability or validity is a simple issue
in the context of measures across time
 Consider the issue of the WAIS, while valid and
reliable, serial tests can be problematic due to
lagged time effects;
 In the case of “disruptive behavior”, serial
measures are core to the value of the assessment
instrument.
Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV
FSPHP April 23-26, 2012
28
Measures of Validity
 Types of validity:
 Face validity;
 Construct validity;
 Predictive validity
 In general these are summarized by Messick:
 “ … the degree to which the empirical evidence and
theoretical rationales support the adequacy and
appropriateness of interpretations and actions based
on test scores.”
 Expanded by Kane to four domains:
 Scoring, generalization, extrapolation, and
interpretation/decisions.
Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV
FSPHP April 23-26, 2012
29
Today’s focus
 While we touch briefly on our first two
elements:
 Scoring, and,
 Generalization.
 Most of the focus of this discussion is on:
 Extrapolation, and,
 Interpretation and decisions.
 The interpretation and decisions element
most clearly differentiate useful instruments
from interesting academic exercises
Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV
FSPHP April 23-26, 2012
30
Scoring
 An extension of face validity
 Is the item appropriate to the construct of
interest;
 We selected a scale shown to be valid in other 360
medical applications
Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV
FSPHP April 23-26, 2012
31
Generalization
 Convergence:
 Our scale(s) generate a Cronbach’s α in excess of 0.9
the degree of exceeding depending on the item set.
 Our scales, using underlying factor structures,
demonstrate factor invariance across at least 3 sets of
raters
 The degree of coherence is clear, individual
differences – the contribution to formal error –
are being examined but two seem theoretically
appropriate: time of remedial training and
identification as disruptive.
Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV
FSPHP April 23-26, 2012
32
Extrapolation
 Extrapolation – Do the scores predict real-
world outcomes of interest, a broad
restatement of predicative validity.
 Four groups
 distressed physician class participants at Vanderbilt
 Distressed physician class participants at PRC
 Comparison sample at Vanderbilt
 Comparison sample from PRC.
.
Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV
FSPHP April 23-26, 2012
33
Extrapolation
No significant different
was found between the
two comparison
samples.
The data were pooled in
subsequent analyses
Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV
FSPHP April 23-26, 2012
34
Extrapolation
The 360
demonstrates
known group
discrimination:
The method is valid
as far as
discriminating
between
professionals
identified as
demonstrating
behavioral
issues.
Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV
FSPHP April 23-26, 2012
35
Extrapolation
Interprofessional
Behavior
Note the reduction
in mean
difference across
time is minimal;
However, the
reduction in
variance, and
particularly skew
is marked.
Extreme
performance,
particularly low
performance is
reduced over
time.
Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV
FSPHP April 23-26, 2012
36
Disruptive behavior – Social
systems
 Preliminary results
suggest that
disruptive physicians
may not differ
significantly from
normal physicians in
mean performance
but may differ
significantly in skew.
Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV
FSPHP April 23-26, 2012
37
37
Extrapolation
Wave analysis for
inter-professional
behavior
Again the change is
mostly in the
reduction of
variance across
time, not in
means.
Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV
FSPHP April 23-26, 2012
38
Disruptive Behavior and Institutional
Functioning
 The presence of the
system disruption
ultimately results in
breakdown:
 Communications;
 Affiliation;
 Roles; and,
 Protocols and duties.
Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV
FSPHP April 23-26, 2012
39
39
Disruptive behavior – Social
systems
 Results suggest that
disruptive physicians
may not differ
significantly from
normal physicians in
mean performance
but may differ
significantly in skew.
Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV
FSPHP April 23-26, 2012
40
Disruptive behavior – Social
systems
 Over time as the
disruptive behavior is
extinguished the
pattern of data will
modify.
 Results suggest the
proportion of extreme
reports falls and
moderate to good
reports increase.
Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV
FSPHP April 23-26, 2012
41
Competencies and Team Function
As outlying performance
decreases team cohesion
increases.
Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV
FSPHP April 23-26, 2012
42
Discussion
 The instrument discriminates between
participants in the remediation exercise and
the comparison sample.
 The instrument shows appropriate sensitivity
and specificity and appears to be valid.
 Analysis of outliers and serial results
 means and distributions appear to be consistent
with expectation
 means and distributions change over time
appropriately.
Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV
FSPHP April 23-26, 2012
43
Discussion
 The tail of the distribution
 appears very sensitive to behavioral change as
reported by other observers.
 The 3C 360° survey is a promising measure of
CME efficacy in changing practice patterns.
Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV
FSPHP April 23-26, 2012
44
Interpretation/Decisions
 The data support the discrimination between:
 Physicians identified as being disruptive, and,
 Physicians from comparison samples.
 As well as:
 Physicians identified as being disruptive, and,
 Physicians in a general remediation program.
 These data suggest that general
interpretation is valid.
Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV
FSPHP April 23-26, 2012
45
Interpretation/Decisions
 The instrument is intended to address:
 Application within referral sources (hospitals,
clinics, academic medical centers); and,
 A methodology that both tracks improvement
and indicates adequate performance.
 These elements are met through a core
competency structure and sensitivity to
changing outliers.
Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV
FSPHP April 23-26, 2012
46
Discussion
 The instrument
 provides a consistent measurement with the literature
and our experience of those areas of functioning
related to interpersonal skill and communications,
professionalism, and team behavior for healthcare
professionals.
 demonstrates appropriate sensitivity and specificity
 provides the basis of effectively assessing intervention
efficacy.
 shows promise as a monitoring instrument and as a
mean of identifying relapse behaviors.
Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV
FSPHP April 23-26, 2012
47
Discussion
 Sample Case Report
Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV
FSPHP April 23-26, 2012
48
Discussion
 Sample Comment page
Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV
FSPHP April 23-26, 2012
49