Reliability and Validity of the
Disruptive Behavior Instrument
for Physicians with Disruptive
Behavior
Philip Hemphill, Ph.D.
Professional Enhancement Program (PEP)
Pine Grove Behavioral Health and Addictive
Services
phemphill@forrestgeneral.com
FSPHP Presenter Disclosure Slide
“Reliability and Validity of the
Disruptive Behavior Instrument for
Physicians with Disruptive Behavior”
Philip Hemphill, PhD
is a paid consultant of
Pine Grove Behavioral Health and Addiction Services
FSPHP
Annual Conference and Meeting
Fort Worth, Texas
April 23-26, 2012
Why use a MSF/360?
1.
2.
3.
4.
5.
6.
7.
8.
Facilitate insight of strengths and weaknesses
for an individual.
Enhance organizational cultural change.
Provide summative assessment of performance.
Evaluate potential.
Improve team effectiveness.
Identify training & coaching needs.
Make organizational values explicit.
Measure progress after training & coaching.
MSF/360 Essentials
 Proven validity & reliability.


Useful in assessing “humanistic” skills.
Weak or no associations between self-rated assessment and
external assessment.
“…methods such as multisource feedback (360) evaluations may be
a necessary next step, particularly when interpersonal, communication
skills, or professionalism needs to be evaluated (p. 1101).”
Source: Davis et al. (2006). Accuracy of physician self-assessment compared with observed
Measures of competence.JAMA, 296 (9), 1094-1102.
Key Players in MSF/360
Raters
Boss
Target
Individual
Facilitator
Vendor
MSF/360 Internal Process
Feedback
Is it consistent with self-perception?
Emotional Response
Reflection and internal processing of
emotion and content.
Decision to accept and act, or not
Source: Sargeant et al. (2009)
Advances in Health Sc Ed.
Action for learning, change
Relevant Literature Review of Multi-Source Feedback Measures
The Fundamental Interpersonal Relations Orientation–Behavior™ (FIRO-B®)




In the late 1950s William Schutz, PhD. developed the FIRO-B
theory to aid in the understanding and predicting of how highperformance military teams would work together.
In developing the FIRO-B theory, Schutz began with the premise
that "people need people." He used the term interpersonal to
indicate any interaction, real or imagined, occurring between
people. He used the term need to describe a psychological
condition that, if not satisfied, leads to a state of discomfort or
anxiety.
He posited that interpersonal needs could be grouped into three
categories: Inclusion, Control, and Affection. The FIRO-B model
describes the interaction of these three categories of interpersonal
need along two dimensions: expressed and wanted.
Awareness Scales and were designed primarily to help individuals
with their self-awareness and of understanding their relation to
other people (Schutz, 1978).
Relevant Literature Review of Multi-Source
Feedback Measures
1.
2.
3.
4.
5.
6.
Developing insight into strengths and weaknesses, for the
individual (Farh & Dobbins, 1989; Hazucha et al., 1993; Martocchio &
Judge, 1997; Fletcher, 1999; Keeping et al., 1999; London et al., 1999;
McCarthy & Garavan, 1999) and the organization (London & Beatty,
1993; Lepsinger & Lucia, 1998; Steensma et al., 1998; Tornow &
London,1998) to act upon.
Enhancing culture change (Cardy & Dobbins, 1994; O’Reilly, 1994;
Crossley & Taylor, 1995; Church & Bracken, 1997; Bracken et al., 1998;
DeSimone, 1998).
Summative assessment of performance (Newble et al., 1999).
Evaluating the potential of individuals, for instance to use in
career advice or selection (Tornow, 1993; Lindsey et al.,1997;
Facteau et al., 1998).
Enhancing team effectiveness (by allowing opportunity to
comment) (Garavan et al., 1997; Towers-Perrin, 1998).
Identifying training needs for the system (as opposed to the
individual) (Towers-Perrin, 1998).
Development of Disruptive Behavior
Instrument





2004 – Developed list of disruptive behaviors and
boundary violation behaviors. Approximately 70
items…
5 people monitored over 12 months – Instrument
too long; complaints from participants; descriptive
data….
2005- Cut instrument down to 50 items(1 Fctr)
2008- Cut down to current version
Been using current version for past 4 years…
Definition of Disruptive Behavior
Disruptive behavior is any inappropriate behavior,
confrontation, or conflict, ranging from verbal abuse to
physical or sexual harassment. Disruptive behavior causes
strong psychological and emotional feelings, which can
adversely affect patient care.
Rosenstein A, O’Daniel M. (2008). Managing disruptive physician
behavior: Impact on staff relationships. Neurology, 70, 1564-1570.
Disruptive Behavior Items







Avoids taking responsibility for actions
Inappropriately blames others
Places other interests above the safety and welfare of
community members

Suspects, without sufficient basis, that others are exploiting,
harming, deceiving, or out to “get” him
Seems to have a preoccupation with unjustified doubts about
the loyalty or trustworthiness of others
Reads hidden, demeaning, or threatening meanings into benign
remarks or events
Perceives attacks on his character/reputation not apparent to
others and is quick to react angrily/counterattack


Shows rigidity and stubbornness
Displays an unpredictable mood
Responds to constructive criticism in an
immature/irresponsible fashion
Takes too long to requests from others

Arrives late

Fails to complete paperwork in a timely manner
Responds to requests in an avoidant, unreliable, and/or
uncooperative manner













Displays a sense of entitlement
Shows perfectionism that interferes with task completion
Behaves in a sexually inappropriate manner (verbally or
physically)
Lacks empathy Intimidates others or attempts to intimidate
others
Exhibits physically aggressive or assaultive behaviors
Exhibits arrogant behaviors and/or has an arrogant attitude
Demonstrates an inflated sense of self-importance
Takes advantage of others in order to get what he wants
Is reluctant to delegate tasks or to work with others unless
they submit to exactly his way of doing things
Expresses anger/hostility inappropriately
Exhibits passive-aggressive behavior
Uses abusive, belittling, condescending, demeaning, and/or
threatening language
CLIENT SIGNS/
UPDATES
MONITORING
CONTRACT WITH
PHP/LAP AND/OR
Client signs release allowing
for electronic correspondence,
and client given monitoring
instructions and materials with
due date.
Client identifies
MC, gives MC
instructions and
rating materials,
and with MC
identifies
approx. 15
raters..
Client
participates in
interventions
while
continuing
rating process
Rating process
is repeated at
predetermined
intervals until
four
uneventful
reports
Ratee email
addresses
Client attends
treatment and
rating process
is temporarily
suspended
PHP/LAP
recommends
interventions
No significant
impairment is
present
Collection of
surveys online.
PHP/LAP
recommends
intensive/
residential
treatment
Significant
impairment is
present and
PHP/LAP
intervenes
Client, MC,
and other
relevant
stakeholders
review the
feedback report
Report
produced and
emailed to
persons
indicated on
release.
Client
completes
treatment
MONITORING
RATING
FLOWCHART
33 questions




28 Likert-type questions regarding ratee’s behavior
1 question concerning how many hours rater works
with ratee
1 question about the rater’s title
3 open-ended questions that solicit comments
Client 43 Overall Ratings Over Time
35.0
30.0
Overall Score
25.0
20.0
15.0
10.0
5.0
0.0
1
2
3
4
5
Rating Period
Aggression: Verbal
Blaming Others
Disobeying Rules
Lack of Opennes to Others
Response to Criticism
Client 109 Overall Ratings Over Time
35.0
30.0
Overall Score
25.0
20.0
15.0
10.0
5.0
0.0
1
2
3
4
5
Rating Period
Aggression: Verbal
Lack of Openness to Others
Blaming Others
Response to Criticism
Disobeying Rules
Sexual and Inappropraite Remarks
Survey Dimensions














Appropriate Documentation
Arrogance
Citizenship
Compliance
Empathy
Exploitation of Others
Expression of Anger
Intimidation
Job Attitudes
Multicultural Sensitivity
Openness to Others
Perfectionism
Respect for Others
Response to Criticism













Responsibility for Actions
Rigidity
Satisfaction with Ratee
Sexual Appropriateness
Stability of Mood
Stress Management
Supervision
Suspiciousness
Tardiness
Team Impact
Threatening Behavior
Verbal Aggression
Working with Others
Disruptive Behavior Instrument

Descriptive Characteristics of Sample:

N = 22 ratees (498 surveys)



Survey Breakdown by rater:





Average Age = 38 years
Male = 19 ; Female = 3
Self-Report = 53
Staff = 328
Supervisor = 37
Peer = 79
Number of Times each patient was assessed:

Range – 1-8 times
Disruptive Behavior Instrument
• Originally hypothesized to be based
on 5 factors
Aggression
• Aggression: Non-verbal
• Aggression: Physical
• Aggression:Verbal
• Aggression: Written
Interpersonal Difficulties
• Intimidation
• Lack of Empathy
• Lack of Multicultural Sensitivity
• Lack of Openness to Others
• Problematic Perfectionism
• Response to Criticism
• Rigidity
• Suspiciousness
Sexually inappropriate behavior
• Sexual Inappropriateness
• Sexual/Inappropriate Remarks
Self Regulation
• Mood Instability
• Poor Stress Management
Disruptive Behavior
• Arrogance
• Blaming Others
• Disobeying Rules
• Disrespect for Staff/Colleagues
• Ease of Working with Ratee
• Employee Turnover Intentions
• Exploitation of Others
• Poor Citizenship
• Tardiness
Disruptive Behavior Instrument

Results of hypothesized 5-factor loadings; 59% cumulative total variance explained
Factor 1
Q11
Factor 2
1
2
3
4
5
.746
.014
.000
.118
.293
1
Q4
Q6
Q5
Q1
Q3
Q23
Q7
Q17
Q14
.746 -.134 -.006 -.095
Q9
3
4
5
.702
.195 -.417 -.288
.080 -.023
.732 -.055
.249 -.110
.023 -.120
.267
.726 -.194 -.040 -.060
.064
.620 -.306 -.183 -.033 -.194
.560 -.210
.057 -.095
.257
.171 -.039
.017
.550 -.349 -.090 -.054
.014
.058
.267
.513 -.257 -.255
.510 -.335
.204 -.345
.102 -.090
.238 -.066
4
5
-.482 -.315 .702 -.063 -.029
.693 -.093
.108
.005
-.114
.683 -.211 -.032 -.054
-.136
.672
.235 .100 .632 .201 .034
Q19
.083 -.321 .431 .141 -.011
Q13
.115 -.050
Factor 4
.000
1
2
3
4
5
Q25
.160
.091
.212
.718
.095
Q28
-.107 -.102
.049
.638
-.076
Q21
-.163
.633 -.222
.077 -.019
-.209
.622 -.179
.021
.134
-.187
.574
.025 -.079
.140
Q20
Factor 5
Q8
.515
3
Q10
-.179
Q18
.661 -.261 -.115 -.049 -.159
.616 -.056
2
.036
Q15
.733
1
Q26
Q16
-.157
Q2
2
Q24
.207
Q12
Factor 3
.024
.546 -.236
.026 -.002
Q22
1
2
3
4
5
.109
.078
.011
.025
.860
Raw Data Eigenvalues, Mean & Percentile
Random
Data
Eigenvalues
Ncases 498; Nvars 27; ;
Ndatsets 1000; Percent 95
Root
1.000000
2.000000
3.000000
4.000000
5.000000
6.000000
7.000000
8.000000
9.000000
10.000000
11.000000
12.000000
13.000000
14.000000
15.000000
16.000000
17.000000
18.000000
19.000000
20.000000
21.000000
22.000000
23.000000
24.000000
25.000000
26.000000
27.000000
Raw Data
10.462501
1.836325
1.408055
1.231678
1.178393
.966591
.890236
.805858
.786433
Means
1.456515
1.386788
1.335702
1.291131
1.253103
1.216903
1.183359
1.152611
1.121760
.707731
.671235
.647008
.567647
.518775
.514043
.454498
.414122
.395714
.363423
.337990
.312618
.294869
.281812
.268155
.258453
.230247
.195588
1.092700
1.064358
1.037406
1.010437
.984026
.958262
.932558
.907291
.882943
.857912
.832012
.806413
.780470
.753717
.725304
.695053
.661538
.619728
Percentile
1.528056
1.437043
1.380089
1.331221
1.287905
1.247532
1.212107
1.180525
1.146968
1.117568
1.087904
1.061681
1.033661
1.007120
.981678
.955167
.929837
.906811
.882594
.855692
.831047
.806496
.780180
.753368
.724017
.694250
.658277
Disruptive Behavior Instrument

Recent analyses reveal that the items are loaded on three distinct factors
Factor 1
Factor 2
Factor 3
1
Q12
Q5
Q4
Q11
Q1
Q6
Q3
Q17
Q23
Q2
Q7
Q9
Q14
2
3
.780 -.068 -.031
.770 -.099
1
Q18
.763
.194
.128
.742
.054
.305
.740 -.085 -.162
Q15
Q20
.117
.717 -.077 -.194
.618 -.210 -.054
.609 -.079
Q16
Q19
-.163
.727
-.066
-.208
.697
.029
-.251
.666
.098
-.275
.628
.098
-.192
.621
-.005
.066
-.531
.171
-.234
.496
.028
.125
.487
-.398
-.276
.443
.073
.132
Q13
.603
1
3
Q25
2
3
.127
.043
.636
.025
-.034
.604
-.089
-.034
.408
.144
-.276
.319
.002
Q21
.732 -.051
2
.008 -.044
.575 -.254
.026
.528 -.416
.016
.522 -.007 -.006
Q24
Q8
Q22
Q28
Q10
* Excluded Item 26
Disruptive
Behavior
Instrument
Factor 1: Interpersonal
Aggression
Items: 1, 2, 3, 4, 5, 6, 7, 9, 11, 12, 14, 17,
23
Factor 2: Professionalism
Items: 8, 13, 15, 16, 18, 19, 20, 21, 24, 26
Critical Items:
Items: 10, 22, 25, 28
1. Aggression: Non-Verbal
2. Aggression: Physical
3. Aggression: Verbal
4. Aggression: Written
5. Arrogance
6. Blaming Others
7. Disobeying Rules
8. Disrespect for
Staff/Colleagues
Interpersonal Aggression
Interpersonal Aggression
Interpersonal Aggression
Interpersonal Aggression
Interpersonal Aggression
Interpersonal Aggression
Interpersonal Aggression
Professionalism
9. Ease of Working With Ratee
10. Employee Turnover
Intentions
Interpersonal Aggression
Critical Item
11. Exploitation of Others
12. Intimidation
13. Lack of Empathy
14. Lack of Multicultural
Sensitivity
Interpersonal Aggression
Interpersonal Aggression
Professionalism
Interpersonal Aggression
15. Lack of Openness to Others
16. Mood Instability
17. Poor Citizenship
Professionalism
Professionalism
Interpersonal Aggression
18. Poor Stress Management
19. Problematic Perfectionism
20. Response to Criticism
21. Rigidity
22. Sexual Inappropriateness
23. Suspiciousness
24. Tardiness
25. Sexual and inappropriate
Remarks
Professionalism
Professionalism
Professionalism
Professionalism
Critical Item
Interpersonal Aggression
Professionalism
Critical Item
26. Substance Abuse
27. Sexual Inappropriateness
28. Prescribing medications
Professionalism
Critical Item
Critical Item
Disruptive Behavior Instrument
Total Variance Explained
Factor
Extraction Sums of Squared Loadings
Rotation Sums of
Squared Loadings
Total
% of Variance
Cumulative %
Total
1
10.463
38.750
38.750
9.296
2
1.836
6.801
45.551
7.498
3
1.408
5.215
50.766
1.655
Extraction Method: Principal Component Analysis.
a. When components are correlated, sums of squared loadings cannot be added to obtain a total
variance.
Disruptive Behavior Instrument
Component Correlation Matrix
Factor
1
2
3
1
1.000
-.543
1.35
2
-.543
1.000
-.031
3
.135
-.031
1.000
Extraction Method: Principal Component Analysis.
Rotation Method: Promax with Kaiser Normalization.
Disruptive Behavior Instrument
Reliability for Three Factors
Cronbach’s Alpha (α)
N of Items
Factor 1
.922
13
Factor 2
.879
9
Factor 3
.305
4
Poor reliability for items on Factor 3 suggests that a 2 Factor structure may
be a better fit.
Disruptive Behavior Instrument
2 Factors
Interpersonal Aggression
• Aggression: Non-verbal
• Aggression: Physical
• Aggression: Verbal
• Aggression: Written
• Arrogance
• Blaming Others
• Disobeying Rules
• Exploitation of Others
• Intimidation
• Lack of Multicultural Sensitivity
• Poor Citizenship
• Suspiciousness
Professionalism
• Disrespect for
Staff/Colleagues
• Lack of Empathy
• Lack of Openness to Others
• Mood Instability
• Poor Stress Management
• Problematic Perfectionism
• Response to Criticism
• Rigidity
• Tardiness
Disruptive Behavior Instrument

Instead of a third factor, items were named “critical items”.

5 Critical Items:

“I think about quitting or transferring to a different job because of
the ratee.”

“Touches or looks at one or more patients/coworkers in an
inappropriate/sexually suggestive manner.”

“Makes sexual remarks in the workplace that are inappropriate
or tells jokes that makes others uncomfortable.”

“Performs unnecessary exams on female and/or male patients.”

“Demonstrates lax boundaries when prescribing controlled
substances, giving out samples, or storing samples of
controlled substances.”
Disruptive Behavior Instrument
 Conclusions/Limitations:
 We
have 2 main factors and 5 critical
items.
 Two factors:
 Interpersonal Aggression
 Professionalism
Future Directions:






Measuring and refining Inter-rater Reliability
Measuring and refining Test-Retest Reliability
How is this data utilized?
What are cutoff scores?
Can we track disruptive behavior without
treatment?
Can instrument be utilized outside of medicine?
We do not deal much in
fact when we are
contemplating ourselves.
Mark Twain