Effectiveness of two-stage training for brief interventionists in a multi-site trial
Michael P. Bogenschutz,1 Alyssa A. Forcehimes,1 Christine Sanchez,1 Dennis M. Donovan,2 Chris Dunn, John Baer,2 Karin
Wilson,1 Raul N. Mandler,3 Harold I. Perl,3 Theresa B. Moyers1
1University
of New Mexico,
2University of Washington,
3National Institute on Drug Abuse
INTRODUCTION
Effective training in psychosocial treatment modalities is critical to
maintaining fidelity in both research and practice. Maintaining fidelity may be
particularly challenging in the Emergency Department settings due to the fast
pace and competing urgent and emergent priorities. Here we evaluate the
effectiveness of a two-stage training for interventionists participating in NIDA
Clinical Trials Network Study 0047: Screening, Motivational Assessment,
Referral, and Treatment in Emergency Departments (SMART-ED).
METHODS
The primary aim of the SMART-ED study is to contrast substance use and
substance-related outcomes among patients who endorse problematic
substance use during an ED visit and are randomly assigned to one of three
treatment conditions: 1) minimal screening only (MSO); 2) screening,
assessment, and referral to treatment (if indicated or requested) (SAR); and 3)
screening, assessment, and referral (if indicated or requested) plus a brief
intervention (BI) with two telephone follow-up booster sessions (BI-B).
Individuals presenting to an ED who endorse problematic substance use on
screening are randomized in 1:1:1 ratio to MSO vs. SAR vs. BI-B. Those
assigned to the BI-B group receive a brief intervention consisting of
motivational enhancement therapy (MET) adapted for use in the ED, followed
by the informational pamphlet and referral if indicated or requested. The BI-B
group also receive two booster telephone calls which will occur within 1 week
of enrollment if possible, and in no cases later than 1 month following
enrollment. The study was implemented in two waves, such that the first two
sites were received training and began recruitment first, followed 6 months
later by the second group of 4 sites. The booster interventionists were trained
separately and serve all 6 sites.
Prior to completing any training, interventionists provided informed consent
and completed questionnaires including demographic information, training
and experience, and knowledge of MI and other counseling approaches. The
Short Understanding of Substance Abuse Scale (SUS, Humphreys et al., 1996)
was used to assess their theoretical model for understanding addiction. This
scale yields scores for three reliable subscales representing the Disease Model,
the Psychological Model, and the Eclectic Model. Interventionists then
received a 2-day training in basic Motivational Interviewing skills, conducted
locally, followed 1 week later by a 2-day training in the specific counseling
intervention used in the trial, conducted as part of the national protocol startup meeting. They then completed practice sessions with consenting ED
patients. Audiotapes of these sessions were reviewed by an expert rater, using
the Motivational Interviewing Treatment Integrity scale (MITI, version 3.1.2)
(Moyers et al., 2005) and content checklists. The MITI yields 4 summary
scores as well as several calculated scores based on coding of utterances
during the rated tape. To become certified, interventionists had to score at least
4.0 (“competent”) on the 5-point Global Clinician Rating from the MITI and
80% on content checklists, for 2 out of 3 sessions.
RESULTS
Twenty-two counselors have received training to date across the two waves of
the study, plus two booster counselors. Table 1 summarizes counselor
demographics and scores on the SUS, as well as non-parametric tests for
differences between the wave 1 and wave 2. Wave 1 counselors were older
and lower on the Psychological Model scale of the SUS.
TABLE 1: Interventionist characteristics
Female Gender
Race
AI/AN
Asian
African-Am.
Caucasian
Ethnicity
Hispanic
Age
Wave 1
Wave 2
Significance
Booster
Total
5/8
9/14
N.S.
2/2
16/24
0/2
0/2
0/2
2/2
1/24
1/24
3/24
19/24
N.S
1/8
1/8
1/8
5/8
0/14
0/14
2/14
12/14
To explore interventionist characteristics that might account for variance in
treatment fidelity observed in the training cases, bivariate correlations were
computed between the MITI ratings and selected participant characteristics (See
Table 3). Age, counseling experience, and self-reported use and understanding of
MI were not significantly related to performance in training cases. Of the SUS
subscales, only Psychological Model scores were significantly (and negatively)
related to Global Clinical Ratings, MI Spirit, and Percent MI-Adherent.
TABLE 3: Correlates of treatment fidelity (Pearson’s r, p)
N.S..
2/8
36.0 (9.2)
1/14
29.9 (3.3)
.022
0/2
34.5 (6.4)
3/24
37.1 (6.8)
Years counseling exp.
1.7 (2.3)
1.8 (2.5)
N.S.
5.0 (5.7)
2.0 (2.7)
Licensed as counselor
4/8
6/14
N.S.
2/2
12/24
Understanding of
Substance Abuse
Disease Model
11.9 (6.1) 9.0 (6.5)
N.S.
7.0 (0.0)
9.8 (6.1)
Psychological Model 12.3 (2.2) 15.3 (2.8)
.021
12.5 (0.7)
14.0 (2.9)
Eclectic Model
15.5 (3.0) 17.0 (4.0) N.S.
20.5 (0.7)
16.8 (3.7)
Categorical measures shown as proportions, continuous measures shown as Mean (S.D.).
Between-group comparisons are for Wave 1 vs. Wave 2, and use Mann-Whitney U for scale
variables due to non-normal distributions.
MITI Global Clinician Ratings averaged 4.50 ± .56 on a scale of 1-5,
well above the threshold for competency (4.0), and higher than scores
typically recorded after a two-day training (See Table 2). Scores on the
other three summary ratings (MI Spirit, Direction, and Empathy) were
also well above the criterion for competency. The four computed scores
(Reflection to Question Ratio, Percent Open Questions, Percent Complex
Reflections, and Percent MI-Adherent) were all in the “Beginning
Proficiency” range, except for Percent Complex Reflections, which was
in the Competent range. All but 2 participants met criteria for
certification based on the first 2 practice sessions, and all were eventually
certified. All interventionists scored 100% on the content checklists for
both sessions. Surprisingly, significant differences were found between
Waves 1 and 2 on three of the four summary ratings as well as one of the
computed scores (Percent MI-Adherent), with higher scores for Wave 1 in
each case.
TABLE 2: Ratings of first two training cases
Wave 1
Wave 2
Significance Booster
Global Clinical
4.91 .15
4.20 .55 .001
TABLE
Rating 3: Regression model of predictors
MI Spirit
4.91 .18
4.06 .57 <.0005
5.00
.00
.56
of training scores
5.00
.00
4.42
.62
Direction
4.88
.35
4.50
.59
N.S.
5.00
.00
4.67
.52
Empathy
4.94
.18
4.29
.67
.007
5.00
.00
4.56
.61
Reflection to
Question Ratio
1.73
.77
1.55
.59
N.S.
1.08
.32
1.57
.64
Percent Open
Questions
Percent Complex
Reflections
46.5
15.5
52.4 14.6
N.S.
52.8
30.1
50.5
15.5
57.4
6.2
53.6
7.1
N.S.
52.0
3.5
54.7
6.7
0
89.2
13.4 .011
100
0
93.7
11.4
Percent MI-Adherent 100
MI Spirit
Direction
Empathy
Ratio of
Questions
to
Reflections
Percent
Open
Questions
Percent
Complex
Reflections
Percent
MIadherent
Age
.352
.091
.376
.070
.277
.190
.229
.281
.120
.577
.291
.168
.241
.257
.184
.389
Self-reported use
of MI approach
.256
.239
.316
.141
-.041
.852
.242
.265
-.087
.693
-.282
.192
-.056
.799
.026
.906
Self-reported MI
understanding
.148
.499
.251
.248
-.195
.373
.056
.800
-.177
.418
.003
.988
.005
.982
.160
.467
Years of
experience
.152
.478
.142
.509
.313
.137
.005
.982
-.270
.201
.069
.748
.001
.995
.058
.787
Disease Model
.248
.254
.298
.168
.008
.969
.227
.299
.085
.699
-.045
.839
.025
.911
.352
.099
Psychological
Model
-.444
.030
-.443
.030
-.364
.080
-.358
.086
-.296
.161
-.020
.927
.047
.828
-.433
.034
Eclectic Model
.028
.896
.015
.943
.154
.472
-.062
.775
-.258
.223
.033
.878
-.187
.383
-.024
.911
DISCUSSION
Overall, the performance of interventionist on pilot cases in this trial was
very good, even though the interventionists were not highly experienced and
only half were licensed. This supports the effectiveness of the two-stage
training approach that was used. Given small sample size and the large
number of comparisons that were made, the inferential findings of this study
should be interpreted cautiously. Still, the relatively large differences in
performance between waves 1 and 2 are surprising, and are not readily
explained by differences between the groups. This raises the possibility that
differences in the first (local) phase of the training could have accounted for
some of the observed differences. The finding that Psychological Model
scores were inversely related to proficiency scores was unexpected, and
requires replication. It remains to be seen whether session ratings during the
training phase will predict performance during the main trial.
CONCLUSIONS
Total
4.50
Global
Clinical
Rating
The two-stage interventionist training used in this study produced very good
results across the two waves of interventionists trained. Focusing first on
fundamentals and later on specific intervention content may bestow an
advantage for learning and implementing brief interventions based on a
motivational interviewing approach. The significant differences in ratings
between the two waves underscore the need for formal fidelity monitoring
even when training procedures are highly specified.
REFERENCES
Moyers, T. B., Martin, T., Manuel, J. K., Hendrickson, S. M. L., & Miller, W. R. (2005). Assessing
competence in the use of motivational interviewing. Journal of Substance Abuse Treatment, 28, 19–26.
Humphreys K, Greenbaum MA, et al. (1996). "Reliability, validity, and normative data for a short version
of the Understanding of Alcoholism Scale." Psychology of Addictive Behaviors 10(1): 38-44.
ACKNOWLEDGEMENTS
Supported by the National Drug Abuse Treatment Clinical Trials Network