A chance for change:
Building an outcome monitoring feedback
system for outpatient mental health care
dr. Kim de Jong
“Without direct feedback on
how their clients are
progressing, clinicians are
essentially wearing a blindfold
while shooting at a target”
(Sapyta, Riemer & Bickman, 2005)
Effectiveness of psychotherapy
• RCTs show psychotherapy is successful: 67% of
patients improve reliably
• In clinical practice, results are less optimistic:
–35% improved reliably (Hansen, Lambert & Forman, 2002)
–effect sizes were 50% smaller (Barkham et al, 2008; Weisz et al., 1995)
–Selection bias in RCTs accounts in part for this: 57% of
patients eligible for inclusion improve (Blais et al, 2011)
• Monitoring patient progress might be a promising
way to improve outcomes
We need more efficient care
The evidence on feedback
• Efficacy of feedback is mixed:
–No effect on “on track” cases
–Effect on “not on track” cases ranges from small
to large effect sizes (e.g. Bickman et al, 2012; d = 0.20)
Shimokawa et al., 2010, d=0.92)
–Some studies show effect on treatment duration
–In some studies feedback has a negative effect on
outcome (e.g. Carlier et al., 2010)
• What are effective elements of feedback?
The evidence on feedback (2)
• Largest effects found by Lambert e.a.:
–High frequency (every session)
–Expected recovery curves and warning signals
(red flag)
–Clinical support tools on common factors
• Most feedback systems do not have these
elements
Feedback theory*
Goal
commitment
Goal
or standard
Feedback
propensity
Behavior
change
Valence (+ / -)
Content
Timing
Feedback
Self-efficacy
Perceived
validity
*Based on the CFIT model by Bickman et al.
Why do we need feedback?
I’m not that
satisfied with
the therapy…
Hmm… yes… I
understand…
That is part of
your
psychiatric
problems.
How well do therapists predict
treatment failure?
• Final Outcome was predicted for 550 Clients
• 3 were predicted to have a negative outcome,
whereas 40 had actual negative outcomes
• Staff were accurate 1 time (2,5%)
• Algorithms for feedback were correct 77% of
the time in predicting deteriorated patients
(Hannan, Lambert, Harmon, Nielsen, Smart, Shimokawa, Sutton, 2005)
How well do therapists predict? (2)
Feedback
n=169; de Jong & Peetoom, in preparation
FEEDBACK STUDIES
Problem statement
• Developing an outcome monitoring system for
outpatient mental health care in the
Netherlands
– Does outcome monitoring feedback improve outcomes?
– Does outcome monitoring feedback affect treatment
duration?
– Do therapist characteristic moderate the effectiveness of
feedback?
Feedback studies
• Study 1
– 3 groups: control, therapist , therapist+patient feedback (n=474)
– Feedback: 1 progress chart, feedback text, risk items
– Questionnaire (OQ-45) prior to every session
– Private practices and outpatient mental health care (institutions)
• Study 2
– 2 groups: control, feedback (n=418)
– Feedback: 2 progress charts, 2 tables, risk items
– OQ-45 prior to sessions 1-5, 10, 15, 20, etc.
– Outpatient mental health care (institutions)
– Examining the role of the therapist
Patient en therapist feedback
Results short-term therapies
Results long-term therapies
Research questions
• What is the effectiveness of ‘simple’ feedback?
• Do therapist characteristics influence the
effectiveness of feedback?
Study design
• Outpatient psychotherapy
• Random assignment to control group or
feedback condition (n=418)
• Administration of OQ-45 prior to sessions 1-5,
10, 15, 20, etc. (max. 1 year)
• Examining the role of the therapist
– Feedback related characteristics
– Self-efficacy
– Use of feedback
Feedback
• Feedback: 2 progress charts, 2 tables, risk
items
Effect feedback on outcome
95
OQ-45 Total score
85
75
NOT controle
NOT feedback
65
OT controle
55
OT feedback
45
35
Start
Eind
Effect feedback active use
95
OQ-45 Total score
85
75
NOT controle
NOT feedback
65
OT controle
55
OT feedback
45
35
Start
Eind
Therapist characteristics
• Effect on outcomes:
– Therapists with higher internal feedback propensity
achieve change slower
– Therapists with a positive attitude towards ROM , achieve
change faster
– Therapists with a higher self-efficacy achieve faster
change in NOT cases
• Effect on use of feedback
– Positive attitude related to use of feedback
– Women more likely to use feedback than men
(de Jong et al., 2012)
Attitude
• Higher self-efficacy correlated with more
positive attitude towards ROM
• Higher external feedback propensity
correlated with more positive attitude
• Personality of therapist
– Higher on Extraversion -> more positive attitude
– Higher on Neuroticism -> more negative attitude
(de Jong & Hordijk, in preparation)
Conclusion
• Feedback can help improve treatment
outcomes, but is a complex process and does
not work equally well under all circumstances
– Therapist do not always use feedback, even when it
is provided
– Therapists that have higher self-efficacy are more
likely to benefit from feedback
– Commitment to use feedback predicts actual use
and effectiveness
Currently running studies
• New patient groups:
–
–
–
–
Feedback + Clinical Support Tool (with GGZ NHN and Dimence)
Group psychotherapy (with GGZ NHN)
Youth mental health care (with Praktikon / BergOp)
Pain monitoring in chronic pain (with Antonius)
• Us of outcome monitoring in clinical practice
– Prediction model OQ built in RequestROM
– Stepwise indication from GP to Specialized GGZ (with GGZ NHN)
– Mechanisms of change in feedback (U Penn)
DISCUSSION
Implications for clinical practice
• Providing feedback to whom?
• Characteristics of the feedback
– Frequency of measurements
– Using prediction models
– Tailor feedback to therapist characteristics
– Measuring process variables as well as outcomes
QUESTIONS?
Contact me: KJong@fsw.leidenuniv.nl
Twitter: kdej_psyres
Website: www.kimdejong.net