Class 13 and 14
Jacobson et al (1996)
APA (2006)Evidence Based
Practice in Psychology
1
Experimental Between Group
Designs
3.
Post-Test Only Control
Pre-Test -- Post-Test Control
Solomon Four Group (combination of

Factorial Design
1.
2.


1 and 2 above)
more than one independent variable; interactions gender x treatment
Dependent Sample Design (Matching)
2
Experimental Between Group
Designs





Post-Test Only Control
Pre-Test -- Post-Test Control
Solomon Four Group
Factorial Design (Treatment X Therapist)
Dependent Sample Design (Matching)

# of previous episodes and severity depression, presence of dysthimia,
gender, marital stauts
3
Types Outcome Studies
Kazdin (chap 18)
1.
2.
3.
4.
5.
6.
7.
Treatment Package Strategy
Dismantling Strategy
Constructive Strategy
Parametric Strategy (structural components)
Common factors Control Group
Comparative Outcome Strategy
Client and Therapist Variation Strategy
Moderation Designs
Types Outcome Studies
Kazdin (chap 18)
1.
2.
1.
2.
3.
4.
5.
Treatment Package Strategy
Dismantling Strategy
What are the active ingredients ?
Constructive Strategy
Parametric Strategy (structural components)
Common factors Control Group
Comparative Outcome Strategy
Client and Therapist Variation Strategy
Moderation Designs
Research Focus
 How
treatment effects change:
identify change mechanisms
VS.
 How
well treatment works
6
Change Mechanisms

Activation Hypotheses ( BA)


Coping Skills Hypotheses (AT)


(activation)
Change behaviors- become active and access
sources of reinforcement- occurs early in therapy
Learn effective cognitive coping strategies
(coping + activation)
Beck Hypotheses (CT)

Change cognitive structures or core schemas
(coping + activation + cognitive schemas)
7
Treatment Groups

BA Condition: Behavioral Activation Hypotheses:


AT Condition: Coping Skills Hypotheses



Behavioral Activation
Behavioral Activation
Coping Skills - Automatic Thoughts
CT Condition: Cognitive Therapy Hypotheses



Behavioral Activation
Automatic Thoughts
Core Schemas
Chapter 7
8
Mechanisms of Change
Construct
Measure

Behavioral Activation

Pleasant Events Schedule

Dysfunctional Thinking

Automatic Thoughts Q.

Cognitive Schemas

Expanded Attributional Styles
How did the authors examine if the mechanisms of
change worked in therapy as predicted by theory?
Chapter 7
9
How Treatment Effects Change

To what extent treatment groups differed in
post-test measures of each change
mechanism
Measure




Ttreat. G
Behavioral activation – Pleasant Event SCd
Coping skills- AT (Dist Thinking)
Core schemas
- Exp Attrb Style
BA
AT
CT
To what extent was there change from pre-to
post- treatment in each change mechanism
10
How Well Treatments Work

Which of the three treatment conditions
yielded better outcomes regarding
Depression at termination and at 6 month
follow up?




HRSD
BDI
Recovered
Improved Rates
11
Research Questions
Analyses
Treatment Outcome
group differences Depression
Type of Analyses
CVs
IVS
DVs
Mechanism of Change:
Treatment groups differed in
post-test measures of each
change mechanism (Behavioral
Activation, Automatic Thoughts &
Core Schemas)
Type of Analyses
CVs
IVS
DVs
Mechanism of Change:
Type and # of Analyses
Treatment groups change from
pre-to post- treatment in each
change mechanism (BA – AT-CS)
12
Research Questions
Analyses
Treatment Outcome
group differences in Depression
3 x 4 MANCOVA
CVs Pre-test BDI HRSD
IVs Treatments: CT, AT, BA
Therapists (n=4)
Treatment X Therapist
DVs Post-Test BDI HRSD
Mechanism of Change:
Treatment groups differed in
post-test measures of each
change mechanism (Behavioral
Activation, Automatic Thoughts &
Core Schemas)
Mechanism of Change:
Treatment groups change from
pre-to post- treatment in each
change mechanism (BA – AT-CS)
13
How Well Treatments Work

Which of the three treatment conditions
yielded better Depression outcomes at
termination and at 6 month follow up?
3 X 4 MANCOVA
IVS
3 Treatments (CT, AT, BA)
4 Therapists
Treatment X Therapist (Interaction)
DVs - Depression
HRSD – Clinician
BDI Self- Report
14
Which group to use in outcome analyses
Group
Sessions
Intent-to treat
Total sample
n = 149
Dropouts
1<Sessions <12
(not included in MANCOVA analyses; p. 299)
n = 12
Completers
At least 12/20 sessions
n = 137
Maximum Completers 20 sessions
n = 129
15
Results MANCOVAsTreatment
Outcome: Post- Test Depression

Main Effect Treatment ?

Main Effect Therapist

Treatment X Therapist Interaction ?
?
Results p. 300 -- ANOVAs Table 3
16
Results MANCOVAs Treatment
Outcome: Post- Test BID & HRSD
Treatment Group
N=
149
129
137

Main Effect Treatment
NS
NS
NS

Main Effect Therapist
NS
NS
NS

Treatment X Therapist
NS
NS
NS
Results p. 300-- ANOVAs Table 3
17
Table 3 -- ANCOVAS
??
18
19
ANCOVA Post-Test
Main Effects for Therapists
Post-Test Measures
Therapists
BDI
HRSD
TH-1
BDIT1
HRSDT1
TH-2
BDIT2
HRSDT2
TH-3
BDIT3
HRSDT3
TH-4
BDIT4
HRSDT4
20
ANCOVA - Post-Test
Effects Therapist X Treatment Interaction
Therapist
BDI
Post-test Scores by
Treatment
BA
AT
CT
HRSD
Post-test Scores by
Treatment
BA
AT
CT
TH-1
BDIT1
BDIT1 BDIT1
HRSDT1
HRSDT1
HRSDT1
TH-2
BDIT2
BDIT2 BDIT2
HRSDT2
HRSDT2
HRSD T2
TH-3
BDIT3
BDIT3 BDIT3
HRSDT3
HRSDT3
HRSDT3
TH-4
BDIT4
BDIT4 BDIT4
HRSDT4
HRSDT4
HRSD T4
21
Outcome: Follow-up 6 months

Overall Impact of Therapy





ANCOVAS
IV:Treatments
DVs: Follow-Up HRSD BDI
Covariate: Pretest Score
Changes in Follow-up Time




ANCOVAS
IV: Treatments
DVs: Follow-Up HRSD BDI
Covariate: ???
22
Outcome: Follow-up 6 months

Overall Impact of Therapy





ANCOVAS
IV:Treatments
DVs: Follow-Up HRSD BDI
Covariate: Pretest Score
Changes in Follow-up Time




ANCOVAS
IV: Treatments
DVs: Follow-Up HRSD BDI
Covariate: ???
23
Outcome: Follow-up 6 months

Overall Impact of Therapy





ANCOVAS
IV:Treatments
DVs: Follow-Up HRSD BDI
Covariate: Pretest residualized change score
Changes in Follow-up Time




ANCOVAS
IV: Treatments
DVs: Follow-Up HRSD BDI
Covariate: Pre-test score
24
Clinical Significance: Termination

Improved

No major depression at post-test:


LIFE Interview: psychiatric rating
Recovered

No major depression and BDI<8
25
26
Clinical Significance:
Mean Improvement/Recovered Rates
across Treatments – Post -Test (p.299 text)
Group
Intent-to treat
Total sample (149)
Completers 12/20
N=37
Maximum Completers
N = 129
Dropouts
N=12
Improved
Recovered
62.3%
51.5%
58.3%
58.3%
66.0%
54.5%
16.7%
5.6%
27
28
Conclusion
29
Conclusion

12 to 20 therapy sessions of either Behavioral
Activation (BA), Automatic Thoughts (AT) and
BA or Schema Restructuring and BA and AT
results in similar outcome in treating depression
for adult clients who meet exclusionary criteria
and were treated by highly trained and
monitored therapists.

Regarding the mechanisms of change in CBT??
30
Mechanisms of Change
Construct
Measure

Behavioral Activation

Pleasant Events Schedule

Dysfunctional Thinking

Automatic Thoughts Q.

Cognitive Schemas

Expanded Attributional Styles
How did the authors examine to what
extent the mechanisms of change worked
in therapy as predicted by theory?
Chapter 7
31
How Treatment Effects Change

To what extent treatment groups differed in
post-test measures of each change
mechanism
Measure




Behavioral activation – Pleasant Event SCd
Coping skills- AT (Dist Thinking)
Core schemas
- Exp Attrb Style
Ttreat. Grps
BA
AT
CT
To what extent was there change from pre-to
post- treatment in each change mechanism
32
Research Questions
Analyses
Treatment Outcome
group differences Depression
3 x 4 MANCOVA
CVs Pre-test BDI HRSD
IVs Treatments: CT, AT, BA
Therapists (n=4)
Treatment X Therapist
DVs Post-Test BDI HRSD
Mechanism of Change:
Treatment groups differed in
post-test measures of each
change mechanism (Behavioral
Activation, Automatic Thoughts &
Core Schemas)
Mechanism of Change:
Treatment groups change from
pre-to post- treatment in each
change mechanism (BA – AT-CS)
33
Research Questions
Analyses
Treatment Outcome
group differences Depression
3 x 4 MANCOVA
CVs Pre-test BDI HRSD
IVs Treatments: CT, AT, BA
Therapists
Treatment X Therapist
DVs Post-Test BDI HRSD
Mechanism of Change:
Treatment groups differed in posttest measures of each change
mechanism (Behavioral Activation,
Automatic Thoughts & Core Schemas)
3 ANCOVAS
(one CV & DV per analysis)
CV Pre-test:
E Attrb Q, ATQ, PES
IV Treatments: (CT, AT, BA)
DV Post-test: E Attrb Q, ATQ, PES
Mechanism of Change:
Treatment groups change from
pre-to post- treatment in each
change mechanism (BA – AT-CS)
34
Research Questions
Analyses
Treatment Outcome
group differences in Depression
3 x 4 MANCOVA
CVs Pre-test BDI HRSD
IVs Treatments: CT, AT, BA
Therapists
Treatment X Therapist
DVs Post-Test BDI HRSD
Mechanism of Change:
Treatment groups differed in posttest measures of each change
mechanism (Behavioral Activation,
3 ANCOVAS
(one CV & DV per analysis)
Automatic Thoughts & Core Schemas)
CV Pre-test:
E Attrb Q, ATQ, PES
IV Treatments: (CT, AT, BA)
DV Post-test: E Attrb Q, ATQ, PES
Mechanism of Change:
Treatment groups change from preto post- treatment in each change
mechanism (BA – AT-CS)
3 Paired T Tests per Treatment
Pre-post differences in each
mechanism of change for each
treatment : E Attrb Q, ATQ, PES
35
Paired T Tests: Pre-test/Post-test Change
Treatment
Pleasant
Events
Schedule
Behavioral
Activation PESch**
Automatic
Thoughts
Questionnaires
Expanded
Attributional
Styles
ATQ
EAS
Automatic
Thoughts
PESch**
ATQ**
EAS
Cognitive
Therapy
PESch**
ATQ**
EAS**
Clients in all conditions significantly
improved on the three measures (p. 301)
36
Residual change score (RCS) in Mechanism of
Change from Pre-to Mid-T and RCS in Depression
from Mid-T to Post-T (p.301)
37
Threats to Statistical Conclusion Validity
Jacobson et al. (1996)
Are the observed relations among variables accurate?
1. Power
2. Unreliability of
Treatment
Implementation
38
Threats to Statistical Conclusion Validity
Are the observed relations among variables accurate?
1. Power
2. Unreliability of
Treatment
Implementation
•N=149 and 3 groups/about 50 participants
per group. Large N for an Exp study (+)
•Outcome measures are well-known – high
internal reliability (+)
•No information is given about alphas with
study sample Power analyses not reported
•Therapists were experienced in CT and
trained for study (+)
•Training followed a manual prepared for
each treatment group (+)
•Therapy tapes were listened on an ongoing basis and therapists flagged if they
deviated (+)
•Analyses of adherence based on 27 taped
sessions showed that treatments were
distinct and consistent with their respective
protocols (+)
39
Threats to Statistical Conclusion Validity
Are the observed relations among variables accurate?
3. Extraneous Variance
in the Experimental
Setting
4. Heterogeneity of
Participants
40
Threats to Statistical Conclusion Validity
Are the observed relations among variables accurate?
3. Extraneous Variance
in the Experimental
Setting
4. Heterogeneity of
Participants
•Do not appear to be any – The same four
therapists administered all treatments
•However procedures and settings are not
described in detail
•Study had many exclusion criteria
including co-morbidity, taking psychotropic
medication, suicidal… p. 296. (+)
•However, don’t know #s in pool of
volunteers from which the 152 accepted
patients were taken from
•72% were women (+)
•No info regarding race/ethnicity, SES (-)
41
Threats to Internal Validity
Can we conclude that there is a causal relation between the IV and
the DV?
1. Selection to Treat.
Groups
2. History
3. Attrition
4. Repeated Testing
Effects
5. Reaction to
Control Group
Assignment
42
Threats to Internal Validity
Can we conclude that there is a causal relation between the IV and
the DV?
1. Selection to Treat.
Groups
•Used Randomization after matching for
episodes dep., dysthimia, severity of
depression, gender , marital status (+)
2. History
•Therapy appeared to occur for everyone at
once – but this is not addressed
3. Attrition
•Small -only 15 out of 152 - 8% -Attrition during
acute treatment was comparable across
treatment conditions – p.296 right (+)
4. Repeated Testing
Effects
•Five administration of measures – at pre-test;
post –test and 6, 12, 18 month follow-up (-)
5. Reaction to
Control Group
Assignment
•No placebo or no-treatment control group (+)
•Every one received treatment (+).
•Maybe some realized not receiving whole
treatment (?)
43
Threats to Construct Validity
To what extent variables capture desired constructs
1.
Mono-Operation Bias
2.
Mono-Method Bias
3.
Experimenter
Expectancies
44
Threats to Construct Validity
To what extent variables capture desired constructs
1.
2.
3.
Mono-Operation Bias
Mono-Method Bias
Experimenter
Expectancies
•Used two measures to assess
outcome: Depression (+)
•HRSD is interview based and
BDI is self-report (+)
•Potential risk (-)
•Should have worked in favor of
the CT condition that included all
aspects of CBT-- therapists and
researchers were aligned with
CBT
45
Threats to External Validity
Can we generalize observed relations across persons,
settings and times
1.
Person-Units
2. Outcome Measures
4.
Settings
46
Threats to External Validity
Can we generalize observed relations across persons,
settings and times
1.
Person-Units
2. Outcome Measures
4.
Settings
•Highly selected sample – p. 296 (-)
•Exclusionary criteria (-)
•Primarily female = 73% (-)
•No info regarding race/ethnicity, SES (-)
•They used interview based and self
report measures of Depression +
•Examined outcome using clinical
significance index +
•Empirical Question…..
47
Empirically Supported
Therapies - EST
1993 APA Division 12
 Identify Efficacious treatments for
specific disorders
 FDA Criteria: Specificity



Specific ingredients vs. placebo effects
Randomized double-blind, placebo
control design
48
What components of treatment are
responsible for therapy effects?

Specific Ingredients


Common Factors --- Placebo

49
unique to each therapy approach
underlie most approaches
Scientific advances that lead to the
Randomized Placebo Control Group Design

Germany: 1850 Wundt’s experimental method in
psychology – 1st psychology lab

Britain:1800 Galton & Pearson – use of normal
curve in assessment of distribution mental abilities

France: Physicians start comparing treatments across
groups of patients: treatment groups
50
Scientific advances that lead to the
Randomized Placebo Control Group Design

1930s Fisher- ANOVA statistics - randomization

1930- 1950 Placebo treatment -
physiochemical ingredients vs. patients’
expectations, hopes, beliefs

Mesmerized…. Placebo
51
Well-established treatments
Division 12 Task Force Criteria
I.
At least two good between-group design
experiments must demonstrate efficacy in one
or more of the following ways:
A. Superiority to pill or psychotherapy placebo, or to
other treatment (wait list control is not enough)
B. Equivalence to already established treatment with
adequate sample sizes
53
OR
Well-established treatments (2)
II.
A large series of single-case design
experiments must demonstrate efficacy with
A. Use of good experimental design and
B. Comparison of intervention to another treatment
54
And,
Well-established treatments (3)
III.
Experiments must be conducted with
treatment manuals or equivalent clear
description of treatment
IV.
Characteristics of samples must be
specified
V.
Effects must be demonstrated by at least
two different investigators or teams
55
Probably efficacious treatments
I.
Two experiments must show that the treatment is
superior to waiting-list control group
OR
II.
One or more experiments must meet
well-established criteria IA or IB, III, and IV
above, but V is not met (2 investigator teams)
OR
III.
A small series of single-case design
experiments must meet well-establishedtreatment criteria
56
Treatment: Efficacious or
Probably Efficacious ?




Family Intervention (Smith et al.)
Anorexia in adolescent and young adult
women
Superior cognitive behavioral intervention
(well established treatment) in 2 randomized
clinical studies
Both studies conducted by Smith et al.
57
Evidence Based Psych.
Practice

Best available research

Results from randomized Controlled Clinical Trials
and other types of empirical studies

Clinical Expertise

Client Characteristics
58
Concerns EVT movement:



Brief, straightforward manualized treatments:
Cognitive Behavioral
Emphasis on specific effects/ignore common
factors
Lack of applicability to wide range of clients;


comorbidity, males, race, ethnicity, social class
Mandates to use EVT- restriction to choice of
treatment
59
APA Task Force Acceptable Research
Designs









Clinical observation
Qualitative studies
Systematic case studies
Single-case experimental designs
Public health and ethnographic research
Process–outcome studies
Studies in naturalistic settings
Randomized Controlled Clinical Trials
Meta-analysis
60
APA Task Force Research Designs
included in Div. 12 criteria for EVT









Clinical observation
Qualitative studies
Systematic case studies
Single-case experimental designs
Public health and ethnographic research
Process–outcome studies
Studies in naturalistic settings
Randomized Controlled Clinical Trials
Meta-analysis
61
Evidenced Based Practice
Empirically Supported Tmts.
Effectiveness/Clinical utility:
Efficacy:
Main Focus:
Main Focus:
Knowledge from clinical
expertise
Knowledge from clinical
expertise
Clients’ individual differences Clients’ individual differences:
Therapists’ individual
differences:
Therapists’ individual
differences;
62
Evidenced Based Practice
Empirically Supported Tmts.
Effectiveness: does it work;
generalizability; feasibility
Efficacy: does treatment
cause outcome
Main Focus: Client
Main Focus: Intervention
Knowledge from clinical
expertise is directly applied
Knowledge fromclinical
expertise
Indirect- to generate
hypotheses
Clients’ individual differences Clients’ individual differences:
Central
Nuisance
Therapists’ individual
differences: Central
Therapists’ individual
differences; Nuisance
63
EBP or EST??

Know the person who has the disorder

Know the disorder the person has
64
EBP or EST??

Know the person who has the disorder

Know the disorder the person has
65
Components Shared by Approaches to
Psychotherapy (Wampold, 2000)
1.
2.
Emotionally charged, confiding relationship
In a healing setting
3.
Rationale, conceptual scheme, or myth to
explains patient’s symptoms
4.
There is a ritual or procedure that engages
client and therapist based on the rationale (Frank
& Frank, 1991).
66
Common Elements in Therapeutic Rituals
and Procedures (1/2)
5.
The relationship helps combat clients’
feelings of alienation
6.
The process of therapy provides hope for
improvement
7.
Therapist provides new learning experiences
67
Common Elements in Therapeutic Rituals
and Procedures (2/2)
8.
Client’s emotions are aroused as a result of
therapy and the clients expects to improve
9.
Therapist enhances client’s sense of mastery
and self-efficacy
10.
Therapist provides opportunities for practice
68
Wampold concluded that:

Specific theoretical ingredients are not
differentially related to outcome

Specific ingredients are necessary to construct a
coherent treatment



In which therapists have faith, and
that provides a convincing rational for clients
Therefore, knowledge of theoretical approaches is
necessary to build coherent interventions and treatments
69