Estimation and Adjustment of Bias in
Randomised Evidence Using Mixed
Treatment Comparison Meta-analysis
Sofia Dias, NJ Welton, AE Ades
with Valeria Marinho, Georgia Salanti, Julian Higgins
Avon RSS, May 2010
Department of Community Based Medicine
Overview
• Motivation
• Treatment networks and MTC
• Adjusting for Bias in Mixed Treatment
Comparisons Meta-analysis (MTC)
• The MTC model
• Example: Fluoride dataset
• Probability of bias model
• Results and Conclusions
2
Mixed Treatment Comparisons
• Often more than two treatments for a given condition
• Network of trials comparing different interventions
for a condition
• Direct and indirect evidence available on treatment effects
• Because of the network structure, there is enough
information to estimate and adjust for bias within the
network
• For bias adjustment, there is no need to rely on
exchangeability assumption between meta-analyses in
different fields
3
Example: The Fluoride Data
•
6 different interventions for preventing dental
caries in children and adolescents
1.
2.
3.
4.
5.
6.
•
No Treatment
Placebo
Fluoride in Toothpaste
Fluoride in Rinse
Fluoride in Gel
Fluoride in Varnish
Active Treatments
From 6 Cochrane Reviews*
4
*Marinho et al., 2002; 2003; 2004 (Cochrane Library)
Network and Number of trials
1
T
Pl
1
4
6
13
3
• 130 trials
3
1
69
NT
4
9
31
G
1
V
4
• eight 3-arm trials
• one 4-arm trial
• 150 pairwise
comparisons
R
5
Introduction to MTC
1. Six treatments 1,2,3,4,5,6
2. Take treatment 1 (No Treatment) as reference
3. Then the treatment effects d1k of all other
treatments relative to 1 are the basic parameters
4. Given them priors:
d1,2, d1,3,…, d1,6~ N(0,1002)
6
Functional parameters in MTC
• The remaining contrasts are functional parameters
d2,3 = d1,3 – d1,2
d2,4 = d1,4 – d1,2
CONSISTENCY assumptions
…
1
2
3
d4,6 = d1,6 – d1,4
d5,6 = d1,6 – d1,5
• Any information on functional parameters tells us
indirectly about basic parameters
• Either FE or RE model satisfying these conditions
7
Notation
• Data
i = 1,…,130 study index
k = 1, 2, 3,…,6 treatment index
rik – number of caries occurring in trial i,
treatment k, during the trial follow-up period
Eik – exposure time in arm k of trial i
(in person years)
Fluoride: Poisson MTC RE model
i = 1,…,130
rik ~ Poisson(ik Eik )
Exposure time in person years
rate at which events occur in arm k of trial i
log(ik )  i  ik I k 1
 ik ~ N  d1,t  d1,t ,
ik
i1
MTC consistency
equations
2

Priors
 ~ U (0,10)
d1, j ~ N (0,1002 ) j  2,..., 6
i ~ N (0,1002 ) i  1,...,130
9
MTC results: LHR relative to No Treatment
Pl
T
R
Residual
deviance is
278.6
G
(270 data points)
V
-.8
-.6
-.4
-.2
0
10
6
Posterior mean of residual deviances for each point
3
63
102
1
2
42
0
MTC
4
5
42
20
40
60
study number
80
100
120
11
Check how evidence is combined
in the network
• Poor fit can indicate inconsistency in the
network
• For each pair, separate direct evidence from
indirect evidence implied by the rest of the
network*
• Can see how evidence is combined in the
network to give overall MTC estimate
• Helpful to locate pairs of comparisons where
there may be problems
12
*Dias et al., Stats in Med. 2010
LHR for Placebo v Toothpaste
15
(Pl,T) is split
10
Direct
MTC
5
Density
Bayesian
p-value
= 0.32
0
Indirect
-1.0
-0.5
0.0
0.5
13
log-hazard ratio
LHR for Placebo v Varnish
10
15
(Pl,V) is split
Density
Bayesian
p-value
= 0.04
5
MTC
Indirect
0
Direct
-1.0
-0.5
0.0
0.5
14
log-hazard ratio
LHR for Rinse v Varnish
10
15
(R,V) is split
Density
Bayesian
p-value
= 0.02
5
MTC
Indirect
0
Direct
-1.0
-0.5
0.0
0.5
15
log-hazard ratio
But we have additional information on the risk of bias of all included
studies
BIAS MODELS
16
Treatments
NT
P
T
R
G
Allocation
concealment
No of
studies
V
Total
Blinding
adequate
unclear
inadequate
Double
Single
?
1
0
1
0
0
1
0
4
1
3
0
3
1
0
3
0
3
0
1
0
2
1
0
1
0
1
0
0
3
0
2
1
0
2
1
9
0
5
4
0
6
3
4
0
3
1
0
3
1
61
8
46
7
61
0
0
25
2
20
3
22
0
3
9
0
6
3
9
0
0
3
0
3
0
3
0
0
1
0
1
0
1
0
0
1
0
0
1
0
1
0
4
0
3
1
2
2
0
1
0
1
0
0
1
0
130
11
98
21
103
17
10
17
MTC RE model with bias
log(ik )  i  ik  ik Xi  Ik 1
1 if study at risk of bias
Xi  
0 otherwise
ik ~ N (bi ,  )
 ik ~ N  d1,t  d1,t ,
ik
i1
MTC consistency
equations
2

Priors
 ~ U (0,10)
d1,k ~ N (0,1002 ) k  2,..., 6
i ~ N (0,1002 ) i  1,...,130
18
MTC Bias Model
• Assume non-zero mean bias, bi = b ≠ 0, in
comparisons of NT or Pl with Active
treatments
• For Active-Active comparisons assume mean
bias is zero
• Expect bias to increase size of treatment effect:
b<0
19
Fluoride: Risk of Bias indicators
• Allocation concealment
•
•
•
•
Best empirical evidence of bias
But… 98/130 studies ‘unclear’
Only 11/130 studies ‘adequate’
Some comparisons have no adequately concealed
trials
• Blinding also available to inform risk of bias
status
• Used “Any bias” as a composite indicator of bias:
54/130 studies at risk of bias.
20
Probability of Bias Model
• Any study with unclear allocation
concealment has a probability p of being at
risk of bias
• Adequately concealed trials are not at risk of
bias
• Inadequately concealed trials are at risk of bias
• Use only allocation concealment as bias
indicator
• Bias terms identifiable in this rich network
21
Probability of Bias Model
ik  i  ik  ik Xi  Ik 1
1

X i   Bi
0

if allocation concealment is inadequate
if allocation concealment is unclear
if allocation concealment is adequate
Bi ~ Bernoulli( p) and p ~ Beta(1,1)
22
Comparing Model Fit
ResDev* pD
MTC with no bias adjustment
DIC
Between trial
heterogeneity
278.6
259.3 537.9 0.22 (0.19, 0.26)
AnyBias
277.6
257.9 535.5 0.15 (0.12, 0.18)
Probability of bias
274.6
253.0 527.6 0.12 (0.10, 0.15)
Bias adjustment
* Compare with 270 data points
23
Posterior mean of residual deviances for each
point: MTC and Prob of bias models
3
2
1
42 102
42
63
0
Probab of bias
4
5
6
Study 42: Placebo v
Toothpaste
(1 of 69 trials)
Allocation
concealment unclear
Study 63: No Treat v
Varnish
(1 of 4 trials)
Allocation
concealment unclear
and not “double
blind”
Study 102: Placebo
v Varnish
(1 of 3 trials)
Allocation
concealment unclear
0
1
2
3
MTC
4
5
6
24
Treatment effects relative to No Treatment (LHR)
Unadjusted MTC (solid) and Probability of Bias model (dashed)
Pl
T
R
G
V
25
-.8
-.6
-.4
-.2
0
Varnish effects
• Cochrane Review to assess efficacy of
Fluoride Varnish (Marinho et al, 2004)
• Noted that the small number and poor
methodological quality of varnish trials might
be overestimating the true effect of this
intervention.
• The results of the bias-adjusted analysis
support this hypothesis.
26
Which treatment is best?
Unadjusted MTC Bias-adjusted MTC
Probability
Probability
Best (%) Rank
Best (%)
Rank
No Treatment
0
6
0
6
Placebo
0
5
0
5
Toothpaste
3.6
2.9
9.3
2.7
Rinse
4.1
2.8
53.8
1.6
Gel
3.7
3.2
12.4
2.9
Varnish
88.5
1.2
24.6
2.8
27
Results: Probability of Bias
• Bias
• posterior mean = -0.19, CrI (-0.36, -0.02)
• posterior sd = 0.40, CrI (0.29, 0.55)
• Trials with unclear allocation concealment are at risk
of bias with probability p
• Posterior mean of p = 0.13
• Model identified 5 trials (with unclear allocation
concealment) as having a high probability of bias
28
Prob of bias for studies with unclear allocation concealment
42
63
143
145
148
149
150
142
151
144
146
147
o – unclear
allocation
concealment
+ – unclear
allocation
concealment and
single blind
∆ – unclear
allocation
concealment and
unclear blinding
status
102
0.6
0.4
0.2
0.0
Proportion
0.8
1.0
1721
0
20
40
60
80
100
120
29
Study
Other findings
• Between trial heterogeneity in treatment
effects reduced in bias-adjusted model
• Model with Active-Active bias was also fitted
with similar results: Active-Active bias had
posterior mean of zero
• But assumptions on direction of bias…
• Assumed bias would favour the newest treatment
(also the most intensive)
30
Conclusions
• Bias estimation and adjustment possible within
MTC because there is a degree of redundancy
in the network
• Assumption that study specific biases are
exchangeable within the network
• Uses only internal evidence
• Weaker than required from using external evidence
• Ideas extend to multiple bias indicators
• But will need a very rich evidence structure
31
Consequences for Decision Modelling
• Uses only internal evidence
• May be more acceptable to patient groups,
pharmaceutical industry…
• Risk of bias indicator chosen based on empirical
research
• Results may change if different bias indicators
chosen
Again:
• Assessment of model fit & sensitivity analysis
crucial if decisions based on these models are to
have credence
32
References
• Our website: http://bristol.ac.uk/cobm/research/mpes
• Dias S, Welton NJ, Marinho VCC, Salanti G, Higgins JPT and
Ades AE (2010) Estimation and adjustment of Bias in randomised
evidence using Mixed Treatment Comparison Meta-analysis.
Journal of the Royal Statistical Society A, to appear Vol 173 issue
4 (available online).
• Dias S, Welton NJ, Caldwell DM and Ades AE (2010) Checking
consistency in mixed treatment comparison meta-analysis.
Statistics in Medicine, 29, 945-955.
• Schulz KF, Chalmers I, Hayes RJ and Altman DG (1995)
Empirical Evidence of Bias. Dimensions of Methodological
Quality Associated With Estimates of Treatment Effects in
Controlled Trials. JAMA, 273, 408-412.
33