Webappendix 2
1. METHODS USED
A. Effect Size : Raw Mean Difference (RMD)





A.1. Assessment of publication bias
Funnel Plot
A.2. Frequentist Approach
Simple random effects meta-analysis (simple REMA)
Network random-effects meta-analysis (NMA)
Simple random-effects meta-regression analysis (simple RE meta-regression) with one covariate:
initial severity
Simple RE meta-regression analysis with two covariates: initial severity and publication year
A.3. Bayesian Approach
 Simple REMA
 NMA
 NMA random-effects meta-regression analysis
o NMA random-effects meta-regression analysis using 12 different prior distributions for the
heterogeneity
 Simple RE meta-regression analysis with one covariate: initial severity
o Simple RE meta-regression analysis with one covariate: initial severity, using 12 different prior
distributions for the heterogeneity
B. Effect Size : Standardised Mean Difference (SMD)





B.1. Assessment of publication bias
Funnel Plot
B.2. Frequentist Approach
Simple REMA
NMA
Simple RE meta-regression analysis with one covariate: initial severity
Simple RE meta-regression analysis with two covariates: initial severity and publication year
B.3. Bayesian Approach
 Simple REMA
 NMA
 NMA RE meta-regression analysis
o NMA RE meta-regression analysis using 12 different prior distributions for the heterogeneity
 Simple RE meta-regression analysis with one covariate: initial severity
o Simple RE meta-regression analysis with one covariate: initial severity using 12 different prior
distributions for the heterogeneity
1
2. DESCRIPTION OF THE MODELS
Meta-analysis models can be viewed equivalently either as a special case of a weighted
linear regression or as a hierarchical model. In a frequentist framework linear regression
approaches are used (known also as ‘contrast-based’ models), whereas in a Bayesian
implementation we use a hierarchical approach (known also as ‘arm-based’ models).
All frequentist approaches were implemented in STATA, whereas all Bayesian models in
the freely available software WinBUGS 1.4.31. For all Bayesian models two chains, after a burnin period of 10000 Markov Chain Monte Carlo (MCMC) draws, were run until convergence. We
used a visual inspection of the two Markov chains in the history plot to judge whether
convergence was achieved.
 Simple random effects meta-analysis (REMA)
Simple REMA ‘contrast-based’ model
Let 𝑦𝑖,𝑇𝑃 be the observed relative treatment effect of treatment T relative to placebo (P),
e.g. raw mean difference (RMD) between the 2 groups, in study 𝑖 = 1, . . 𝑘, with variance 𝑣𝑖,𝑇𝑃 .
Assuming 𝑚𝑖,𝑃 and 𝑚𝑖,𝑇 are the individual study means in placebo and treatment group
respectively, 𝑠𝑑𝑖𝑃 and 𝑠𝑑𝑖𝑇 represent the standard deviation in each group and 𝑛𝑖𝑃 and 𝑛𝑖𝑇 are
the respective sample sizes, then the RMD and its variance are obtained as
𝑦𝑖,𝑇𝑃 = 𝑚𝑖,𝑇 − 𝑚𝑖,𝑃
𝑣𝑖,𝑇𝑃
2
2
𝑠𝑑𝑖𝑃
𝑠𝑑𝑖𝑇
=
+
𝑛𝑖𝑃
𝑛𝑖𝑇
The model is structured under the assumption that the study variances, 𝑣𝑖,𝑇𝑃 are fixed and
known. Under the random effects (RE) model the observed effect measures are modelled as
𝑦𝑖,𝑇𝑃 = 𝜇 𝑇𝑃 + 𝛿𝑖,𝑇𝑃 + 𝜀𝑖,𝑇𝑃
𝜀𝑖,𝑇𝑃 ~𝛮(0, 𝑣𝑖,𝑇𝑃 ),
2
𝛿𝑖,𝑇𝑃 ~𝑁(0, 𝜏 𝑇𝑃
)
where 𝜇 𝑇𝑃 is the mean of the distribution of the underlying effects, 𝛿𝑖,𝑇𝑃 represent the random
variation in the treatment effects across studies (RE) and 𝜀𝑖,𝑇𝑃 is the random error in study 𝑖 =
1, . . 𝑘. We set 𝜏 2 the between-study variability due to differences in the true effect sizes rather
than chance, and we call it heterogeneity.
In the frequentist setting we use the inverse variance method, where the summary
treatment effect 𝜇 𝑇𝑃 and its variance are estimated as
2
𝜇 𝑇𝑃
∑𝑘𝑖=1 𝑤𝑖,𝑇𝑃 𝑦𝑖,𝑇𝑃
=
∑𝑘𝑖=1 𝑤𝑖,𝑇𝑃
𝑉𝑎𝑟(𝜇 𝑇𝑃 ) =
1
∑𝑘𝑖=1 𝑤𝑖,𝑇𝑃
with 𝑤𝑖,𝑇𝑃 = 1/(𝑣𝑖,𝑇𝑃 + 𝜏 2 ) representing the weight assigned to each study. We estimate 𝜏 2
using the DerSimonian and Laird (DL) estimator2. We fitted simple REMA in STATA using
metan3 command.
Simple REMA ‘arm-based’ model
Equivalently, under the random effects meta-analysis the observed treatment effect 𝑦𝑖,𝑇𝑃 is
normally distributed with mean 𝜃𝑖,𝑇𝑃 and uncertainty reflected by the study variance 𝑣𝑖,𝑇𝑃 .
𝑦𝑖,𝑇𝑃 ~𝑁(𝜃𝑖,𝑇𝑃 , 𝑣𝑖,𝑇𝑃 )
Both linear regression and hierarchical models are equivalent as 𝛿𝑖,𝑇𝑃 is the difference between
the mean 𝜇 𝑇𝑃 and the underlying study-specific mean 𝜃𝑖,𝑇𝑃 .We assume that the true effects 𝜃𝑖,𝑇𝑃
vary between studies and are sampled from a normal distribution with expectation 𝜇 𝑇𝑃 .
2
𝜃𝑖,𝑇𝑃 ~𝛮(𝜇 𝑇𝑃 , 𝜏 𝑇𝑃
)
In the Bayesian framework we use the exact hierarchical model:
2
⁄𝑛𝑖𝑃 )
𝑚𝑖𝑃 ~𝑁(𝜆𝑖𝑝 , 𝑠𝑑𝑖𝑃
2
⁄𝑛𝑖𝑇 )
𝑚𝑖𝑇 ~𝑁(𝜆𝑖𝑇 , 𝑠𝑑𝑖𝑇
𝜆𝑖,𝑃 = 𝑢𝑖
𝜆𝑖,𝑇 = 𝑢𝑖 + 𝜃𝑖,𝑇𝑃
2 ).
𝜃𝑖,𝑇𝑃 ~𝑁(𝜇 𝑇𝑃 , 𝜏 𝑇𝑃
where 𝑢𝑖 is the mean of placebo from the baseline assumed to be normally distributed
𝑢𝑖 ~𝑁(𝑚𝑢 , 𝜎𝑢2 )
We set the following prior distributions
𝜇 𝑇𝑃 ~𝑁(0,10000)
𝑚𝑢 ~𝑁(0,10000)
𝜏 𝑇𝑃 ~𝑁(0,1), 𝜏 ≥ 0
3
𝜎𝑢 ~𝑁(0,1), 𝜏 ≥ 0
For standardised mean difference (SMD) effect measure we use the same model, where
SMD is obtained as
𝑦𝑖,𝑇𝑃 =
𝑚𝑖,𝑇 − 𝑚𝑖,𝑃
𝑠𝑑𝑖𝑝𝑜𝑜𝑙𝑒𝑑
∙ 𝐽𝑖
2 +(𝑛 −1)∙𝑠𝑑2
(𝑛𝑖,𝑃 −1)∙𝑠𝑑𝑖,𝑃
𝑖,𝑇
𝑖,𝑇
with pooled standard deviation 𝑠𝑑𝑖𝑝𝑜𝑜𝑙𝑒𝑑 = √
𝑛𝑖,𝑃 +𝑛𝑖,𝑇 −2
and 𝐽𝑖 a correction factor4
for the overestimation of the real difference due to small sample sizes 𝐽𝑖 = 1 − 4(𝑛
3
.
𝑖,𝑃 +𝑛𝑖,𝑇 −2)−1)
 Simple random effects meta-regression
Simple RE meta-regression ‘contrast-based’ model
We extend the previous model to include a study-level covariate 𝑥𝑖,𝑇𝑃 that represents
initial severity as
𝑦𝑖,𝑇𝑃 = 𝛽𝑥𝑖,𝑇𝑃 + 𝛿𝑖,𝑇𝑃 + 𝜀𝑖,𝑇𝑃
𝜀𝑖,𝑇𝑃 ~𝛮(0, 𝑣𝑖,𝑇𝑃 ),
(1)
2
𝛿𝑖,𝑇𝑃 ~𝑁(0, 𝜏 𝑇𝑃
)
2
We estimate the between-study heterogeneity 𝜏 𝑇𝑃
using the DL2 method. We fitted simple metaregression in STATA using metareg5 command. In the case where we model two covariates
(initial severity 𝑥𝑖 and publication year 𝑧𝑖 ) formula (1) becomes
𝑦𝑖,𝑇𝑃 = 𝛽𝑥𝑖,𝑇𝑃 + 𝛾𝑧𝑖,𝑇𝑃 + 𝛿𝑖,𝑇𝑃 + 𝜀𝑖,𝑇𝑃 .
Simple RE meta-regression ‘arm-based’ model
Equivalently, we extend the simple REMA hierarchical model as
𝑦𝑖,𝑇𝑃 ~𝑁(𝜃𝑖,𝑇𝑃 , 𝑣𝑖,𝑇𝑃 )
2
𝜃𝑖,𝑇𝑃 ~𝛮(𝛽𝑥𝑖,𝑇𝑃 , 𝜏 𝑇𝑃
)
In the Bayesian setting the exact hierarchical model used is the following
𝜆𝑖,𝑃 = 𝑢𝑖
∗
𝜆𝑖,𝑇 = 𝑢𝑖 + 𝜃𝑖,𝑇𝑃
∗
𝜃𝑖,𝑇𝑃
= 𝜃𝑖,𝑇𝑃 + 𝛽𝑥𝑖,𝑇𝑃
2 )
𝜃𝑖,𝑇𝑃 ~𝑁(𝜇 𝑇𝑃 , 𝜏 𝑇𝑃
4
We prefer though to centre the initial severity values around their mean, that is we
subtract the mean initial severity (𝑥̅ 𝑇𝑃 ) from each trial-specific covariate (𝑥𝑖,𝑇𝑃 ), so as to improve
the efficiency of the model estimation (‘correction’ for the ‘regression to the mean’ artefact):
∗
𝜃𝑖,𝑇𝑃
= 𝜃𝑖,𝑇𝑃 + 𝛽(𝑥𝑖,𝑇𝑃 − 𝑥̅ 𝑇𝑃 )
The parameters 𝜇 𝑇𝑃 and 𝑢𝑖 are given independent non-informative priors, whereas we set
a weakly informative prior for 𝜏 as described previously. A vague prior is also assigned to 𝛽:
𝛽~𝑁(0,10000)
 Network random-effects meta-analysis (NMA) for the star-shaped network
NMA ‘contrast-based’ model
Network meta-analysis can be viewed as a special case of multivariate meta-analysis.
Consider for example a simple star-shaped network of evidence including three treatments
𝐴, 𝐵, 𝐶, and assume there are studies comparing 𝐴 versus 𝐵 and 𝐴 versus 𝐶 treatments, with
common comparator 𝐴. Denoting by 𝑦𝑖,𝐴𝐵 and 𝑦,𝑖𝐴𝐶 the observed effect measures (e.g. RMD) 𝐴
versus 𝐵 and 𝐴 versus 𝐶, respectively, then each observed treatment effect is sampled from a
normal distribution as
𝑦𝑖,𝐴𝐵
𝜀𝑖,𝐴𝐵
𝜇𝐴𝐵
𝛿𝑖,𝐴𝐵
(𝑦 ) = (𝜇 ) + (
) + (𝜀 )
𝛿𝑖,𝐴𝐶
𝑖,𝐴𝐶
𝑖,𝐴𝐶
𝐴𝐶
𝛿𝑖,𝐴𝐵
𝜏2
𝜏 2 /2
0
(
) ~𝑁 (( ) , ( 2
))
𝛿𝑖,𝐴𝐶
0
𝜏 /2
𝜏2
𝜀𝑖,𝐴𝐵
𝑣𝑖,𝐴𝐵
0
(𝜀 ) ~𝑁 (( ) , (
0
𝑖,𝐴𝐶
0
0
𝑣𝑖,𝐴𝐶
))
Then under the consistency assumption the estimated pooled effect size of treatment 𝐵 versus
treatment 𝐶 is derived as
𝜇𝐵𝐶 = 𝜇𝐴𝐶 − 𝜇𝐴𝐵
The model can be easily extended to more than three treatments. For further details see
White et al6.
It should be noted that all comparisons in the network share a common 𝜏 2 , which allows
comparisons to ‘borrow strength’ from each other. In the frequentist setting we employ the
model in STATA using the mvmeta command7 and we estimate a fixed 𝜏 2 using the restricted
maximum likelihood (REML) estimator8. We also the probability that a treatment is the best
(P(best)) for all antidepressants versus placebo comparisons7.
5
NMA ‘arm-based’ model
Consider the previous simple star-shaped network of evidence. The observed treatment
effect measures 𝑦𝑖,𝐴𝐵 and 𝑦,𝑖𝐴𝐶 are sampled from a normal distribution as
𝑦𝑖,𝐴𝐵 ~𝑁(𝜃𝑖,𝐴𝐵 , 𝑣𝑖,𝐴𝐵 ),
𝑦𝑖,𝐴𝐶 ~𝑁(𝜃𝑖,𝐴𝐶 , 𝑣𝑖,𝐴𝐶 )
and similarly for the random effects
𝜃𝑖,𝐴𝐵 ~𝑁(𝜇𝐴𝐵 , 𝜏 2 ),
𝜃𝑖,𝐴𝐶 ~𝑁(𝜇𝐴𝐶 , 𝜏 2 ).
Under the consistency assumption 𝜇𝐵𝐶 = 𝜇𝐴𝐶 − 𝜇𝐴𝐵 .
The idea is extended to more than three treatments, where for any two treatments j, k =
{A, B, C, D, E} compared in study 𝑖 the model for a specific comparison j versus k can be written
as
𝑦𝑖,𝑗𝑘 ~𝑁(𝜃𝑖,𝑗𝑘 , 𝑣𝑖,𝑗𝑘 )
𝜃𝑖,𝑗𝑘 ~𝑁(𝜇𝑗𝑘 , 𝜏 2 )
Setting A the reference treatment and assuming consistency the means of the randomeffects distributions are obtained as
𝜇𝑗𝑘 = 𝜇𝐴𝑘 − 𝜇𝐴𝑗
Note that all comparisons in the model share the same amount of heterogeneity. In the
Bayesian framework τ2 is a random variable given a weakly informative prior distribution. We
use the same prior distributions for μjk , 𝑢𝑖 and τ parameters as in simple REMA model. We also
produced treatment ranking across all antidepressants versus placebo comparisons by estimating
the surface under the cumulative ranking (SUCRA)9.
 NMA random-effects meta-regression
NMA RE meta-regression ‘arm-based’ model
Extending the NMA hierarchical model to include a study-level covariate xi,𝑗𝑘 that
represents initial severity we use the following hierarchical model in a Bayesian setting
𝜆𝑖,𝑗 = 𝑢𝑖
∗
𝜆𝑖,𝑘 = 𝑢𝑖 + 𝜃𝑖,𝑗𝑘
∗
𝜃𝑖,𝑗𝑘
= 𝜃𝑖,𝑗𝑘 + 𝛽(𝑥𝑖,𝑗𝑘 − 𝑥̅𝑗𝑘 )
6
𝜃𝑖,𝑗𝑘 ~𝑁(𝜇𝑗𝑘 , 𝜏 2 )
𝜇𝑗𝑘 = 𝜇𝐴𝑘 − 𝜇𝐴𝑗
We set the same prior distributions for 𝜇𝑗𝑘 , 𝑢𝑖 , τ and 𝛽 as previously.
Prior Distributions for 𝝉 in NMA RE meta-regression
It has been shown that the choice of prior distribution is crucial, especially when few studies are
included in the dataset10. We therefore employ 12 different priors in the NMA meta-regression
model so as to evaluate any differences in the results. The following table shows the prior
distributions we have set for the heterogeneity in the Bayesian model.
Table 1. Prior distributions for the heterogeneity.
Prior 1
Prior 2
Prior 3
Prior 4
Prior 5
Prior 6
Prior 7
Prior 8
Prior 9
Prior 10
Prior 11
Prior 12
1⁄𝜏 2 ~𝑃𝑎𝑟𝑒𝑡𝑜(1,0.001)
1⁄𝜏 2 ~𝑃𝑎𝑟𝑒𝑡𝑜(1,0.25)
1⁄𝜏 2 ~𝐺𝑎𝑚𝑚𝑎(0.01,0.01)
1⁄𝜏 2 ~𝐺𝑎𝑚𝑚𝑎(0.1,0.1)
𝜏~𝑈𝑛𝑖𝑓𝑜𝑟𝑚(0,100)
𝜏~𝑈𝑛𝑖𝑓𝑜𝑟𝑚(0,2)
𝜏~𝑁(0,100), 𝜏 > 0
𝜏~𝑁(0,1), 𝜏 > 0
𝜏 2 ~𝑈𝑛𝑖𝑓𝑜𝑟𝑚(0,1000)
𝜏 2 ~𝑈𝑛𝑖𝑓𝑜𝑟𝑚(0,4)
𝑙𝑜𝑔(𝜏 2 )~𝑈𝑛𝑖𝑓𝑜𝑟𝑚(−10,10)
𝑙𝑜𝑔(𝜏 2 )~𝑈𝑛𝑖𝑓𝑜𝑟𝑚(−10,1.386)
7
3. ADVANTAGES AND LIMITATIONS
a. Simple random effects meta-analysis
Increases power and precision. Quantifies the treatments’ effectiveness and its uncertainty.
Quantifies between-study heterogeneity. The validity depends on the quality of trials4;11.
b. Network random effects meta-analysis
Extension of Simple meta-analysis. Provides more powerful results by incorporating all evidence
in the network12;13. Insights are provided when pairwise meta-analysis is not available. More
specifically, in the case of a star-shaped network informed by AB, AC, AD comparisons, NMA
uses all available study data to infer about the relative effectiveness of BC, BD, CD.
c. Simple random effects meta-regression
The effects of multiple factors are investigated. We test whether there is a linear relationship
between treatment effect and a covariate that differs across studies (e.g. initial severity)14.
However, we should be aware of false-positive findings, i.e. finding a statistically significant
result when there is no relationship in reality.
Comparing random-effects meta-analysis with random-effects meta-regression we determine
how much heterogeneity is explained by the covariate. However, there is always the risk of
confounding, i.e. a known or unknown covariate to be associated both with the covariate of
interest and the treatment effect. We should be careful when investigating the relationship
between treatment-effects and initial severity as they are inherently correlated. The Bayesian
approach provides more reliable inferences than the frequentist one for this association by using
an uninformative prior distribution for heterogeneity15;16. This method has low power to detect
any relationship when the number of studies is small. There is also a potential for biases (e.g.
aggregation bias)17.
d. Network random effects meta-regression
Extension of simple meta-regression analysis. The same characteristics as in NMA analysis. A
difference in the results on these two models can be due to the adjustment of the covariate18.
e. Bayesian approach in general
Especially useful for small meta-analyses. Can assess robustness by using different priors. This
method accounts for full uncertainty. However, the results depend on priors when few trials are
available. It is possible to estimate the uncertainty of the heterogeneity in contrast to the
frequentist approach. In the frequentist approach the heterogeneity parameter is assumed a
known constant value, but in a Bayesian setting we set a prior distribution which allows us to
infer about its (posterior) distribution. The heterogeneity uncertainty is always introduced in the
results. When few studies are available the Bayesian estimation of heterogeneity may be
problematic due to the choice of the prior distribution10. It is possible experts’ opinion to be
introduced in the model.
f. Frequentist approach in general
Doesn’t estimate uncertainty for the heterogeneity. Difficult to estimate heterogeneity with few
trials.
8
4. REFERENCES
(1) Lunn DJ, Thomas A, Best N, Spiegelhalter D. WinBUGS - a Bayesian modelling
framework: concepts, structure, and extensibility. Statistics and Computing 2000;325337.
(2) DerSimonian R, Laird N. Meta-analysis in clinical trials. Control Clin Trials 1986;7:177188.
(3) Harris R, Bradburn M, Deeks J, Harbord R, Altman D, Sterne J. metan: fixed- and
random-effects meta-analysis. Stata Journal 2008;8:3-28.
(4) Borenstein M, Hedges LV, Higgins JPT, Rothstein HR. Introduction to Meta-analysis .
1st edition ed. Chichester, UK: John Wiley&Sons, 2009.
(5) Harbord R M, Higgins J P T. Meta-regression in Stata. Stata Journal 2008;8:493-519.
(6) White IR, Barret JK, Jackson D, Higgins JPT. Consistency and inconsistency in multiple
treatments meta-analysis: model estimation using multivariate meta-regression. Research
Synthesis Methods 2012;3:111-125.
(7) White IR. Multivariate random-effects meta-regression: Updates to mvmeta. Stata
Journal 2011;11:255-270.
(8) Raudenbush S.W. Analyzing Effect Sizes: Random Effects Models. In: Cooper H.,
Hedges LV, Valentine J.C., eds. The Handbook of Research Synthesis and MetaAnalysis. 2nd edition ed. Russell Sage Foundation, New York; 2009;295-315.
(9) Salanti G, Ades AE, Ioannidis JP. Graphical methods and numerical summaries for
presenting results from multiple-treatment meta-analysis: an overview and tutorial. J Clin
Epidemiol 2011;64:163-171.
(10) Lambert PC, Sutton AJ, Burton PR, Abrams KR, Jones DR. How vague is vague? A
simulation study of the impact of the use of vague prior distributions in MCMC using
WinBUGS. Stat Med 2005;24:2401-2428.
(11) Higgins J., Green S. Cochrane Handbook for Systematic Reviews of Interventions. 2011.
(12) Salanti G, Higgins JP, Ades AE, Ioannidis JP. Evaluation of networks of randomized
trials. Stat Methods Med Res 2008;17:279-301.
(13) Salanti G. Indirect and mixed-treatment comparison, network, or multiple-treatments
meta-analysis: many names, many benefits, many concerns for the next generation
evidence synthesis tool. Research Synthesis Methods 2012;3:80-97.
9
(14) Salanti G, Marinho V, Higgins JP. A case study of multiple-treatments meta-analysis
demonstrates that covariates should be considered. J Clin Epidemiol 2009;62:857-864.
(15) Sutton AJ, Abrams KR. Bayesian methods in meta-analysis and evidence synthesis. Stat
Methods Med Res 2001;10:277-303.
(16) Thompson SG, Smith TC, Sharp SJ. Investigating underlying risk as a source of
heterogeneity in meta-analysis. Stat Med 1997;16:2741-2758.
(17) Petkova E, Tarpey T, Huang L, Deng L. Interpreting meta-regression: application to
recent controversies in antidepressants' efficacy. Stat Med 2013.
(18) Salanti G, Dias S, Welton NJ et al. Evaluating novel agent effects in multiple-treatments
meta-regression. Stat Med 2010;29:2369-2383.
10