Application of Evidence Synthesis:
Effectiveness of immunosuppressive
therapies in renal transplantation
Jim Chilcott
Mike Holmes
Work undertaken as part of a submission to NICE on behalf of Novartis UK
Immunosuppression in Renal Transplantation
•
One of the most serious problems facing renal transplant patients
is possibility that their own bodies try to reject or destroy the
transplant.
•
One of the best predictors of long term graft loss is the incidence
of acute rejection - the most commonly defined outcome measure
in renal transplant trials.
•
To avoid acute rejection patients regularly take a cocktail of
immunosuppressive drugs usually for the rest of their lives.
NICE are currently in the process of attempting to construct guidance in
this field and a key element of this appraisal is the assessment of
evidence on the impact of drug regimens on acute rejection.
Comparators
•
Immunosuppressive regimens can consist of:
An induction drug used at the time of transplantation 3 (4)
Basiliximab / Daclizumab / None ( /ATG)
Base maintenance drug 6
Sandimmun / 2x Neoral / Tacrolimus / 2x Sirolimus
Adjunctive drug 3 (4)
Azathiopine / 2x MMF / None
Giving 54 (96) different regimens - excluding dosing strategies etc
•
For the purposes of this assessment this was reduced to a set of
16 different regimens, based upon common usage, licensing etc.
Problems with the Evidence Base
Direct evidence base consists of a set of RCTs with two / three
comparators. Full complement of problems that may beset an
experimental evidence base:
•
•
•
•
Multiple pieces of evidence on individual comparisons.
Different types of outcome.
Different measures of effect.
Biases to internal validity – For example, conflicting biases from under
and over drug dosing between arms in the trials.
•
Biases to external validity – For example, consistent under or over drug
dosing compared to common practice.
•
Indirect evidence – Whilst acute rejection is in itself of disbenefit to a
patient, the primary motivation in preventing acute rejection is in reducing
potential associated long term graft loss. Thus virtually the entire
randomised evidence is indirect.
•
•
Mixed comparisons – As discussed above.
Gaps in the experimental evidence.
The AR evidence base
Sandimmune+
azathioprine
Simulect+Neoral
+ azathioprine
Sandimmune
II
I
III
Daclizumab+Neoral
+azathioprine
I
Simulect+Neoral
+MMF
I
I
Neoral+
azathioprine
Neoral+
MMF
II
I
II
Sandimmune+
MMF
II
Tacrolimus+
azathioprine
I
II
Simulect+
Neoral
I
Tacrolimus+
MMF
Neoral
I
I
Daclizumab+
Neoral
II
Neoral C2
monitoring
Sirolimus+Neoral
(<3 months)
Sirolimus+Neoral
(continued)
Synthesis of evidence on AR impact
Two options for synthesising evidence
•
Bayesian synthesis of evidence
Uses all the evidence in a coherent framework to
estimate both relative and absolute treatment effects
•
Only alternative: series of standard pair-wise metaanalyses to estimate relative treatment effects bolted
onto naïve average of baseline effects.
Bayesian models of treatment effectiveness
Two models of treatment effectiveness have been explored:
•
Treatment combination model
Each regimen, defined by an induction, base maintenance and adjunct
drug is considered as a separate therapy in estimating effectiveness.
•
Individual drug effect model
Effectiveness of a regimen is assumed to be built up from the
effectiveness of its constituent drugs. Has the potential advantage of
making a fuller use of the available evidence, however assumes no
interactions or synergistic effects between drugs.
Both models assume a fixed treatment effect with a random baseline effect
The treatment combination model
Let there be n
m

trials, subscripted i,
arms subscripted j,
treatment combinations, subscripted k,
Assume each trial baseline rejection rate, logit(mu(i)), (i.e. rejection rate
under baseline therapy, in this case Sandimmun®(ciclosporin)) varies
randomly and that mu(i) is normally distributed with mean m and standard
deviation .
i.e.
mu(i) ~ N(m,2)
Assume a fixed treatment effect, dk for treatment combination k.
Let pi,j be the estimated probability of acute rejection in study i, arm j, under
treatment k, then:
logit(pi,j) = mu(i) + dk
and
pi,j = exp (mu(i) + dk) / (1+ exp (mu(i) + dk)) .
Similarly where pk is the probability of acute rejection for an ‘average’
transplant population under treatment k, then:
pk = exp (m + dk) / (1+ exp (m + dk)) .
The individual drug effect model
Assume that a combination maintenance treatment is defined as a baseline
drug, together with an optional induction therapy and optional adjunct agent.
Let there be n trials, subscripted i,
m arms subscripted j,
ind induction therapy drugs, subscripted kind,
base baseline therapy drugs, subscripted kbase,
adj adjunct therapy drugs, subscripted kadj,
Assume each trial baseline rejection rate, logit(mu(i)), varies randomly and:
mu(i) ~ N(m,2)
Assume for each drug k a fixed treatment effect, dk .
Let pi,j be the estimated probability of acute rejection in study i, arm j, under
treatment kind+kbase+kadj, then:
and
.
logit(pi,j) = mu(i) + dkind + dkbase + dkadj
pi,j = exp (mu(i)+dkind +dkbase +dkadj) / (1+ exp (mu(i) +dkind +dkbase +dkadj))
Similarly for an ‘average’ transplant population under treatment k defined by
kind, kbase and kadj then:
pk = exp (m +dkind +dkbase +dkadj) / (1+ exp(m +dkind +dkbase +dkadj))
The Markov Chain Monte Carlo (MCMC)
Technique
•
WinBUGS (Vs1.3) has been used to estimate posterior
distributions for baseline and treatment effect parameters from
vague priors.
•
Convergence has been assessed using Gelman-Rubin statistic.
•
First 5 thousand ‘burn in ‘ simulation discarded. Adequate
convergence achieved within 20,000 iterations, i.e. G-R statistic
converged to 1.00 .
Prop of AR
steroid
Treatment
Study
n
r
responsive Actual
Bugs
Lower CI Upper CI Deviance Rank
Sandimmune+Aza
Niese (12 months)
41
23
56.1%
46.0%
39%
54%
1.94
Neoral+Aza
Niese (12 months)
45
19
42.2%
38.8%
32%
47%
0.51
Sandimmune+Aza
Abendroth, Buchholz (24 months)
27
15
55.6%
44.3%
37%
53%
1.56
Neoral+Aza
Abendroth, Buchholz (24 months)
28
10
35.7%
37.1%
31%
45%
0.19
Sandimmune+Aza
Pollard (12 months)
98
46
87.8
46.9%
43.0%
37%
49%
0.95
Neoral+Aza
Pollard (12 months)
195
67
89.7
34.4%
35.9%
31%
41%
0.72
Sandimmune+Aza
Miller (12 months)
207
96
74.9
46.4%
46.6%
41%
52%
0.63
Tacrolimus + Aza
Miller (12 months)
205
63
89.3
30.7%
26.7%
23%
31%
2.22
Sandimmune+Aza
Mayer ()
145
63
43.4%
43.1%
38%
48%
0.43
Tacrolimus + Aza
Mayer ()
303
73
24.1%
24.0%
20%
28%
0.57
Neoral+Aza
Morris-Stiff (6 months)
89
35
39.3%
38.7%
33%
46%
0.42
Tacrolimus + Aza
Morris-Stiff (6 months)
90
29
32.2%
26.2%
21%
32%
1.98
Neoral+Aza
Margreiter (6 months)
271
101
0.46
37.3%
34.6%
30%
39%
1.44
Tacrolimus + Aza
Margreiter (6 months)
286
56
0.46
19.6%
23.0%
19%
27%
2.42
Tacrolimus + Aza
Johnson (2000) (12 months)
76
13
0.31
17.1%
22.7%
17%
28%
1.71
Neoral + MMF (2mg/day)
Johnson (2000) (12 months)
75
15
0.47
20.0%
20.3%
15%
27%
0.46
Tacrolimus + MMF (2mg/day) Johnson (2000) (12 months)
72
11
0.73
15.3%
12.5%
7%
20%
1.43
Neoral + MMF (2mg/day)
Wang (12 month mean)
32
5
0.2
15.6%
19.8%
13%
27%
0.59
Tacrolimus + MMF (2mg/day) Wang (12 month mean)
25
1
0
4.0%
12.1%
6%
20%
2.17
Neoral + MMF (2mg/day)
Kreis (2 years)
38
7
94.7
18.4%
20.9%
14%
29%
0.42
Neoral
McDonald (6 months)
130
54
91.5
41.5%
45.3%
39%
51%
1.24
Sirolimus (2mg/day) + Neoral McDonald (6 months)
227
56
96
24.7%
20.7%
17%
25%
2.93
Sirolimus (N+S<3mnths)
Johnson (2001) (12 months)
215
43
20.0%
19.2%
15%
24%
0.87
Sirolimus (2mg/day) + Neoral Johnson (2001) (12 months)
215
29
13.5%
17.1%
13%
21%
2.69
Sirolimus (N+S<3mnths)
Gonwa (6 months)
92
17
18.5%
20.3%
15%
26%
0.63
Sirolimus (2mg/day) + Neoral Gonwa (6 months)
89
15
16.9%
18.1%
14%
23%
0.43
Sandimmune
European MMF Cooperative Study Group (6 months)
166
77
46.4%
46.9%
39%
54%
0.97
Sandimmune + MMF (2mg)
European MMF Cooperative Study Group (6 months)
165
28
17.0%
18.8%
15%
23%
0.90
Sandimmune+Aza
Sollinger (6 months)
164
63
38.4%
41.0%
35%
46%
1.00
Sandimmune + MMF (2mg)
Sollinger (6 months)
165
33
20.0%
19.3%
15%
24%
0.57
Sandimmune+Aza
Tricontinental MMF Renal Transplantation Study
164Group (6 59
months)
36.0%
40.1%
34%
46%
1.73
Sandimmune + MMF (2mg)
Tricontinental MMF Renal Transplantation Study
171Group (6 34
months)
19.9%
18.7%
15%
23%
0.68
Neoral
Nashan (1997) (6 months)
186
73
0.41
39.2%
40.7%
35%
46%
0.79
Basiliximab+Neoral
Nashan (1997) (6 months)
190
51
0.63
26.8%
28.3%
23%
34%
0.90
Neoral+Aza
Ponticelli (6 months)
172
50
0.66
29.1%
32.5%
27%
38%
1.56
Basiliximab+Neoral+aza
Ponticelli (6 months)
168
31
0.71
18.5%
18.5%
13%
25%
1.03
Neoral
Kahan (12 months)
173
85
0.40
49.1%
46.8%
41%
53%
0.97
Basiliximab+Neoral
Kahan (12 months)
173
61
0.44
35.3%
33.6%
28%
40%
0.95
Neoral
Charpentier (6 months)
134
64
0.67
47.8%
45.2%
39%
52%
0.90
Daclizumab+Neoral
Charpentier (6 months)
141
39
0.72
27.7%
27.6%
21%
35%
0.99
Neoral+Aza
Vincenti (12 months)
134
47
0.60
35.1%
35.3%
30%
41%
0.50
Daclizumab+Neoral+aza
Vincenti (12 months)
126
28
0.64
22.2%
22.2%
16%
30%
0.99
Neoral + MMF (2mg/day)
Lawen (6 months)
64
17
0.71
26.6%
22.8%
16%
31%
1.03
Basiliximab+Neoral+MMF
Lawen (6 months)
59
9
0.67
15.3%
13.2%
7%
21%
1.01
Basiliximab+Neoral+MMF
Lebranchu (6 months)
36
3
8.3%
11.9%
6%
20%
0.79
Neoral C2 monitoring
Mo2art (3 months)
117
12
10.3%
10.2%
5%
16%
1.00
7
39
11
46
24
32
34
4
43
38
45
6
12
3
9
41
13
36
5
44
14
1
29
2
35
42
22
26
18
37
8
33
30
28
10
16
23
25
26
21
40
20
15
17
31
19
LeadAuthor
Niese
Niese
Abendroth, Buchholz
Abendroth, Buchholz
Pollard
Pollard
Miller
Miller
Mayer
Mayer
Morris-Stiff
Morris-Stiff
Strategy
Sandimmun+Aza
Neoral+Aza
Sandimmun+Aza
Neoral+Aza
Sandimmun+Aza
Neoral+Aza
Sandimmun+Aza
Tacrolimus + Aza
Sandimmun+Aza
Tacrolimus + Aza
Neoral+Aza
Tacrolimus + Aza
n
41
45
27
28
98
195
207
205
145
303
89
90
r
Actual
23 56.1%
19 42.2%
15 55.6%
10 35.7%
46 46.9%
67 34.4%
96 46.4%
63 30.7%
63 43.4%
73 24.1%
35 39.3%
29 32.2%
Bugs
estimate Lower CI Upper CI Deviance Rank
45.6%
39%
54%
2.06
5
38.2%
32%
46%
0.56
39
44.1%
37%
52%
1.58
9
36.8%
31%
44%
0.16
46
43.1%
38%
49%
0.91
29
35.8%
31%
41%
0.66
34
46.2%
41%
52%
0.61
37
26.5%
22%
31%
2.41
2
43.1%
38%
48%
0.40
44
24.1%
21%
28%
0.55
40
38.2%
33%
45%
0.43
43
26.0%
22%
32%
2.08
4
Model consistency checking
Using the combination treatment effect model the individual trial arms
were ranked in descending order of deviance contribution. All trials
with an arm with a deviance contribution greater than 1.8 were
examined to identify possible causes of major bias. Amendments and
exclusions were applied as a result of this consistency checking.
•
The Barone study comparing Sandimmun and Neoral®(ciclosporin microemulsion).6 DC of 3.43 in the
Sandimmun arm. Statistically significant difference in the baseline serum creatinine levels, Sandimmun
3.4 mg/dl vs Neoral 4.4 mg/dl (p<0.05). The serum creatinine levels in the Sandimmun arm equate to
approximately 300 mmols/l, in the other Neoral/Sandimmun studies identified in this review levels
baseline serum creatinine levels were all in the order of 600 – 700 mmols/l,1,27 and no other studies
reported statistically significant baseline levels, thus this study was excluded.
Model consistency checking II
•
The Trompeter study32 had a DC of 2.21 in the Neoral arm. This study was excluded since it is
the only paediatric study within the review with implications for both the underlying population
and treatment protocols.
•
Initially the interim results for Morris-Stiff
26
were included in the assessment, however this study
gave rise to deviance contributions of 3.95 in the Tacrolimus arm and 2.07 in the Neoral arm.
These results however were for 3 months follow up and only included the first 80 patients.
Subsequent results have been reported for 179 patients with 6 months follow up,
25
though these
results are only recorded in a letter their inclusion is justified on the basis of the previously
reported and reviewed study design. These 6 month results have been used in the final
assessment (DCs Tac 2.08, Neoral 0.43)
•
In the initial analysis the long term Sirolimus and Neoral combination arms of McDonald22 and
Johnson
21
had deviance contributions of 1.91 and 1.86 respectively. On closer examination the
Johnson study randomises at 3 months post transplant and reports post 3 acute rejection rates.
Thus for this assessment the Johnson figures have been adjusted by adding the initial 3 month
acute rejection rate onto both arms. This adjustment leads to deviance contributions of 1.53 and
0.87 compared to the above.
Model consistency checking III
Of the five tacrolimus plus azathioprine studies, four studies have deviances greater than 1.8, in
fact this subset of studies include the arms ranked 1,2,4 and 7 in terms of their deviance
contributions, this indicates that there is a great deal of inconsistency within tacrolimus evidence
base, this concurs with the results of the systematic review of tacrolimus. An area for further
research might to explore the impact of internal and external biases within these trials including
potentially the date of the trials, impact of blinding and the impact of Neoral and tacrolimus doses
within the trials.
Pre-synthesis systematic review identified:
•Open label studies
•Exclusion of high risk patients unclear whether before or after randomisation (ie 2 weeks after Tx)
•Differential dosing between arms – 1 arm lower than licensing recommendations
Model Fit Comparison
Model
Data points
Parameters
Posterior mean deviance
Deviance Information Critirion
Treatment Drug
model
model
46
46
16
10
49.5
60.48
273.6
284.6
Sa San
nd di
im m m
m
un une
e+
Az
a
Sa
nd T Ne Ne
im ac
or ora
r
Ne mu olim al+ l
Ta o ne
u Az
cr ral
+ s+ a
ol
im + M MM Az
u
a
M F
Si s + F ( (2m
ro M
2
lim M mg g)
us F ( /da
(N 2m y)
B
Ba as +S< g/da
il
y
s
3
Ba ilix ixim mn )
sil ima a b ths
ix
im b+N +N e )
ab e
or or a
+
l
N
a
Da Da c eo l+a
Si
ro cliz lizu r al+ za
lim u
m
M
us ma a b MF
(2 b+N +N
m
eo eor
g
Ne /d
r a al
or ay) l+a
al
+
z
C2 N a
e
m or
on al
ito
r in
g
Synthesis of evidence on acute
rejection rates by treatment
70%
60%
50%
40%
30%
20%
10%
0%
av - AR
av - LCI
av - UCI
AR Rates
Treatment
av - AR
av - LCI
av - UCI
Sandimmune
49%
39%
59%
Sandimmune+Aza
43%
38%
47%
Neoral
43%
37%
49%
Neoral+Aza
36%
32%
40%
Tacrolimus + Aza
24%
20%
27%
Sandimmune + MMF (2mg)
20%
16%
25%
Neoral + MMF (2mg/day)
21%
16%
28%
Tacrolimus + MMF (2mg/day)
13%
7%
21%
Sirolimus (N+S<3mnths)
22%
16%
29%
Basiliximab+Neoral
30%
24%
37%
Basiliximab+Neoral+aza
21%
14%
29%
Basiliximab+Neoral+MMF
12%
6%
20%
Daclizumab+Neoral
26%
18%
35%
Daclizumab+Neoral+aza
22%
15%
31%
Sirolimus (2mg/day) + Neoral
19%
15%
24%
Neoral C2 monitoring
10%
5%
18%
Limitations and provisos
A number of possible alternatives in the model formulation exist:
•
Use of a random effects model rather than fixed effects
•
Remains a high degree of inconsistency within the model – potential
to explore other factors for example:
•
•
Date of trial
•
Blinded vs Open label RCTs
•
Baseline serum creatinine levels
•
Drug doses
•
Patient characteristics – eg HLA matching, previous transplants.
Combination treatment effect model evidence base contains 2
unlinked networks, comparisons between networks have low validity.
Bayesian synthesis vs Pairwise meta-analysis
AR rates within 1
year
Basic Triple Therapy
CAS
Comparators
TAS
CAS + BAS
CAS + DAC
CMS
Pairs
Pairs
Pairs
Pairs
38.9
34
32.8
40.8
20
20.2
25.2
19.4
Novartis/
Pairs
Winbugs
34
Mean
(excl.
Winbugs)
36
24
20
22
21
Clinical and cost effectiveness of immunosuppressive regimens in renal transplantation. West Midlands
Health Technology Assessment Group, Department of Public Health and Epidemiology, The University of
Birmingham. Dec 2002.
Critique of Bayesian synthesis I
The first criticism is that the “requirement to consider individually the
relevance and/or quality of each study for the specific question under
consideration, and hence attach a degree of uncertainty to the
‘evidence’ from each study” is not fulfilled.” #
True but equally applicable to standard meta-analysis
Bayesian synthesis highlights some inconsistencies and facilitates
incorporation of explanatory factors and quantified bias adjustment.
# Clinical and cost effectiveness of immunosuppressive regimens in renal transplantation. West Midlands
Health Technology Assessment Group, Department of Public Health and Epidemiology, The University of
Birmingham. Dec 2002.
Critique of Bayesian synthesis II
“Synthesis falls into the ‘trap’ of overdependence on absolute arm
results… Rejection rates for an ‘average’ population given a treatment
are very close to the average of the observed rates over the trials –
which suggests that (all the mi are similar in size) it is the ‘treatment
effects’ that are having to model all variation due both to treatment and
to inter-trial variation in risk.” #
Impact of consistent and differential bias between and especially within
studies presents an equal problem to standard meta-analysis.
Differential bias in the treatment arms presents itself in the Bayesian
synthesis as inconsistency which in this particular example ties in with
poor quality ratings for specific trials.
Conclusions for Renal Transplantation
•
Combination treatment model provides best fit to data. Indicates
possibly important interaction between the drugs when used in
combinations, supports clinical or pharmacological opinion.
•
Though baseline results provide rank ordering of effectiveness of
combination therapies, CIs for many regimens have a large overlap
indicating a high degree of uncertainty.
•
Basiliximab and daclizumab have very similar effectiveness profiles
across the different drug combinations that have been investigated.
•
Subset of the evidence with the worst consistency is the evidence for
tacrolimus. The results for tacrolimus should therefore be viewed
with particular caution.
Key Challenges
•
Further research is required comparing standard meta-analyses with
Bayesian syntheses in complex decision problems.
•
Further development is required in protocols for model consistency
checking. Including diagnostic measures of consistency and
guidelines for implementation.
•
Methods for incorporating quantified estimates of bias: eg impact of
open label studies on estimates of effectiveness.
•
Methods for incorporating prognostic factors and non random
effects.