Ver 12 23 Mar 2015
1
2
Response to: Clinch B, Smith J, Kenwright A, Surujbally B, Harding J. Roche feedback on
“Neuraminidase inhibitors for preventing and treating influenza in healthy adults and children”
3
We have the following comments on Roche’s feedback
4
INTRODUCTION
5
6
7
8
We asked Roche on multiple occasions whether they would furnish IPD under the same conditions as those
granted to Professors Hernan & Lipsitch. Roche responded to our emails but never addressed this point
(bmj.com/tamiflu). Eventually Roche provided individual listings in printed pdf format which could not be used
electronically without reloading line by line.
9
10
11
We have never accepted anything with strings attached (such as the confidentiality agreement with a
secrecy clause Roche offered us in the Autumn of 2009). Data for public health drugs should be freely
accessible.
12
13
14
15
16
17
18
19
20
21
Roche criticizes our methods, but only after we completed our review. We never previously received
feedback from Roche despite repeated requests for comments on our published protocol (available online
since December 2010). Our requests for protocol comments were met with the answer that it was not for
Roche to comment, it was for the scientific community. In contrast, GSK provided protocol feedback. Roche
now says it did not provide feedback to us at the time because our protocol contained “limited details and
methods”. If this is the case, why did Roche not explain its views 5 years ago? Our review has gone through
many layers of peer review including protocol review, final review and review by BMJ for the related papers
published there. All peer review comments are available (see Peer review history of reviews on
neuraminidase inhibitors relevant to Cochrane A159
http://www.bmj.com/content/bmj/suppl/2014/04/09/bmj.g2545.DC1/jeft017746.ww8_default.pdf.
22
23
24
25
26
27
28
29
30
Roche claims our findings are “in contrast to … over 100 regulators worldwide.” If this is true, then the 100
regulators must not include the US FDA, as our conclusions are broadly consistent with those of the FDA.
For example, the FDA states: “Serious bacterial infections may begin with influenza-like symptoms or may
coexist with or occur as complications during the course of influenza. Tamiflu has not been shown to prevent
such complications”. If Cochrane reviews had to conform to the conclusions of regulators there would not be
any requirements for independent science. In addition, we have been unable to ascertain how many trials
regulators saw and which documents were used for the approval process. We have listed the trials that EMA
saw (only one trial’s clinical study report was complete) and we have shown that large important trials such
as M76001 were not reviewed in depth by FDA.
31
RISK OF BIAS ASSESSMENT (ROB)
32
33
34
35
Much is made by Roche about our frank admission as to the novelty of carrying out a systematic review on
CSRs. Could Roche point to an experienced body of CSR systematic reviewers? All recently published
summary studies sponsored by Roche are based on publications, which we have shown to be substantially
affected by reporting bias.
36
37
During the period 2010-2013 Roche refused to provide any details and refused to respond to our data
validation strategy queries of April 2011 (see the correspondence at bmj.com/tamiflu).
38
39
If our contention that CSRs should contain all key information about a clinical trial is wrong, could Roche
point us to the location of such information?
40
41
Roche accuses us of being excessively conservative in our ROB assessment but goes on to state that “All
analyses should be based on conservative assumptions”. Which one is right?
42
43
44
45
Roche make the common mistake of considering ROB as one of the study inclusion criteria in our review.
This is not so and never has been so. ROB assessments are taken into account when interpreting the results
of the review. ROB judgments in systematic reviews are by their very nature subjective. This is why we have
detailed our methods both in the review and in the follow-on paper.
Ver 12 23 Mar 2015
46
Some responses to specific criticisms are given below.
47
48
Criticism: Table 1. Key general issues and specific examples of issues with the Cochrane authors’
assessment of the risk of bias
49
50
51
52
53
54
55
56
Random sequence generation - this should not be low risk of bias based on the information provided
because we could not find any trial where the original randomisation list was presented in order, with details
provided for each randomised participant matched to their randomisation number (with unused
randomisation numbers left blank), and with the ability to cross reference this number to the rest of the CSR.
Without this information we cannot say if the sequence generation was truly random. Furthermore without
this information it is possible that study centers may have been dropped if for example oseltamivir effect on
antibodies led to a significant imbalance between treatment groups in the ITT infected subgroup. Does
Roche have the original randomisation list? If so, why was it not included in the CSR?
57
58
59
60
61
To further strengthen our argument from the previous paragraph we have recently discovered that there
appears to be an imbalance between treatment groups in the prophylaxis trials in terms of patients with high
creatinine levels at baseline as we describe in http://www.bmj.com/content/348/bmj.g2545/rr. Given this
imbalance is unlikely to have occurred by chance it implies high risk of bias for both random sequence
generation and allocation concealment for the prophylaxis trials.
62
63
64
65
66
67
Incomplete data for symptoms – the assessment of high risk of bias does not relate to censored data but
rather to missing self-reported symptoms data (the primary outcome of the trials) for individual patients who
were still in the study. The frequency of this missing data was not reported, however, imputation based on
interpolation was used to impute the missing information. If Roche could provide us the symptoms data
where we are able to distinguish between actual data and imputed data then we could reconsider our
classification.
68
69
70
71
72
73
Incomplete outcome data for complications - was apparent as events that appeared to be complications
were either reported as complications or adverse events or both complications and adverse events. If Roche
had provided complete complications data including detailed information on how they were defined,
diagnosed and reported then we would have taken this into account. Similarly incomplete outcome safety
data – clearly occurred in the treatment trials as events thought to be related to the disease were not
collected as adverse events unless they met the criteria for a serious adverse event.
74
75
BIAS, SELECTIVE REPORTING, INCONSISTENCIES AND FLAWS IN METHODS ANALYSING
EFFICACY DATA
76
Some responses to specific criticisms are given below.
77
78
79
80
81
Criticism: IPD was not used despite being available in the CSR
82
83
84
However we studied the possibility of extracting IPD for first alleviation of symptoms so that we can compare
oseltamivir treatment effects for this outcome in the influenza and non-influenza subgroups across the
treatment trials of adults. We did this with IPD extracted from CSRs for M76001 and WV15670.
85
86
Data was extracted for all 1447 enrolled patients of M76001 but a number of problems with the quality of the
data became apparent including:
87
88
89
90
1. 41 (3%) of patients did not have a time recorded (32 in oseltamivir group and 9 in placebo group)
implying they may have withdrawn from the study before any follow up data was recorded?
2. 33 (2%) of patients had alleviation at 0 hours (25 in oseltamivir group and 8 in placebo group) –
hence they did not have symptoms at baseline
We did not have the resources to extract the individual data from the 1000s of patients in the PDFs and then
enter it into an electronic database for analysis. If Roche were serious about wanting us to use IPD they
could have provided it to us as they did Hernan and Lipsitch.
Ver 12 23 Mar 2015
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
3. 249 (17%) did not have data on changes in antibody levels (proportions without data were similar in
the 2 treatment groups)
4. 393 (27%) did not have a culture test at baseline and 87% of those that did have culture test were
diagnosed with influenza
5. Results of culture test at day 3 were used to diagnose patients as influenza infected – this will
introduce a bias if oseltamivir has an effect on viral shedding (it would tend to lessen the chance of
positive diagnosis if oseltamivir reduces viral shedding).
6. 111 (8%) of patients were censored (proportions censored were similar in the 2 treatment groups)
7. Overall 75% of patients in placebo group were diagnosed with influenza compared to 73% in the
oseltamivir group (ITTI). However if we assume the patients culture tested at day 1 is a random
sample of all enrolled patients then we would expect the true positivity rate based on culture test
alone to be 87% (as that is the percentage tested who were deemed positive at day 1). The
percentage would increase if antibody testing was also used to diagnose (i.e. there would
presumably be some patients who tested negative by culture test at baseline but had 4-fold increase
in influenza antibodies). In the trials many participants were not tested by culture or by antibodies.
Had they been tested we would expect a much higher % with influenza
8. If we base diagnosis on culture test at day 1 alone then 63-64% of patients would be deemed
influenza infected however there would be a significant number of misclassified patients in the noninfluenza group
110
111
112
113
114
115
116
117
118
Given influenza diagnosis was based on results of culture test at day 1, day 3 or serology the issues above
provide serious limitations to the ability to accurately diagnose influenza infection. The biggest issue appears
to be that the non-influenza group would appear to include misclassified oseltamivir patients and a high
proportion of patients with unknown infection status. Given the similarity of the treatment trials of oseltamivir,
we presume the data quality would be similar for the other treatment trials (although in a second pilot study
based on pivotal trial WV15670 [data not shown] the quality appears slightly better but similar problems with
influenza diagnosis remain). These serious limitations would mean any subgroup analysis comparing
influenza and non-influenza groups is at high risk of bias hence conclusions on whether oseltamivir has a
non-specific effect could not be drawn with confidence.
119
120
121
122
123
124
125
126
127
Criticism: inaccurate calculation of summary measures
There was no definition for mean survival time provided in the trial protocols. There are various ways to
estimate mean survival based on censored data1 and we used the restricted mean estimation available in
Stata which is different to that available in SAS. When we use the SAS generated estimates we obtain a
slightly reduced treatment effect in favour of treatment:
Time to first alleviation of symptoms in adult treatment (ITT population) based on means
128
129
130
131
132
Criticism: better to use medians rather than means
133
Time to first alleviation of symptoms in adult treatment (ITT population) based on medians*
This is a debatable but ultimately moot point as use of medians makes little difference as shown below
Ver 12 23 Mar 2015
134
135
136
*Please note we are unable to change the labels from mean to median in the RevMan software, so while the
labels say “mean difference”, these are actually displaying “median difference”.
137
138
139
We note a similar result is obtained in the pooled analysis of the ITT population shown on pg 48 of the
Roche report (difference in medians = 17.6 hours), although a meta-analysis including all the available data
should have been performed.
140
Criticism: Misinterpretation of case report form wording for complications
141
142
143
144
145
146
147
We are still unclear of the definitions of complications, how diagnosis was made and how the information
was reported. We would be interested in obtaining this information for every event of pneumonia and
bronchitis so that we can evaluate the effect of method of diagnosis on estimated treatment effect. Based on
the information we have, it appears oseltamivir reduces the symptoms of lower respiratory tract infections but
whether it reduces true pneumonia and bronchitis is unclear. The trial data do not appear to be of sufficient
quality to answer this question. Our conclusion is similar to that of FDA, the only other group (to our
knowledge) to have conducted a thorough analysis of the complications data.
148
149
Criticism: multiple hospitalisations reported for individual patients were included in the comparison of
hospitalisations
150
151
152
153
We were interested in comparing the rates in the 2 treatment groups rather than the proportions as we
believe this is a more informative measure (e.g. a treatment may reduce the rate of hospitalisations but have
little effect on the proportion of patients hospitalised). If calculations are based on proportions the result does
not change, i.e. insufficient evidence of a difference, as shown below.
154
Hospitalisations: proportion of patients hospitalised
155
156
157
Criticism: Analysis showing a beneficial effect of oseltamivir on pneumonia in adult treatment was
ignored
158
159
160
161
162
The statistically significant effects of oseltamivir against “self-reported, investigator-mediated, unverified
pneumonia” was reported in the abstract of the review so we reject the suggestion that we “ignored” this
result. Rather than ignoring it, we discussed the reasons to be caution in interpreting the meaning of this
result. We showed that complications were either never defined in trial protocols or mentioned but lacked
definitions, for example “pneumonia” was not consistently backed up by diagnostic ascertainment (although
Ver 12 23 Mar 2015
163
164
165
166
167
detailed individual data is not included in the CSRs). Our meta-regression finding that oseltamivir
significantly decreased the risk of unverified, self-reported, investigator-mediated pneumonia is in keeping
with the symptom shortening evidence of oseltamivir. Roche’s trials were clearly not designed to provide
compelling evidence about oseltamivir’s potential benefit against important public health outcomes like
pneumonia.
168
169
Criticism: Sensitivity analysis of bronchitis in adult treatment studies showing a statistically significant
beneficial effect of oseltamivir was ignored
170
171
172
173
174
175
176
177
We don’t agree that this has been ignored. It has been reported clearly in the results section of the review.
Like pneumonia, the same limitations on outcome assessment exist for bronchitis. This result is consistent
with the other evidence we have presented, that oseltamivir reduces symptoms but does not prevent
infection, complications of infection, or hospitalisations. Our approach to assessment has always been
conservative, given the importance of neuraminidase inhibitors to society and the documented sizeable
reporting bias of the published record. We require convincing proof of effect and caution in dismissing
possible harms. However it is true that both complications and harms lack definitions in the clinical study
reports, a point we did not make in our conclusions, but will make in the next update of the review.
178
CLAIMS OF NEW SAFETY INFORMATION ON OSELTAMIVIR.
179
180
181
We agree that we did not identify “new” data, as the data that we presented has been available to Roche for
15 years. What is new are its visibility and analyses of the whole dataset. There is nothing inappropriate in
meta-analyses by SOC on or off-treatment as long as this is clearly reported. This is what we have done.
182
183
Finally, we have made all 107 CSRs from GSK and Roche available precisely to enable independent
replication of our analyses and (perhaps) our conclusions.
184
Some responses to specific criticisms are given below.
185
Criticism: Peto method not appropriate when treatment groups are unbalanced
186
187
188
189
We note that the Peto method actually reduces the odds ratio estimate for study WV15673-697 from 4.5 to
2.9 hence it is conservative in this case. Using exact logistic regression after adjustment for study we get
overall OR = 3.8, 95% CI: 1.10 – 13.3, P=0.023 hence the p-value is consistent with the result based on
Peto’s method. This is preferable to dropping the study.
190
Criticism: Erroneous reporting of psychiatric event rates
191
192
193
194
195
196
197
198
Roche may be confused because we discovered some mistakes in the classification of psychiatric adverse
events that have been corrected as documented in a recently published erratum 2. However a limitation of our
analysis is that we could only check the small proportion of adverse events with narratives provided in the
CSRs hence it is unknown if there are additional misclassifications. We based our analyses on the original
grouping used in the CSRs as opposed to Toovey et al who created a new classification many years after
the data was collected by choosing some of the events classified as psychiatric as well as some classified as
neurological and some classified as accidents3,4. Notably confusion, depression and psychosis (for example)
were not included in the Toovey post-hoc definition.
199
Criticism: Dose–response methodology is limited/non-monotonic incidences of psychiatric events
200
201
202
203
204
205
206
Roche has chosen to report these events pooled over the 2 studies which masks the apparent dose
response effect in treatment trials. As reported in the Cochrane review “In the dose-response analysis there
was an increased risk of psychiatric body system adverse events over the entire follow-up period (P = 0.038
based on likelihood ratio test). In trial WV15670, the event rates were: 1/204, 1/206 and 4/205 in the
placebo, 75 mg and 150 mg arms respectively, whereas trial WV15671 had rates of 2/235, 0/242 and 5/242,
respectively”. This result provides statistical evidence of a dose response effect. We used logistic regression
(which Roche has previously recommended for analysis of renal events) with adjustment for study.
Ver 12 23 Mar 2015
207
208
209
210
We did not observe a dose response in the only prophylaxis trials with multiple doses intervention groups
(WV15673/WV15697).
211
Criticism: Assumptions about how oseltamivir “might work”
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
The data provided in Table 9 is new to us. However we note that very little information is provided e.g. which
trials are included and which trials are not? The Table is not based on the same data we included (as the
denominator is smaller than ours). The Table is not based on meta-analysis, is based on a biased
classification of influenza and is not based on sound statistical principles where for subgroup analysis the
treatment effects in each subgroup should be statistically compared before any reporting by subgroup is
undertaken. It is uncertain whether this kind of subgroup analysis should be undertaken in oseltamivir trials
as, due to the drug’s effect on antibody production, an accurate diagnosis cannot be made. Furthermore the
non-influenza group appears to include a high proportion of patients with unknown infection status.
“Influenza negative” patients in Roche’s analysis are not true negatives for influenza, because they include
patients who were not tested as well as false negatives due to reduction of antibody production by
oseltamivir. The immune response of the false negative patients is possibly weak making it difficult for them
to recover quickly from influenza and consequently the recovering time may be prolonged. Exclusion of
these patients with depressed antibody production from the “influenza patients” could help to explain the
reduced effect in the ITT compared to the ITTI populations. In the zanamivir trials, where there was no
evidence of an effect on antibody production, there was no evidence of a difference in treatment effects
between patients with influenza and patients with other illnesses. Are Roche suggesting oseltamivir’s effect
is just on influenza whereas zanamivir (another neuraminidase inhibitor) is on all-cause influenza-likeillness?
230
Criticism: Putative effect on antibody production is not supported by the data
231
232
233
234
235
236
237
238
239
240
241
242
243
244
Roche pointed out an apparent difference between the results reported in the 2012 report and those reported
in the 2014 report. Odds ratios were used in 2012 but relative risks were used in 2014. Pooled Odds ratio
using adult treatment data in the 2014 report is 0.83 (95%CI: 0.72 to 0.94, p=0.0044). Hence results are not
so different between 2012 and 2014. Roche say “In an analysis based on the ITT population, non-infected
patients are included and it is impossible for them to have a true antibody response.” However, “a true
antibody response” can only be determined from the placebo arm which was not influenced by oseltamivir.
Included studies are all randomized controlled trials. Hence proportion of patients with four-fold increase of
antibody production in the oseltamivir arm must be the same as in the placebo arm if oseltamivir did not
influence antibody production. We would like to reiterate that our analyses are based on randomised
comparisons of the ITT populations whereas Roche’s analyses are based on subgroups of participants in
post-hoc observational comparisons confounded by the effect of reduced antibody production by oseltamivir.
In addition the treatment effect sizes Roche have estimated appear consistent with what we have reported.
Can Roche explain why it is that the proportion of infected patients are systematically lower in the oseltamivir
arms compared to the placebo arms in the treatment trials if this is not due to a reduced antibody response?
245
Criticism: Misrepresentation of data and clinical relevance of the prophylaxis indication
246
247
248
249
250
251
252
253
254
The diagrams shown in Figure 4 imply that transmission is prevented by preventing clinical disease of
patients using oseltamivir for personal prophylaxis however that is speculation only as this has not been
tested in a clinical trial. In fact the data from the prophylaxis trials show that oseltamivir’s symptomatic effect
may be different than most think. The trials appear to demonstrate that oseltamivir reduces the likelihood of
testing positive and reduces fever. These effects do not equate with prevention of symptomatic influenza.
Reduced fever is not due to prevention of influenza but rather the pharmacological (toxicological) effects of
oseltamivir inhibiting nicotinic acetylcholine receptor.5 This could potentially lead to greater spread if patients
with influenza have reduced fever that enables them to go about their business as usual rather than isolating
themselves from others and resting at home.
255
References
IMPACT OF DATA THAT HAVE BEEN EXCLUDED OR MISINTERPRETED
Ver 12 23 Mar 2015
256
257
258
259
260
261
262
263
264
1. Barker, C. The Mean, Median, and Confidence Intervals of the Kaplan-Meier Survival Estimate—
Computations and Applications. The American Statistician, February 2009, Vol. 63, No. 1
2. Gravenstein S, Peters P. Erratum. Journal of the American Geriatrics Society 2013; 61:478.
3. Jones M, Hama R, Jefferson T, Doshi P. Neuropsychiatric adverse events and oseltamivir for
prophylaxis. Drug Safety 2012; 35(12):1187–8.
4. Toovey S. The Cochrane authors’ reply. Drug Safety 2012;35(12):1188–90.
5. Muraki K, Hatano N, Suzuki H, Muraki Y, Iwajima Y, Maeda Y, Ono H. Oseltamivir Blocks Human
Neuronal Nicotinic Acetylcholine Receptor-Mediated Currents. Basic Clin Pharmacol Toxicol. 2014
Jun 26.
265
266
Additional considerations
267
268
269
270
271
At page 52 Roche write (our emphasis): “As described earlier for the treatment studies infection is a sine qua
non for an antibody response. Prophylaxis prevents clinical disease in the oseltamivir arm only rendering
comparisons of antibody response meaningless. When asked previously by the Cochrane authors about
these trials, as part of their efforts to extend the analysis of a putative antibody effect, Roche pointed this fact
out and said the data were not suitable to address the question; this remains the case.”
272
273
Are Roche saying that oseltamivir only prevents symptoms and will not affect antibody response because it
does affect infection (which they have already stated publicly)?
274
275
276
277
And below Roche write (emphasis added): “Prophylactic use of oseltamivir has been shown to prevent
coughs and sneezes (clinical influenza), which subsequently prevents any further transmission of the virus”.
Are Roche suggesting purely a halt to mechanical spread by stopping nasal voidance of the virus? How does
oseltamivir work in contact transmission?
278
279
“Mathematical models” are not evidence and they are completely reliant on the effectiveness assumptions
incorporated.
280
281
282
283
284
At page 57 “Roche recommends that the methods section is rewritten to accurately separate what was
planned a priori and what was exploratory/post hoc and to clearly describe how the final set of analyses were
identified.”. What they fail to remind readers is that our post protocol analyses in the 2012 version of the
review were triggered by Roche’s refusal to honour their 2009 pledge to provide us with full CSRs and
refusal to engage in any meaningful correspondence as we documented in bmj.com/tamiflu.
285
286
287
Roche write that “In order to make a robust assessment of safety, all data must be evaluated, including all
relevant clinical studies (RCTs, observational studies, etc.), published literature and spontaneous safety
reports.” Can Roche tell us which regulators have seen all data?
288
289
290
Finally, Table 9, the statement at para 2.1 and other material published by Roche-sponsored authors seem
to suggest that there are additional data to those in the Clinical Study Reports. We would like Roche to
comment on this observation and reveal where such material (if it exists) might be found.
291
292
293
294
295
296
297
298
299
300
301
302
Dr Igho Onakpoya
Centre for Evidence-based Medicine
Nuffield Department of Primary Care Health Sciences
New Radcliffe House
Radcliffe Observatory Quarter
Oxford, United Kingdom
igho.onakpoya@phc.ox.ac.uk
Prof Carl Heneghan
Centre for Evidence-based Medicine
Nuffield Department of Primary Care Health Sciences
Ver 12 23 Mar 2015
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
New Radcliffe House
Radcliffe Observatory Quarter
Oxford, United Kingdom
carl.heneghan@phc.ox.ac.uk
340
341
342
CDM receives funding from the NHMRC for research in this area and for unrelated editorial work; he
receives royalties from books; and funds from the manufacturer of an oncology drug so he could
343
Acknowledgements: Thanks to Tom Jefferson for advice in the past.
344
Dr Mark Jones
School of Public Health
University of Queensland
Public Health Building, Herston Rd
Brisbane 4006, Australia
m.jones@sph.uq.edu.au
Prof Peter Doshi
Department of Pharmaceutical Health Services Research
University of Maryland School of Pharmacy
220 Arch St
Baltimore, MD 21201
pdoshi@rx.umaryland.edu
Prof Chris Del Mar
Centre for Research in Evience Based Practice
Bond University
Gold Coast
Queensland 4226 Australia
cdelmar@bond.edu.au
Competing interests: PD and CH were co-recipients of a UK National Institute for Health Research grant
(HTA – 10/80/01 Update and amalgamation of two Cochrane Reviews: neuraminidase inhibitors for
preventing and treating influenza in healthy adults and children
http://www.nets.nihr.ac.uk/projects/hta/108001). CH receives payment for running educational courses at the
University of Oxford and University of Oxford ISIS consulting services for external teaching and training. He
also receives royalties for books (Evidence Based Toolkit series by Blackwell BMJ Books). PD received
support from the European Respiratory Society for travel to give a talk at the society’s 2012 annual congress
in Vienna. He also received a 2015 new investigator award from the American Association of Colleges of
Pharmacy to fund a PhD student to work on research on how the potential harms of statins are conveyed in
drug labeling and pharmacy leaflets. IO and MJ have no interests to disclose.
deliver an invited conference presentation on prostate cancer screening in 2014.