An empirical study comparing
health technology assessment
reports for drugs vs. medical
devices
Britni Wilcher
Associate Research Fellow in Health Technology Assessment
University of Exeter Medical School, UK
Collaborators
 Oriana Ciani
 Rod Taylor
 Anoukh van Giessen
Objectives
To compare HTA reports of drug and
medical devices for the treatment of
cardiovascular disease at a number of
levels:
Nature of evidence
Treatment effect
HTA methods
Approach to
address uncertainty
Adoption
Recommendation
Study design & methods
• Study design: Systematic Review
• Data collection & analysis
– Stage 1 – Selection and screening of reports
– Stage 2 – Data extraction
– Stage 3 – Data analysis
• Quantitative
• Qualitative (e.g., future research
recommendation, methods adopted to address
technology specific issues)
Preliminary Results: Selection
of HTA Reports
Preliminary Results: Key
Characteristic of Reports
Publication Date
2003 - 2008
2009 - 2013
Agency (by Country)
Australia
Canada
Ireland
United Kingdom
United States of America
Indication
Cerebrovascular
Coronary
Peripheral
Class
Class I (generally regarded as low risk)
Class IIa (generally regarded as medium risk)
Class IIb (generally regarded as medium risk)
Class III (generally regarded as high risk)
1. Calculated with Fisher exact test
P-value1
Drug
(N = 18)
Device
(N = 27)
n(%)
n(%)
2 (11)
16 (89)
2 (7)
25 (93)
1.000
-3 (17)
-12 (67)
3 (17)
1 (4)
12 (44)
1 (4)
8 (30)
5 (19)
0.077
2 (11)
12 (67)
4 (22)
-19 (70)
8 (30)
0.303
0 (0)
3 (11)
5 (19)
19 (70)
Preliminary Results: Nature of
Evidence
1.
2.
3.
4.
Type of clinical study
RCTs
Non-RCTs
Observational studies
Evidence Synthesis2
Other3
Number of Patients
RCTs
Non-RCTs
Observational studies
Evidence Synthesis2
Other3
Type of economic evaluation
Cost analysis
Cost minimisation analysis
Cost-effectiveness analysis
Cost-utility analysis
Cost-benefit analysis
Cost-consequence analysis
n
%
Drug
(N = 18)
Mdn(Rng)
17
4
3
6
1
94
22
17
33
6
5 (1; 35)
5.5 (1; 18)
46 (13; 92)
5.5 (5; 30)
89 (NA)
18
12
13
8
2
67
44
48
30
7
5 (1; 82)
6 (2; 29)
25 (4; 53)
5 (1; 15)
1.5 (1; 2)
13
3
1
1
1
72
17
6
6
6
4,203 (34; 66,477)
4,917 (926; 184,372)
7,636 (NA)
102,594 (NA)
NA
12
5
7
1
2
44
19
26
4
7
1,482 (291; 35,597)
836 (79; 12,217)
646 (76; 13,890)
102,594 (NA)
NA
1
0
8
8
0
0
6
0
44
44
0
0
5 (NA)
-4 (1; 20)
3.5 (1; 8)
---
4
0
9
9
1
0
15
0
33
33
4
0
1.5 (1;2)
-2 (1; 8)
1 (1; 4)
1
--
n
%
Device
(N = 27)
Mdn(Rng)
Calculated with Mann-Whitney
Includes systematic reviews, pooled analyses, meta-analyses, and HTA reports
Includes rapid reviews and sources of evidence that do not fall into the above mentioned hierarchy of evidence categories
Significant p-value
P-value for
Percentage
P-value for
Median1
0.02814
0.1271
0.03074
0.7926
0.8073
0.9174
0.4309
0.2414
0.3006
0.2207
0.2314
0.1797
0.5127
0.3173
0.3329
-0.4514
0.4514
0.4090
--
0.1360
-0.5250
0.1051
---
7
Preliminary Results: Nature of
Evidence
T ype of ev idence considered 3
H e alth p ro b lem an d cu rre nt u se o f te ch no lo gy
D e scrip tio n an d te ch n ical ch aracte ristics o f te ch n o lo gy
Safe ty
C lin ical e ffe ctive ne ss
C o st an d e co n om ic evalu atio n
Eth ical asp ects
O rgan isatio n al asp e cts
So cial aspe cts
Le gal asp ects
D rug
(N = 18 )
D ev ice
(N = 27 )
n (% )
n (% )
P -valu e 1
1 5 (83 )
1 5 (83 )
1 2 (67 )
1 7 (94 )
1 3 (72 )
1 (6 )
1 (6 )
5 (2 8 )
1 (6 )
1 0 (37 )
8 (3 0 )
1 7 (63 )
2 4 (89 )
2 0 (74 )
1 (4 )
1 2 (44 )
3 (1 1 )
1 (4 )
0 .0 03 2
0 .0101 2
0 .7 99
0 .6 40
1 .0 00
1 .0 00
0 .0 06 2
0 .2 35
1 .0 00
1. Calculated with Fisher exact test
2. Significant p-value
3. Components of EUnetHTA HTA Core Model®
8
Preliminary Results: Methods
Drug
(N = 18)
Methods of Data Synthesis
Systematic Review of clinical outcomes
Meta-analysis
Systematic Review of economic evaluations
Economic model
AMSTAR
Total2,3
Drummond Checklist
Total2,3
1.
2.
3.
4.
Device
(N = 27)
Total N
n(%)
Total N
n(%)
P-value1
18
18
18
18
14 (78)
10 (56)
9 (50)
11 (61)
27
27
27
27
23 (85)
9 (33)
10 (37)
17 (63)
0.524
0.139
0.388
0.900
15
7.47(2.23)
24
5.5(2.71)
0.0364
11
7.56(1.81)
17
5.29(2.95)
0.0244
p-value calculated using fisher exact test
Mean (standard deviation) reported; AMSTAR total score: low quality = 0 - 3, medium quality = 4 - 7, high quality = 8 – 11
p-value calculated using Mann-Whitney U Test
Significant p-value
9
Preliminary Results: Uncertainty
Highlights - Handling of uncertainty in the included HTA reports
Did the clinical analysis take uncertainty into consideration?
How did the clinical analysis take uncertainty into account?
95% Confidence Intervals
Sensitivity Analysis
Weighted mean
Did the economic analysis take uncertainty into
consideration?
How did the economic analysis take uncertainty into account?
Cost Effectiveness Acceptability Curves (CEACs)
Distribution
P-values
Sensitivity analysis
Drug
(N = 18)
Device
(N = 27)
n(%)
11 (61)
n(%)
8(73)
3(28)
0(0)
11 (61)
n(%)
11 (41)
n(%)
7(64)
4(36)
1(9)
16 (59)
n(%)
4(36)
1(9)
1(9)
9(82)
n(%)
1(6)
1(6)
1(6)
14(88)
P-value
0.181
0.901
10
Preliminary Results: Treatment
Effect
11
Preliminary Results: Adoption
Recommendation
Policy recommendations yielded from included HTA reports
Recommendation
Unrestricted
Optimised
Only in research
Not recommended
Not reported
Drug
(N = 18)
Device
(N = 27)
n(%)
n(%)
1 (20)
1 (20)
1 (20)
2 (40)
13 (72)
0 (0)
5 (83)
0 (0)
1 (17)
21 (78)
P-value
0.207
12
Strength & Limitation
• We believe this to be the first empirical
comparison of drug and device HTA
reports
• The results must be contextualized as they
represent CVD interventions reflected in
English-language publications in the CRD
HTA database.
13
Preliminary Conclusions
• Device HTA reports compared to Drug
HTA reports appear to be:
– less likely to include RCT vs observational
evidence than drug reports but are similar in
nature of economic evidence
– systematic review and economic evaluation
methods are lower quality
– less likely to provide a detailed description of
technology and current health indication but
more likely to consider organisational issues
– equally likely to handle uncertainty
Thank you!
Email: b.wilcher@exeter.ac.uk
15
BACK-UP SLIDES
16
Research Question 1: Nature of
Evidence
1.1. What is the nature of evidence
included in the HTA reports on
medical devices compared to drugs
for cardiovascular disease?
1.2. What is the scope and nature of
outcomes assessed in HTA reports
on drugs relative to HTA reports on
devices for cardiovascular disease?
17
Research Question 2a: HTA
Methods
2.1. Do the methods (e.g., systematic review,
economic modelling, meta-analysis) applied in
HTA reports on medical devices differ from drugs
for cardiovascular disease?
2.2. Do HTA reports on medical devices and
drugs for cardiovascular disease consider and
adopt methods to evaluate technology specific
challenges?
18
Research Question 2b:
Uncertainty
2.3. What methods are used to address
data uncertainty in HTA of medical devices
compared to drugs for cardiovascular
disease?
19
Research Question 3: Treatment
Effect
3. Is the magnitude of treatment
effects seen in HTA reports of
medical devices comparable to
drugs?
20
Research Question 4: Adoption
Recommendation
4. Is the final technology adoption
recommendation in reports on medical
devices less likely to be positive than HTA
reports of drugs for cardiovascular disease?
21
Stage 1: Selection and screening
of reports
Phase 1: Identification
•
•
Period: 2003 – 2013
Cardiovascular indication
• Cardiac
• Cerebrovascular
• Peripheral
Phase 2: Screening
2 reviewers independently screened all titles & abstracts. Excluded if:
1. Primary indication non
5. Guidelines
cardiovascular disease
6. Abbreviated review
2. Surgical procedure w/o device 7. Not publically available
3. Diagnostic/Prognostic
8. Not in English
4. Non-drug/device
22
Stage 2: Data Extraction
• Section 1: General Information
• 16-items
• Section 2: Nature of Evidence
• 16-items
• Section 3: HTA Methods
• 4-items
• Section 4: Quality of Review
(Systematic Review & Economic
Evaluation)
• 21-items
• Section 5: Technology specific
considerations
• 2-items
• Section 6: Uncertainty
• 4-items
• Section 7: Results
• 3-items
• Section 8: Conclusions
• 2-items
23