The Appraisal, Extraction and
Pooling of Cost and Cost
Effectiveness Studies
JBI/CSRTP/2012-13/0005
Introduction
• Recap of Introductory Module
–
–
–
–
Developing a question (PICO)
Inclusion Criteria
Search Strategy
Selecting Studies for Retrieval
• This module considers how to
appraise, extract and synthesize
evidence from
Cost and Cost Effectiveness studies.
Aim and Objectives
• The objectives of this module are to prepare
participants to:
– critically appraise studies of cost and cost
effectiveness,
– extract data from cost and cost effectiveness
studies,
– summarize the results of cost and cost
effectiveness studies.
Program Overview
Day 1
Time
0900
0915
1030
1100
Session
Introductions and overview of Module
Session 1: Introduction to review of
evidence on cost and cost effectiveness.
Session 2: Critical Appraisal of Cost and
Cost Effectiveness studies
Morning Tea
Session 2 Continued…
1130
Session 3: Study Data and Data Extraction
1230
1330
Lunch
Session 4: Protocol Development
1400
Session 5: Synthesis/Reporting in reviews of
Costs and Cost Effectiveness
Afternoon tea
Session 6: Appraisal, Extraction and
Synthesis using JBI-ACTUARI
Session 7: Protocol Presentations
Session 8: Module Assessment
End
0945
1420
1445
1530
1600
1700
Group Work
Group Work 1: Identification of Cost and Cost
Effectiveness. Report back
Group Work 2: Critically appraising Cost and Cost
Effectiveness studies. Report back
Group Work 3: Data extraction from Cost and Cost
Effectiveness studies.
Report back
Group Work 4: Using CReMS, develop a draft
protocol
Group Work 5: JBI-ACTUARI software trial
(PICO Question and Inclusion Criteria only).
Session 1: Introduction to review of
evidence on cost and cost effectiveness
Common study designs
• Prospective experimental or quasi experimental
effectiveness studies with cost or cost effectiveness
components
• Modelling studies
Four approaches to analysis
•
•
•
•
Cost-minimization analysis (CMA);
Cost-effectiveness analysis (CEA);
Cost-utility analysis (CUA);
Cost-benefit analysis (CBA).
Methods, measures, benefits
Types of
studies
Costs or
measures
Benefits or
Consequence
measures
Comments
Cost
Minimization
Analysis (CMA)
Costs measured in
monetary units (e.g..
Dollars)
Not measured
CMA is not a form of full economic analysis,
the assumption is that benefits or
consequences are the same, therefore the
preferred option is the cheapest
Cost
Effectiveness
Analysis (CEA)
Costs measured in
monetary units (e.g..
Dollars)
Benefits measured in
natural units (e.g..
mmHg, cholesterol
levels, symptom free
days, years of life
saved)
Results are expressed as dollars per case
or per injury averted. Different incremental
summary economic measures are reported
(e.g.. Incremental cost-effectiveness ratio)
Cost Utility
Analysis (CUA)
Costs measured in
monetary units (e.g..
Dollars)
Benefits expressed in
summary measures as
combined quantity and
quality measures (e.g..
QALY, DALY etc)
Two dimensions of effects measured
(quality and length of life); results are
expressed for example as cost per QALY
Cost Benefit
Analysis (CBA)
Costs measured in
monetary units (e.g..
Dollars)
Benefits measured in
monetary units (e.g..
Dollars)
Benefits are difficult to measure monetarily,
values used are Net Present Value (NPV)
and Benefit Cost Ratio (BCR)
Cost-minimization analysis (CMA)
• In cost-minimization analysis (CMA) only the costs
of the interventions are compared; the outcomes
are assumed to be equivalent.
PICO Questions – Cost Minimization
• What is the evidence on costs (direct and indirect) of
laparoscopic compared to open appendectomy for
patients aged 15 years or over (assuming the longterm outcome is the same in both groups)?
Cost-effectiveness analysis
• Costs are measured in monetary units;
• The outcome is common to both alternatives but the
effect size and direction may vary;
• Outcomes are measured in natural/clinical units;
– (e.g. mortality, myocardial infarctions, lung
function, weight, bleeds).
Cost–effectiveness Plane
• A four-quadrant figure of cost difference plotted
against effect difference:
– quadrant I, intervention more effective and more costly
than comparator;
– quadrant II, intervention more effective and less costly
than comparator;
– quadrant III, intervention less effective and less costly than
comparator; and
– quadrant IV, intervention less effective and more costly
than comparator. (Culyer, 2005:77-78)
Cost Effectiveness Plane
Q4
Q3
Q1
Q2
PICO Questions – Cost Effectiveness
• What is the cost effectiveness of percutaneous
coronary intervention with drug-eluting stents (PES)
compared to bare-metal stents (BMS) to reduce
angina symptoms for patients undergoing singlevessel percutaneous coronary intervention?
Cost-utility analysis
•
•
•
•
Costs are measured in monetary units;
Outcomes are common to both alternatives;
Effect size and direction may vary;
Outcomes are measured as healthy years (typically
measured as quality-adjusted life-years (QALYs)).
Outcome measures for CUA
• The primary outcome for CUA is expressed as
quality-adjusted life-years (QALYs);
• Other generic outcome measures for CUA:
– Disability-adjusted life-year (DALY);
– Healthy years equivalent (HYE);
– Saved-young-life-equivalent.
(Drummond et al., 2005:14)
PICO Questions – Cost Utility
• What is the cost-utility of the cochlear implant in
adults (age >18 years) with profound bilateral, postlingual deafness compared with no intervention?
Cost-benefit analysis
• Costs are measured in monetary units;
• Outcomes are identified as single or multiple effects;
• The effects are not necessarily common to both
alternatives;
• Outcomes are measured in monetary units.
PICO Questions – Cost Benefit
• What is the cost-benefit of donepezil compared to
galantamine for cognitive function in patients with
mild to moderate Alzheimer’s disease?
Searching for Evidence
•
•
•
•
Cost and Cost Effectiveness keywords
Clinical keywords
General databases
Specific databases
Specific economic databases
NHS Economic Evaluation Database (NHS EED)
Health Economic Evaluation Database (HEED)
Cost-effectiveness Analysis (CEA) Registry
Health Technology Assessment (HTA) database
Paediatric Economic Database Evaluation (PEDE)
European Network of Health Economic Evaluation
Databases (EURONHEED)
• COnnaissance et Decision en Economie de la Sante
(CODECS)
•
•
•
•
•
•
Group Work 1: Identification of Economic
Evaluation Study Designs
• Refer to Workbook.
• Report back
Session 2: Critical Appraisal of Cost and
Cost Effectiveness Studies
1004 references
172 duplicates
832 references
Scanned Ti/Ab
715 do not meet
Incl. criteria
117 studies
retrieved
82 do not meet
Incl. criteria
35 studies for
Critical Appraisal
Why Critically
Appraise?
• Combining results of
poor quality research
may lead to misleading
understandings of
issues explored
The Critical Appraisal Process
• Every review must set out to use an explicit
appraisal process. Essentially,
– A good understanding of research design is
required in appraisers; and
– The use of an agreed checklist is usual.
Critical appraisal of cost and cost
effectiveness evidence
• Primary purpose of critical appraisal is to assess a
study’s quality and determine the extent to which a
study has excluded the possibility of systematic
flaws in its design, conduct and analysis.
JBI Critical Appraisal Checklist for cost and
cost effectiveness studies
1. Is there a well defined question?
2. Is there a comprehensive description of alternatives?
3. Are all important and relevant costs and outcomes for
each alternative identified?
4. Has clinical effectiveness been established?
5. Are costs and outcomes measured accurately?
6. Are costs and outcomes valued credibly?
JBI Critical Appraisal Checklist for cost and
cost effectiveness studies
7. Are costs and outcomes adjusted for differential timing?
8. Is there an incremental analysis of costs and
consequences?
9. Were sensitivity analyses conducted to investigate
uncertainty in estimates of cost or consequences?
10. Do study results include all issues of concern to users?
11. Are the results generalizable to the setting of interest in the
review?
JBI Economic evidence appraisal
1. Is there a well defined question?
– Costs and effects;
– Comparison of alternatives;
– Perspective of the analysis (including the decisionmaking context).
Effects
• Mortality measurements;
• Morbidity measurements;
• Health-related quality of life
measurements.
Perspective
• The ‘viewpoint’ adopted for the purposes
of an economic appraisal (cost–
effectiveness, cost–utility studies and so
on) which defines the scope and character
of the costs and benefits to be examined.
Perspectives
•
•
•
•
•
•
Societal perspective;
Health sector perspective;
Other sector perspective;
Health insurance perspective;
Hospital perspective;
Patient perspective.
JBI Economic evidence appraisal
2. Is there a comprehensive description of
alternatives?
– Important alternatives
– Do-nothing alternative
JBI Economic Evidence Appraisal
3. Are all important and relevant costs and
outcomes for each alternative identified?
– Was the range wide enough for the research
question;
– Does it cover all relevant perspectives;
– Were capital as well as operating costs
included.
Typical classification of costs
•
•
•
•
•
•
Functional costs;
Financial and economic costs;
Direct, indirect and intangible costs;
Capital and recurrent costs;
Fixed and variable costs;
Opportunity costs
Functional costs
• Can be classified into categories:
–
–
–
–
–
–
–
personnel;
buildings and space;
equipment;
supplies and pharmaceuticals;
transportation;
training;
information, education and communication.
Financial and economic costs
• Financial costs are defined as the actual
money spent on the resources;
• Inclusion of the costs of all resources,
regardless of their financial cost is known
as the economic cost.
Direct, indirect and intangible Costs
• Direct costs are associated directly with a
healthcare intervention (e.g. drugs,
staffing);
• Indirect costs refer to the productivity
gains or losses (e.g. time off work, illness);
• Intangible costs refer to the nonmonetary assets that can not be readily
seen (e.g. anxiety, fatigue, pain or
suffering from an illness or treatment).
JBI Economic Evidence Appraisal
4. Has clinical effectiveness been established?
– Was this through experimental research?
• If so did the trial protocol reflect what would happen in regular
practice?
– Was effectiveness established through a synthesis of clinical
studies?
– Were observational data or assumptions used to establish
effectiveness?
• If so what were the potential biases in results?
JBI Economic Evidence Appraisal
5. Are costs and outcomes measured accurately?
– Were any of the identified items omitted from the
measurement?
• If so does this mean that they carried no weight in the
subsequent analysis?
– Were there any special circumstances (e.g. joint use of
resources) that made measurement difficult?
• If so, were these circumstances handled appropriately
JBI Economic Evidence Appraisal
6. Are costs and outcomes valued credibly?
– Were the sources of all values clearly identified?
• Possible sources include market values, patient or client preferences and
views, policy maker’s views and health professional’s judgements;
– Were market values employed for changes involving resources
gained or depleted?
– Where market values were absent (e.g. volunteer labour) or did
not reflect actual values (such as clinic space donated at a
reduced rate) were adjustments made to approximate market
values?
JBI Economic Evidence Appraisal
7. Are costs and outcomes adjusted for differential
timing?
– Were costs and outcomes that occur in the future
‘discounted’ to their present values?
– Was there any justification given for the discount rate
used?
JBI Economic Evidence Appraisal
8. Is there an incremental analysis of costs and
consequences?
– Were the additional (incremental) costs generated by
one alternative over another compared to the additional
effects, benefits or utilities generated?
JBI Economic Evidence Appraisal
9. Were sensitivity analyses conducted to investigate
uncertainty in estimates of costs or outcomes?
– If a sensitivity analysis was employed, was justification
provided for the range of values (or for key study
parameters)?
– Were the study results sensitive to changes in the values
(within the assumed range for sensitivity analysis or within
the confidence interval around the ratio of costs to
outcomes)?
JBI Economic Evidence Appraisal
10.Do study results include all issues of concern to
users?
– Are the results of cost and effect for the alternative
interventions?
– Do they clearly specify the relative size of the effects
for the interventions?
– Do they clearly show how costs differ for the two
interventions?
– Can we use them with the Cost Effectiveness Plane?
– Did the study take account of other important
factors in the choice or decision under
consideration (e.g. distribution of costs or
outcomes or relevant ethical issues)?
– Did the study discuss issues of implementation
such as the feasibility of adopting the preferred
program given existing financial or other
constraints and whether any freed resources could
be re-deployed to other worthwhile programs?
JBI Economic Evidence Appraisal
11. Are the results generalizable to the setting of
interest in the review?
– Did the study make clear that the findings on costs and
effects were generated in a specific setting using
particular assumptions?
– Was the generalizability of the results to other settings
and patients/client groups discussed?
Group Work 2: Critical Appraisal of
evidence from economic evaluation studies
• Workbook
• Report back
Session 3: Study data and Data Extraction

1004 references
172 duplicates
832 references
Scanned Ti/Ab
117 studies
retrieved
35 studies for
Critical Appraisal
26 studies incl.
in review
715 do not meet
Incl. criteria
82 do not meet
Incl. criteria
Data most frequently
extracted
Considerations in Data Extraction
• Source - citation and contact details
• Methods - study design, concerns about flaws
• Participants –number, characteristics and suitability for
inclusion
• Interventions - description of interests
• Outcomes - outcomes and time points
• Results - for each outcome of interest
• Miscellaneous - funding source, etc
ACTUARI: Data Extraction
•
•
•
•
•
•
•
•
•
Interventions and Comparator
Setting
Geographical context
Participants
Source of effectiveness data
Author’s conclusion
Reviewer’s comments
Clinical effectiveness results
Economic results
First level extraction
ACTUARI: Extracting data from economic
studies
• ACTUARI data extraction
• Four options available for economic evaluation
methods
Second level extraction
Group Work 3
• Data Extraction from economic evaluation studies;
• Refer to Workbook;
• Report back.
Session 4: Protocol Development
in CReMS
Group Work 4
• Develop a draft protocol in CReMS
• Refer to Workbook;
Session 5: Synthesis/Reporting cost and
cost effectiveness evidence
Synthesis/Reporting economic evidence
• Presentation of results of synthesis:
– Tables of results;
– Narrative summary;
– Hierarchical decision matrix.
Tabular summary of economic evidence
Narrative summary of economic evidence
• “...The median and mean willingness to pay for a 25%
reduction in symptoms were $US27 and $US87 per month
(1997 values), respectively. Median and mean estimates
nearly tripled for a 50% reduction. ...Willingness to pay of
patients with urinary symptoms was between £74 and £92
per year (1999/2000 values) for complete continence with
no adverse effects, substantially lower than in the
Swedish[58] and US[40] studies. Individuals without
symptoms valued this outcome at only between £14 and
£21 per year.”
ACTUARI decision matrix summary of
economic evidence
Session 6: Appraisal, Extraction and
Synthesis using JBI-ACTUARI
Analysis of Cost, Technology and
Utilization Assessment and Review
Instrument (ACTUARI)
Screeniez next slide with drop downs active
New screenie to illustrate pathway
Session 7: ACTUARI trial
Group Work 5:
• JBI ACTUARI Software Trial
Session 8: Protocol development
Session 9: Assessment
Session 10: Protocol Presentations