NICE
Scientific advice &
Diagnostics assessment
programme
How to generate evidence to
support value claims for
diagnostics
Dr Grace Jennings and Dr Sarah Byron
September 23, 2014
What is NICE?
An independent institute that identifies how to:
• prevent, diagnose and treat disease and ill health in most
effective ways
• reduce inequalities and variation
• ensure quality and value for money for the NHS
: Pharmaceuticals, Devices/Diagnostics, Clinical
care, Social Care, Safe Staffing, Interventional Procedures, Public Health
NICE’s Procedural Principles
Scientific
Rigour
Timeliness
Support for
implementation
Inclusiveness
Accountability
for
reasonableness
Transparency
Independence
Review
Challenge
A NICE Process
Independent
review
Input from
topic experts
Evidence
submissions
Decision
Committee
Stakeholder
Perspectives
Public Consultation
Decision
Scarce resources
DIFFICULT CHOICES FOR
DECISION MAKERS
HTA
Health Technology Assessment (HTA)
Evidence
Product
Market
Policy-making
Evidence-based way of guiding the efficient
allocation of health care resources
Benefits of engaging with NICE
How NICE makes decisions
Two key questions asked by NICE
• How well does the technology work compared to
established practice in the National Health Service (NHS)?
• How much does this course of action cost compared to
established practice in the NHS?
End of Life
Other Health
Benefits
Extent of
Uncertainty
Innovation
ClinicalEffectiveness
CostEffectiveness
Equality &
Diversity
Decisions
at NICE
Decision-making at NICE
Other Social
Value
Judgements
Value Proposition
Impact on
health system
resources
Fit with health
system
priorities
Justifiable
Price
Improved
health
outcomes
Value
Incremental
benefit for
patients
Well
constructed
evidence base
Value varies depending on your perspective
Perspective
Understand the perspective of your decision maker and know
what question they want to answer
NICE takes the perspective of the
National Health Service (NHS) and
Personal Social Services (PSS)
Defining the clinical question:
Components
•
•
•
•
Population
Intervention
Comparator
Outcomes
PICO
Putting your case together
Population
Usually the patients indicated in the
marketing authorisation
Intervention Technology to be appraised
Comparators Established practice in the NHS
Outcomes
Outcomes which have an impact on:
- survival
- health related quality of life (HRQoL)
Navigating NICE
Why seek scientific advice?
Product developers may be interested in ensuring their clinical
development studies and other plans generate the evidence
which is relevant to NICE
Increase the likelihood that clinical trials and other research
activities undertaken meet NICE evidence requirements
Typical issues raised for advice
Clinical Trial Programme
Study population
Comparators
Acceptability of endpoints, surrogate endpoints
Trial design
Appropriateness of health-related quality of life and other Patient
Reported Outcomes
• Positioning in the clinical pathway
•
•
•
•
•
Economic Evaluation
• Plans for using specific economic models
• Sources of data , observational studies, analyses
Additional issues raised in previous
diagnostics advice projects
• Study design
• Diagnostic cohort design, case-control study, cluster
randomisation trial
• Sensitivity and specificity
• Diagnostic accuracy
• Stratification by risk factor
• Clinician blinding
• Outcomes relevant to NICE
• Role of new diagnostic in existing treatment pathway
• Assessment of cost-effectiveness
• Model structure and input assumptions
Key stages of advice processes
Submission of briefing package with
specific questions
Clarification of content/identification of
issues
Face-to-Face Meeting
Advice: Company minutes or formal
written advice report
Advice for developers of screening tests
• Screening not part of NICE remit
• Screening and diagnostic tests often similar
• Diagnostic test may be used as screening test in treatment
pathway
• National Screening Centre
• Part of Public Health England
• Areas of screening needs explored
• Models developed by external academic group
• New initiative – advice from NICE Scientific Advice in
conjunction with NSC for screening tests
Questions?
Programmes at NICE
Device
Test
Sponsors notify
topics to NICE
Medical Technologies Advisory Committee
(MTAC)
Medical
Technologies
Evaluation
Programme
(MTEP)
Diagnostics
Assessment
Programme
(DAP)
Technology
Appraisals
(TA)
Topic Selection
STEP 1
The Company submits a notification form to Medical
Technologies Evaluation Programme that details:
• Product description
• Patient population
• Current management and comparator(s)
• Claimed patient benefit
• Claimed healthcare system benefit
• Claimed sustainability benefit
• Costs
• Patient safety
Notifying a product to NICE
STEP 2.
NICE produces a briefing note for MTAC which decides whether
the technology is suitable for the evaluation at NICE
STEP 3.
If selected, MTAC routes the technology to the appropriate
Programme.
Process takes ~10 weeks
Questions about the process and eligibility?
Contact MTEP:
medtech@nice.org.uk
MTEP vs DAP
Medical Technologies Evaluation
Programme
• Single product evaluated
• Early stage evidence
• Innovative devices and
diagnostics
• More benefit for the same
cost or
• Same benefit for less cost
Diagnostic Assessment Programme
•
•
•
•
Multiple products evaluated
Single products evaluated
More cost for more benefit
Complex care pathways
DAP and MTEP encourage further research into
promising technologies
Diagnostics Assessment Programme
• Specialist programme to undertake complex
assessments of diagnostic technologies
• Decision making by independent Diagnostics Advisory
Committee
• Assessment of single or multiple technologies
• No formal manufacturer submission required
• Systematic review of evidence and modelling to
estimate outcome benefits and cost effectiveness is
undertaken as part of the assessment
Overview of assessment process
• Guidance topic referred to DAP from MTEP
• Scoping
• Assessment of evidence by external assessment group
• Draft recommendations developed by DAC
• Draft recommendations released for public consultation
• DAC consider feedback and develop final recommendations
• Final recommendations approved by GE for publication
• Guidance released for resolution period
• Guidance published
Two key questions asked by NICE
• How well does the technology work compared to
established practice in the National Health Service (NHS)?
• How much does this course of action cost compared to
established practice in the NHS?
Scoping
Single technology notified
and referred to DAP
DAP technical lead
• Diagnostic pathway
• Care pathway
• Alternative
technologies
• Relevant population(s)
• Costs
• Outcomes
• Potential equality
issues
• Potential
implementation
barriers
Draft scope
Scoping workshop
Registered stakeholders
Revised scope
ASG meeting
Specialist Committee
members
Standing DAC member
Final scope
Assessment of single or multiple technologies
1. How is the
condition
managed in the
NHS?
2. Where does my
product fit in the
care pathway?
3. What does my
product deliver?
Understanding benefits: diagnostics
Diagnostic Test
Positive
False
positive?
Negative
False
negative?
Treatment
Improved
survival/
Quality of life
Understanding benefits: diagnostics
Patient benefits rarely arise from the diagnostic directly
– they come mainly from treatments informed by the
diagnostic
• The treatment pathway or the range of pathways must
be understood for the value of the diagnostic to be
assessed
• Test side effects should be included
Assessment
What data do you need?
Regulator
HTA
• Product safety: laboratory
testing with clinical trial data
for devices with greatest risk
• Evidence on clinical
effectiveness (compared to
established practice):
trials, evidence synthesis
• Evidence of efficacy: does
the device meet its intended
purpose? (not necessarily
from comparative studies)
• Evidence on cost effectiveness:
trials, modelling
• Evidence on relative
safety/adverse events
Trueman P et al 2011
Evidence considerations
• The ideal evidence would be a good quality ‘end-to-end’ study –
follows patients from testing, through treatment, to final outcomes
• Typically not available for diagnostics
• Search for data on test accuracy, direct outcomes from the test,
indirect health outcomes from the test result, and costs
• Identified evidence can then be combined through a linked
evidence approach
Diagnostic
accuracy
Impact on
treatment
decisions
Impact on
outcomes
Evidence hierarchy
Systematic
reviews of
RCTs
RCTs
Controlled observational
studies (e.g. case-control)
Uncontrolled observational
studies (e.g. case reports)
Expert opinion
Study design
• Outcomes: patient focussed outcomes are particularly
important, as opposed to intermediate or surrogate outcomes
• e.g. a reduction in tumour size will be given less weight
than evidence about clinical benefit such as improved
survival or quality of life
• Size: Studies with larger numbers of patients will usually be
preferred as estimates of benefits and harms will be more
accurate
• Duration: Studies should have sufficient follow up to capture
final outcomes where possible
• e.g. very important for prognostic tests
Diagnostic tests: Outcomes data
Ideally comparative ‘end-toend’ clinical studies including
the test and subsequent
treatments should be
conducted
Test side effects should be
included
Not possible
Identify studies on the
effectiveness of those
subsequent treatments
Use a systematic approach to
identifying relevant studies
Diagnostic tests: Outcomes data
• Measurements of test accuracy are necessary:
Condition as determined by
“Gold Standard”
Test
outcome
Condition
positive
Condition
negative
Test
outcome
positive
True
Positive
False
Positive
PPV
Test
outcome
negative
False
negative
True
Negative
NPV
Sensitivity
Specificity
Diagnostic tests: Outcomes data
Cut off points
Example:
SonoVue (sulphur hexafluoride microbubbles)
Contrast agent for contrast-enhanced ultrasound imaging
of the liver
Characterising
incidentally detected
focal liver lesions
Detection of
potential liver
metastases
Characterising
focal liver lesions
(cirrhosis)
No end-to-end studies available
High quality accuracy data – SonoVue vs CT and MRI
Relevant evidence on care pathway and outcomes
Characterising
incidentally detected
focal liver lesions
Detection of
potential liver
metastases
Characterising
focal liver lesions
(cirrhosis)
Cost
effective
Slightly
less cost
effective
than CT
and MRI
Adoption
recommendation
Adoption
recommendations
where CT and
MRI not
appropriate
+
Research
recommendations
to explore
potential broader
applicability
Decision making
Recommendations
• Adoption recommendations
• Research recommendations
• Not recommended
Guidance development
• Decision making in presence of uncertainty
• Public consultation can change decision making
• Clarity in recommendations on indication
o Rule-in / rule-out / diagnosis / monitoring
o Setting
 Supported by evidence, minimise risk of indication creep and
inappropriate use of tests that may lead to misdiagnosis
 Cost-effective use of NHS resources
• ‘Committee considerations’ describe uncertainties and
rationale behind decision-making.
Examples of recommendations
Diagnostic Guidance (DG5)
SonoVue (sulphur hexafluoride microbubbles) –
contrast agent for contrast-enhanced ultrasound imaging of the liver
Contrast-enhanced ultrasound with SonoVue is recommended for
characterising incidentally detected focal liver lesions in adults in whom an
unenhanced ultrasound scan is inconclusive. An unenhanced ultrasound
scan in which a focal liver lesion is detected, but not characterised, is
defined as inconclusive.
Recommendations
• Adoption recommendations
• Implementation support
• Health Technologies Adoption Programme
• Research recommendations
• NICE research commissioning
• Not recommended
NICE: Companion diagnostics
Technology Appraisal
Programme
Diagnostics Assessment
Programme
Appraisals of new and existing
medicines and treatments
Specialist programme to undertake
complex assessments of diagnostic
technologies
Topics from department of health
Topics from manufacturers or clinical
sponsors
Assesses single or multiple
technologies
Assesses single or multiple
technologies
STA - Manufacturer submission
No formal manufacturer submission
MTA – Manufacturer submission and
evidence gathered by academic group
Structured information request
Evidence gathered by academic group
Evidence assessed by independent
external group
Evidence assessed by independent
external group
STA – 35 weeks
MTA – 62 weeks
62 weeks
NICE: Companion diagnostics
• Alternative technologies
• Timing is an important factor
o Appraisal programme coordinates publication of guidance with market
authorisation of drug
o CDx may be developed simultaneously with drug
o CDx can come to market years after drug approval or may already be
available
o Rapid development of CDx and supporting technologies
Technology Appraisals: Companion diagnostics
• In January 2013, NICE published update to methods
guide
• Costs of CDx testing incorporated into evaluation of clinical and
cost effectiveness
• Sensitivity analysis to assess impact of CDx cost on cost
effectiveness of pharmaceutical
• Diagnostic accuracy can be examined and incorporated in cost
effectiveness analysis
• Potential issues of alternative CDx can be highlighted in
guidance without assessment of evidence
Example of CDx in TA programme
• TA 208 Trastuzumab for HER2-positive metastatic gastric cancer
• MA included testing with fluorescence in situ hybridisation (FISH) then
revised to include silver in situ hybridisation (SISH)
• Timing of MA meant that only FISH was included in NICE appraisal
• Trial used parallel testing strategy
• Sequential testing strategy in manufacturer’s model
• Only ICH2 positive received FISH test
• ERG scenario analyses for both sequential and parallel testing strategies
• Sequential ICER £66,982 per QALY
• Parallel ICER £71,637 per QALY due to increased incremental costs
• Committee concluded that sequential testing was most appropriate for
people with metastatic gastric cancer
Example of CDx in DAP programme
• EGFR-TK mutation testing in adults with locally advanced or
metastatic non-small-cell lung cancer
• Evidence
o Two tests used in clinical trials
o Three tests had accuracy data
o Linked to clinical trial data
o Remaining tests had no trial or accuracy data
o Included a survey of labs providing EGFR-TK testing
o test characteristics and costs
o Data from an EGFR-TK national external quality assurance
scheme study
EGFR testing - Recommendations
• 5 tests recommended but insufficient evidence to make
recommendations for others
• Key issues:
• Test validation
• Competent execution
• Participation in external quality assurance scheme
• Research recommendation
• Studies comparing different EGFR-TK mutation methods that link
to patient outcomes
• Many assumptions in assessment
Diagnostics guidance (http://www.nice.org.uk/dg9)
Summary
Ways of assessing alternative CDx:
• Only assess CDx used in clinical trial
• include general commentary on use of alternative CDx tests
• Where appropriate, undertake sensitivity analysis on
diagnostic accuracy and cost to understand the importance
of CDx in relation to patient outcome benefits and cost
effectiveness
 Technology Appraisal programme
• Assess each alternative CDx / treatment package separately
• Separate clinical and cost effectiveness analyses
 Diagnostics Assessment programme
Health Technology Adoption
Programme (HTAP)
Main objectives:
• Increase the uptake of NICE recommended technologies
• Establish an agreed approach for NICE to develop
effective partnerships with Academic Health Science
Networks (AHSN)
• To support the expansion of the Medical Technologies
Industry in the UK by providing adoption advice to
suppliers
Two approaches to adoption in the NHS:
• Site demonstrator projects for commonly used
technologies
• Adoption projects for more complex, less commonly
used technologies
Produce adoption support packs to accompany guidance produced
in MTEP, DAP, TA and Quality Standards (QS).
Please contact Jae Long (jae.long@nice.org.uk), Assistant Project
Manager for HTAP, for further details.
57
Key contacts
• NICE Scientific Advice
• Richard Chivers (richard.chivers@nice.org.uk)
• http://www.nice.org.uk/about/What-we-do/Scientific-advice
• NICE/Medilinks seminar http://www.medilinkem.com/events/events-calendar/2014/10/21/nicemedtech-workshop
• NICE DAP
• Sarah Byron (sarah.byron@nice.org.uk)
• http://www.nice.org.uk/About/What-we-do/Our-Programmes/NICE-guidance/NICE-diagnosticsguidance
• NICE Medical Technologies Evaluation Programme
• Lee Dobson (lee.dobson@nice.org.uk)
• http://www.nice.org.uk/About/What-we-do/Our-Programmes/NICE-guidance/NICE-medicaltechnologies-evaluation-programme
• Health Technologies Adoption Programme
• Jae Long (jae.long@nice.org.uk)
• http://www.nice.org.uk/About/What-we-do/Into-practice/Health-Technologies-Adoption-Programme
Any questions?