Challenges in conducting a systematic review of the diagnostic accuracy
of genetic tests: an example of the genetic diagnosis of familial
hypercholesterolaemia
P Sharma1, G Mowatt1, F Stewart1, C Boachie1, Z Miedzybrodzka2, W Simpson4, D Boyers1,3, M Kilonzo3, P McNamee3
1. Health Services Research Unit, University of Aberdeen, 2. Clinical Genetics Centre, University of Aberdeen, 3. Health Economics Research Unit,
University of Aberdeen, 4. Clinical Biochemistry Laboratory, NHS Grampian, Aberdeen. CONTACT: Pawana Sharma at p.sharma@abdn.ac.uk
RESULTS - QUALITATIVE ANALYSIS
BACKGROUND AND AIM
The availability and uptake of genetic tests is increasing. In 2010 the Aberdeen HTA
group were commissioned by the UK National Institute for Health Research Health
Technology Assessment (NIHR HTA) Programme to undertake a systematic review of
Elucigene FH20 and LIPOchip for the diagnosis of familial hypercholesterolaemia1 on
behalf of the National Institute for Health and Care Excellence (NICE) Diagnostics
Assessment Programme.
Figure 1: Key methodological challenges in conducting a systematic
review of the diagnostic accuracy of genetic tests
Difficulty
defining the
reference
standard
AIM: To explore the methodological challenges involved in undertaking a systematic
review of the diagnostic accuracy of genetic tests.
Heterogeneity
across studies
Lack of valid
diagnostic
indices
METHODS
Inability to
perform
meta-analysis
1) Descriptive analysis of the issues encountered and measures undertaken to
resolve these issues during the conduct of the systematic review1 (Table 1).
2) Comparative analysis to assess the extent to which the issues faced during this
review were also encountered by other reviews. This comprised:
• Overview of systematic reviews of the diagnostic accuracy of genetic tests
• Qualitative analysis of the methodological aspects (for example, whether or not
meta-analysis was performed, susceptibility to bias, quality of reporting etc.)
Key methodological
challenges in
conducting a
systematic review of
the diagnostic
accuracy of genetic
tests
Difficulty
adhering to the
strict inclusion
criteria as
defined a priori
Evidence
gap in terms
of both
quantity and
quality
• Quantitative assessment of clinical validity, (for example, by comparing the
proportion of studies that reported diagnostic indices such as sensitivity,
specificity, and the variability of these results)
RESULTS - QUANTITATIVE ASSESSMENT OF CLINICAL VALIDITY
RESULTS - DESCRIPTIVE ANALYSIS
Table 1: Examples of issues encountered, measures undertaken to resolve
the issues and suggested recommendations
Issues encountered
Description of
the issue
Decision
made
Dilemma in selection Applying the original
of studies
strict selection criteria
would have resulted in
only 3 studies being
included.
Selection criteria were
relaxed, in consultation
with the expert
advisory panel.
Comprehensiveness
of the reference
standard
Studies using a
reference standard
that did not include
coverage of PCSK9
were included as this
was still considered to
be sufficiently
comprehensive in
correctly classifying FH.
The PCSK9 gene, a part
of the recognised
reference standard, is
rare and was
discovered in 2003,
hence would not have
been included in the
reference standard
prior to this date.
Lack of false positive Only the index testdata
negatives went on to
receive further genetic
tests, including the
reference standard.
Meta-analysis not
conducted as
planned
Limited evidence
base for economic
modelling
The lack of sufficient
diagnostic indices and
heterogeneity across
studies (e.g., clinical
criteria, version of
tests used).
It was not possible to
provide pooled
estimates.
Recommendation
suggested
100%
90%
Sharma 2012 (our review)
80%
Input from scientific
experts versed in the
technology should be
sought from the design
phase.
While considering
reference genetic tests,
the evolution of gene
discovery over time,
reflecting developments
in genetics, should also
be taken into account.
Gerhardus 2007
Bryant 2008
70%
60%
50%
40%
30%
20%
Specificity was
assumed to be 100%
(no false positives).
However, these studies
were judged to be
affected by partial and
differential verification
biases.
10%
Diagnostic measures
may not be adequately
reported and metaanalysis may not be
possible or appropriate.
An alternative analysis
plan should be
Forest plots of
considered in advance
sensitivity and
should this eventuality
specificity were used to occur.
visualise the
heterogeneity amongst
the included studies.
Sensitivity was taken
from a single study and
100% specificity was
assumed.
Figure 2: Diagnostic indices and variability of these results as reported by
the studies included in three reviews
Sensitivity analyses
should explore the
impact of study choices
and assumptions on
cost-effectiveness
results.
Decisions addressing these issues were made in consultation with our project expert advisory
panel.
0%
Sensitivity
Specificity
Positive predictive
value
Negative predictive
value
95% Confidence
interval
CONCLUSIONS
• Genetic tests are continuing to evolve, reviewers should obtain input from
scientific/technical experts from the study design phase and undertake intensive
scoping of the review question.
• The paucity of well designed studies included in the three genetic test reviews made
a valid assessment of the tests difficult. Study investigators should design studies to
allow calculation of specificity as well as sensitivity where possible.
REFERENCES
1. Sharma P et al. Elucigene FH20 and LIPOchip for the diagnosis of familial hypercholesterolaemia:
a systematic review and economic evaluation. Health Technol Assess 2012;16(17).
2. Bryant J et al. A systematic review of the clinical validity and clinical utility of DNA testing for
hereditary haemochromatosis type 1 in at-risk populations. J Med Genet 2008;45:513-518.
3. Gerhardus A et al. Diagnostic accuracy of methods for the detection of BRCA1 and BRCA2
mutations: a systematic review. European Journal of Human genetics, 2007;15:619-627.
ACKNOWLEDGMENTS
RESULTS - COMPARATIVE ANALYSIS
Two additional systematic reviews2,3 of the diagnostic accuracy of genetic tests were
selected for comparison. Six major methodological challenges were identified from
the three reviews (Figure 1). A quantitative assessment of clinical validity is presented
in Figure 2.
The Health Services Research Unit is core funded by the Chief Scientist Office of the Scottish Government Health and
Social Care Directorates. The systematic review of Elucigene FH20 and LIPOchip was funded by the HTA programme
(project number 10/70/01) and published in Health Technol. Assess. 2012;16(17). Further information available at:
http://www.hta.ac.uk/project/2450.asp. The views and opinions expressed are those of the authors and do not
necessarily reflect those of the Chief Scientist Office, HTA programme, NIHR, NICE, NHS or the Department of Health. We
thank the NICE assessment subgroup specialist members and Lara Kemp, Cynthia Fraser and Graeme MacLennan for their
support.
Website: http://www.abdn.ac.uk/hsru