Pitfalls in Companion Diagnostics
Don't underestimate the power of conditional
probabilities
© 2014 SynteractHCR. All rights reserved.
SHARED WORK. SHARED VISION.
2
A mystery in numbers and its solution
In this presentation, Dr. Stephan de la Motte,
Chief Medical Advisor, defines:
The conditional nature of diagnostics
Companion diagnostics as an entire therapeutic strategy
Sensitivity and specificity with regard to diagnoses
The power of conditional probabilities
How to determine the best hypothesis in study protocols
when you have a companion diagnostic
© 2014 SynteractHCR. All rights reserved.
SHARED WORK. SHARED VISION.
3
Diagnostics is conditional
A biomarker predicts...
suspicion
– if used in clinically healthy
diagnosis
– if disease symptoms are given
prognosis
– if diagnosis is given
response
– if treatment is given
A treatment produces...
response (with a probability) in the right patient
side-effects (with a probability) in any patient
A diagnostic assay leads – in the end – to a therapy outcome.
© 2014 SynteractHCR. All rights reserved.
SHARED WORK. SHARED VISION.
4
Biomarker X scenario
Biomarker X is target for new drug, but does not influence
response to standard of care nor normal course of disease
– E.g., tumor-specific kinase inhibited specifically by new drug
Biomarker X is present in 5% of patients with the disease
– Companion diagnostic prevents many patients being treated with
new drug unnecessarily
Assay has 97% sensitivity and 98% specificity
– Good quality assay
New drug in patients with marker X  80% responders
Standard of care  10% responders
– Huge advantage of new drug over standard of care
This is an ideal scenario for Companion Diagnostics, right?
© 2014 SynteractHCR. All rights reserved.
SHARED WORK. SHARED VISION.
5
An ideal scenario, right?
WRONG!
Because...
80% is the true effectiveness of the new drug
60% is the observed responder rate in a clinical trial
The clinical trial...
significantly underestimates the true value of the new drug
leads to more than twice as many patients getting the new
drug unnecessarily
WHY?
© 2014 SynteractHCR. All rights reserved.
SHARED WORK. SHARED VISION.
6
A word of caution...
"Companion Diagnostics" is...
not only a diagnostic
not only a drug
it is an entire therapeutic strategy!
The next slide shows this strategy of the Biomarker X
scenario as a tree of consecutive, conditional probabilities.
© 2014 SynteractHCR. All rights reserved.
SHARED WORK. SHARED VISION.
7
Decisional algorithm tree
Assay outcome
Biomarker
(invisible)
Yes
Yes
0.97
Assay pos.,
new drug?
No
Therapy outcome
Response to
new drug?
0.03
Effect of
standard?
0.05
Biomarker X
really present?
0.95
Yes
No
0.02
Assay pos.,
new drug?
No
Response to
new drug?
0.98
Effect of
standard?
Yes 0.80
Responder
No 0.20
Non-responder ~ 1%
Yes 0.10
Effect
No 0.90
Normal course ~ 0.1%
Yes 0.10
Responder
No 0.90
Non-responder ~ 2%
Yes 0.10
Effect
No 0.90
Normal course ~ 84%
Patients misallocated:
Not to be treated with new drug, but are, or
to be treated with new drug, but should not be.
© 2014 SynteractHCR. All rights reserved.
Apparent interpretation
SHARED WORK. SHARED VISION.
~ 4%
< 0.1%
~ 0.2%
~ 9%
Patients actually
treated with new drug
8
What we see is not what is real!
We never see the real target
We see only
... the result of an assay
... the outcome of a treatment
𝐴𝑝𝑝𝑎𝑟𝑒𝑛𝑡 𝑟𝑒𝑠𝑝𝑜𝑛𝑑𝑒𝑟𝑠 =
𝑃𝑎𝑡𝑖𝑒𝑛𝑡𝑠 𝑟𝑒𝑠𝑝𝑜𝑛𝑑𝑖𝑛𝑔 𝑡𝑜 𝑡ℎ𝑒 𝑛𝑒𝑤 𝑑𝑟𝑢𝑔
𝐴𝑙𝑙 𝑝𝑎𝑡𝑖𝑒𝑛𝑡𝑠 𝑤ℎ𝑜 𝑔𝑒𝑡 𝑡ℎ𝑒 𝑛𝑒𝑤 𝑑𝑟𝑢𝑔
𝟒% + 𝟎. 𝟐%
=
= 𝟎. 𝟔𝟎
𝟒% + 𝟏% + 𝟎. 𝟐% + 𝟐%
(Percent numbers taken from previous slide #7, rightmost column)
© 2014 SynteractHCR. All rights reserved.
SHARED WORK. SHARED VISION.
9
Sensitivity and Specificity
Don't overlook a diagnosis!
𝑆𝑒𝑛𝑠𝑖𝑡𝑖𝑣𝑖𝑡𝑦 =
𝑃𝑒𝑜𝑝𝑙𝑒 𝑑𝑖𝑎𝑔𝑛𝑜𝑠𝑒𝑑 𝑐𝑜𝑟𝑟𝑒𝑐𝑡𝑙𝑦
𝐴𝑙𝑙 𝑝𝑒𝑜𝑝𝑙𝑒 ℎ𝑎𝑣𝑖𝑛𝑔 𝑡ℎ𝑒 𝑑𝑖𝑠𝑒𝑎𝑠𝑒
Don't make a healthy person sick!
𝑆𝑝𝑒𝑐𝑖𝑓𝑖𝑐𝑖𝑡𝑦 =
𝑃𝑒𝑜𝑝𝑙𝑒 𝑑𝑒𝑐𝑙𝑎𝑟𝑒𝑑 𝑡𝑜 𝑏𝑒 ℎ𝑒𝑎𝑙𝑡ℎ𝑦 𝑐𝑜𝑟𝑟𝑒𝑐𝑡𝑙𝑦
𝐴𝑙𝑙 ℎ𝑒𝑎𝑙𝑡ℎ𝑦 𝑝𝑒𝑜𝑝𝑙𝑒
Sensitivity and specificity indicate the quality of a
diagnostic test and both should be close to 100%
But are these criteria – alone – meaningful?
© 2014 SynteractHCR. All rights reserved.
SHARED WORK. SHARED VISION.
10
The “100% Assay”
"All of you have cancer!"
Everyone who has cancer is diagnosed to have cancer
=> 100% sensitivity
– No disease is overlooked, because no one is declared as healthy
"None of you have cancer!"
Everyone who is healthy is declared to be healthy
=> 100% specificity
– No false diagnosis, because no diagnosis is made at all
Yes, this is nonsense, because the value of a diagnostic
assay depends on much more than sensitivity or specificity.
© 2014 SynteractHCR. All rights reserved.
SHARED WORK. SHARED VISION.
11
Predictive values
Predictive value - If we make an observation, how likely is it that it is correct?
Sensitivity and specificity are always given
Predictive values are often overlooked and their usefulness is underestimated
If we see it, is it real? – Important for treatment decisions
Positive predictive value
𝑃𝑃𝑉 =
𝑇𝑟𝑢𝑒 𝑝𝑜𝑠𝑖𝑡𝑖𝑣𝑒𝑠
𝑇𝑒𝑠𝑡 𝑜𝑢𝑡𝑐𝑜𝑚𝑒 𝑝𝑜𝑠𝑖𝑡𝑖𝑣𝑒𝑠
If we don't see it, does this mean it isn't there? – Important for screening exams
Negative predictive value
𝑁𝑃𝑉 =
𝑇𝑟𝑢𝑒 𝑛𝑒𝑔𝑎𝑡𝑖𝑣𝑒𝑠
𝑇𝑒𝑠𝑡 𝑜𝑢𝑡𝑐𝑜𝑚𝑒 𝑛𝑒𝑔𝑎𝑡𝑖𝑣𝑒𝑠
PPV and NPV quantify the reliability of assay results in actual populations.
PPV and NPV depend on prevalence.
© 2014 SynteractHCR. All rights reserved.
SHARED WORK. SHARED VISION.
12
Biomarker X in 5% of patients
Number of
patients
'X' present
5
'X' absent
95
Assay positive
('X' detected)
True positives
4.85
False positives
1.9
PPV
0.72
True negatives
93.1
NPV
0.998
Assay negative
False negatives
('X' not detected)
0.15
Sensitivity
0.97
Specificity
0.98
Purple numbers = Given
All other colors = Derived
Interpretation:
Although NPV is almost perfect, PPV is not satisfactory.
PPV 0.72 means that 28% of patients with a positive assay don't actually have
the Biomarker X target.
Apparent responder rate of patients treated with new drug is 60%.
© 2014 SynteractHCR. All rights reserved.
SHARED WORK. SHARED VISION.
13
Biomarker X in 50% of patients
Number of
patients
'X' present
50
'X' absent
50
Assay positive
('X' detected)
True positives
48.5
False positives
1
PPV
0.98
True negatives
49
NPV
0.97
Assay negative
False negatives
('X' not detected)
1.5
Sensitivity
0.97
Specificity
0.98
Purple numbers = Given
All other colors = Derived
Interpretation:
Properties of assay and of drug unchanged; only prevalence is changed.
Now, both NPV and PPV are almost perfect.
Apparent responder rate is now ~79%, very close to the theoretically best 80%.
© 2014 SynteractHCR. All rights reserved.
SHARED WORK. SHARED VISION.
14
How to make it more complicated
Previous scenarios were based on the simplified assumption
that Biomarker X predicts only the response to a new drug.
In real life, however:
Biomarkers are not only drug targets, but are associated with
a better or worse prognosis, even under standard of care
A new drug is targeting biomarker X, but it may show some
efficacy also in patients who do not carry X
The next slide shows which probabilities must be modified...
to simulate biomarker X prognosis without new drug
to simulate drug efficacy in patients without biomarker X
© 2014 SynteractHCR. All rights reserved.
SHARED WORK. SHARED VISION.
15
Modified decisional algorithm tree
Biomarker
(invisible)
Assay outcome
Yes
Yes
0.97
Assay pos.,
new drug?
Therapy outcome
Response to
new drug?
0.03
No
Effect of
standard?
0.05
Biomarker X
really present?
0.95
Yes
No
0.02
Assay pos.,
new drug?
No
Response to
new drug?
0.98
Effect of
standard?
Biomarker-associated poor prognosis.
New drug with off-target efficacy.
© 2014 SynteractHCR. All rights reserved.
Apparent interpretation
Yes 0.80
Responder
No 0.20
Non-responder ~ 1%
Yes 0.05
Effect
No 0.95
Normal course ~ 0.1%
Yes 0.25
Responder
No 0.75
Non-responder ~ 1%
Yes 0.10
Effect
No 0.90
Normal course ~ 84%
~ 4%
< 0.1%
~ 0.5%
~ 9%
Outcome changed, not always detectable.
(Compare with previous tree on slide #7.)
SHARED WORK. SHARED VISION.
16
What to do if you have a companion
diagnostic...
Check out the prevalence of the biomarker in your target population!
Calculate predictive values!
Work through the tree of conditional probabilites and see if it makes sense!
Calculate different plausible scenarios (sensitivity analysis – "What if?")
If PPV (positive predictive value) is troublesome, utilize a 2nd diagnostic test
...sequentially (2nd only if 1st is positive)
...parallel (outcome is positive only if both are positive)
A 2nd diagnostic should be complementary to the 1st
...one with high PPV
...the other with high NPV
Avoid raising false expectations!
Write the best hypothesis in your study protocol!
© 2014 SynteractHCR. All rights reserved.
SHARED WORK. SHARED VISION.
THANK YOU!
Dr. Stephan de la Motte
Chief Medical Advisor
www.SynteractHCR.com
© 2014 SynteractHCR. All rights reserved.
SHARED WORK. SHARED VISION.
Global Headquarters
5759 Fleet Street, Suite 100
Carlsbad, CA 92008
+1 760 268 8200