Prognostic and Genetic Tests
Mark Pletcher
6/9/2011
An Example

“Mammaprint”

Gene expression profiling for Breast CA



Grind up the tumor, extract RNA
Incubate with a microarray of DNA fragments
to estimate expression for each gene
70 previously identified genes predict outcomes
Van de Vijver et al. NEJM 2002;347(25):1999-2009
An Example

“Mammaprint”

Pattern of expression correlates with
disease-free and overall survival
Van de Vijver et al. NEJM 2002;347(25):1999-2009
An Example

“Mammaprint”
10-year probability of:
“Good” pattern
“Bad” pattern
Survival
95%
55%
Free of mets
85%
51%
Van de Vijver et al. NEJM 2002;347(25):1999-2009
Outline


Prognostic vs. Diagnostic Tests
Evaluating a Prognostic Test



Accuracy
Utility
Genetic Tests (very briefly)
Prognostic vs. Diagnostic Tests

How is a prognostic test different from
a diagnostic test?
Prognostic vs. Diagnostic Tests
Purpose
Diagnostic Test
Prognostic Test
Identify Prevalent
Disease
Predict Incident
Disease/Outcome
Chance Event
Occurs to
Patient
Prior to Test
Study Design
Cross-Sectional
Maximum
Obtainable
AUROC
1 (gold standard)
After Test
Cohort
<1 (not clairvoyant)
Prognostic vs. Diagnostic Tests

Classic prognosis:

Prediction of death after diagnosis of a
disease
Prognostic vs. Diagnostic Tests

Prognosis, broadly speaking:

Prediction of any future event




Death or recurrence of cancer
Stroke after presentation for TIA
Peri-operative MI in surgical patients
First MI in asymptomatic persons
Prognostic vs. Diagnostic Tests

Prognosis vs. Diagnosis: A Spectrum

Grey areas



Pre-clinical disease: Coronary calcium
“Reversible” disease: Tiny lung CA
Irreversible predisposition: Huntington’s gene
Prognostic vs. Diagnostic Tests

Prognostication ≠ Etiology

Risk factor




Causes the disease
Reducing it may prevent disease
Confounding is crucial issue in observational studies
Risk marker (i.e., prognostic factor)



Predicts the disease
Need not be concerned about unmeasured confounders
Not all risk markers are risk factors…(e.g., CRP)
Evaluating Prognostic Tests

Test Performance






Association
Discrimination
Calibration
Reclassification
Pitfalls
Test Utility
Evaluating Prognostic Tests

Association

Is the marker associated with development
of the disease?


Odds ratio, relative risk, hazard ratio
“Independently associated” means after
adjustment for other known predictors
Evaluating Prognostic Tests


HRadj = 4.6
P<.001
Van de Vijver et al. NEJM 2002;347(25):1999-2009
Evaluating Prognostic Tests

Discrimination


Ability to distinguish between people with
higher or lower risk of disease
Metrics: just like diagnostic tests!?


Sensitivity/specificity
ROC curves
Evaluating Prognostic Tests

Mammaprint
Mets <5yr
Sensitivity = 28/30 = 93%
Specificity = 41/83 = 49%
No mets
Evaluating Prognostic Tests

Coronary artery calcium



Predictor of CHD events
Adds discrimination
AUROC .63.68
FRS = Framingham Risk Score
CACS = Coronary Artery Calcium Score
Greenland et al. JAMA 2004;291(2):210-215
Evaluating Prognostic Tests

Discrimination

Results are specific to a particular time
point


5-year risk of metastases or death
90-day risk of stroke
Evaluating Prognostic Tests

Discrimination
Different results at 5 years….
Evaluating Prognostic Tests

Discrimination
…than at 10 years
Evaluating Prognostic Tests

Discrimination


Often 1 time point is most relevant or
easily communicated, but information is
lost…
Can think of a “set” of discrimination
statistics/ROC curves

Harell’s C-Statistic


Integrated C-statistic for survival data
Similar interpretation as AUROC
Harrell et al. Stat Med 1996;15(4):361-87.
Evaluating Prognostic Tests

Calibration

How close is predicted risk to actual risk?
Evaluating Prognostic Tests

Prognostic test results are often
converted into absolute risk estimates



Like post-test probabilities in diagnosis
Required for clinical interpretation
Estimated directly in a longitudinal study
Evaluating Prognostic Tests

But absolute risk estimates can be “off”


When derivation population different than
target population, etc
Framingham example
D’Agostino et al. JAMA 2001;286(2):180-187
Evaluating Prognostic Tests

Calibration is “orthogonal” to
discrimination



Awful discrimination but good calibration
Awful calibration but good discrimination
Miscalibration leads to worse errors, but
it’s easier to fix…
Evaluating Prognostic Tests

Reclassification

How often does the test lead to
reclassification across a treatment
threshold?


i.e., how often might the test lead to a change
in treatment?
CRP reclassification example
Evaluating Prognostic Tests

Reclassification

How often does the test lead to
reclassification across a treatment
threshold?
Cook et al. Annals of Int Med 2006;145(1):21-29
Evaluating Prognostic Tests

Reclassification metrics

Net Reclassification Improvement (NRI)


Net % reclassified correctly
Depends on specified treatment
thresholds/categories
Evaluating Prognostic Tests

Pitfalls for prognostic test studies

Loss to follow-up and competing risks

Especially problematic if loss is “differential”
Evaluating Prognostic Tests

Pitfalls for prognostic test studies

Bias if clinician knows the test result

e.g. – persons with coronary calcium+ are:


More likely to get revascularization
More likely to get referred to ED if they have chest
pain
Evaluating Prognostic Tests

Pitfalls for prognostic test studies

Overfitting


Test will perform best in sample from which it is
derived
More variables and “choices”  more danger of
overfitting

Gene expression arrays, proteomics
Evaluating Prognostic Tests

Clinical Utility

Does it improve health?
Evaluating Prognostic Tests
Better patient
understanding
of disease/risk
Test Result
Healthier patient
behaviors
1
2
Better health
3
Better clinical
decisions
Pletcher et al. Circulation 2011;123;1116-1124
Evaluating Prognostic Tests

Clinical Utility


Cannot be estimated from test
performance metrics alone
Need to understand downstream
consequences, including




Benefits and harms of interventions based on
test result
Harms from test itself
Quality and length of life
Costs
Evaluating Prognostic Tests

Clinical Utility

Can be estimated directly…


…or indirectly


Randomized trial of test-and-treat strategy
Decision analysis/cost-effectiveness modeling
Same issues for diagnostic tests, and
especially important when screening
apparently healthy people…
Pletcher et al. Circulation 2011;123;1116-1124
Genetic Tests



Potentially useful for mechanistic insight
Prognostic implications across
individuals in a family
Otherwise, must meet same standards
for prognostic utility as other tests

Single gene studies often disappointing
Key concepts




For prognostic tests, an element of time and
chance remain (perfect test impossible)
Discrimination vs. Calibration
Reclassification indices help us understand
how often a test might change management
Clinical utility depends on accounting for net
benefits and harms (and costs)