advertisement

Sensitivity, Specificity and ROC Curve Analysis Criteria for Evaluating a Screening Test •Validity: provide a good indication of who does and does not have disease -Sensitivity of the test -Specificity of the test •Reliability: (precision): gives consistent results when given to same person under the same conditions •Yield: Amount of disease detected in the population, relative to the effort -Prevalence of disease/predictive value Validity of Screening Test (Accuracy) - Sensitivity: Is the test detecting true cases of disease? Ideal is 100%: 100% of cases are detected; =Pr(T+|D+) -Specificity: Is the test excluding those without disease? Ideal is 100%: 100% of non-cases are negative; =Pr(T-|D-) - See Gehlbach, Chp. 10 Example: Screening for Glaucoma using IOP True Cases of Glaucoma IOP > 22: Yes No Yes 50 100 No 50 1900 (total) 100 2000 Sensitivity = 50% (50/100) False Negative=50% Specificity = 95% (1900/2000) False Positive=5% Where do we set the cut-off for a screening test? Consider: -The impact of high number of false positives: anxiety, cost of further testing -Importance of not missing a case: seriousness of disease, likelihood of re-screening Yield from the Screening Test: Predictive Value •Relationship between Sensitivity, Specificity, and Prevalence of Disease Prevalence is low, even a highly specific test will give large numbers of False Positives •Predictive Value of a Positive Test (PPV): Likelihood that a person with a positive test has the disease •Predictive Value of a Negative Test (NPV): Likelihood that a person with a negative test does not have the disease Screening for Glaucoma using IOP True Cases of Glaucoma IOP > 22: Yes No Yes 50 100 No 50 1900 (total) 100 2000 Specificity = 95% (1900/2000) False Positive=5% Positive Predictive Value =33% (50/150) How Good does a Screening Test have to be? IT DEPENDS -Seriousness of disease, consequences of high false positivity rate: -Rapid HIV test should have >90% sensitivity, 99.9% specificity -Screen for nearsighted children proposes 80% sensitivity, >95% specificity -Pre-natal genetic questionnaire could be 99% sensitive, 80% specific Choosing a cut-point: receiver operating characteristic curves • Situation where screening test yields results as a continuous value (e.g., intraocular pressure for glaucoma) • Want to select a value above (or below) which to call “diseased” or “at risk” • How do we select that value? Non-diseased cases Diseased cases Threshold Test result value or subjective judgment of likelihood that case is diseased More typically: Non-diseased cases Diseased cases Test result value or subjective judgment of likelihood that case is diseased 12 Threshold Diseased cases TP Fraction (sensitivity) Non-diseased cases less aggressive mindset FP Fraction (1-specificity) Threshold Diseased cases TP Fraction (sensitivity) Non-diseased cases moderate mindset FP Fraction (1-specificity) Threshold Diseased cases TP Fraction (sensitivity) Non-diseased cases more aggressive mindset FP Fraction (1-specificity) Non-diseased cases Threshold Diseased cases TP Fraction (sensitivity) Entire ROC curve FP Fraction (1-specificity) Highly discriminate (good) Somewhat discriminate (not as good) TP Fraction (sensitivity) Entire ROC curve Reader Skill and/or Level of Technology FP Fraction (1-specificity) Non-informative (no better than chance) Use area under to curve (AUC) to judge discriminating ability. Gehlbach: want AUC>80% Luke Neff: Refractory Burn Shock Data Logistic Regression and ROC Curve Analysis Response Profile Ordered Value PET Total Frequency 1 0 22 2 1 20 Testing Global Null Hypothesis: BETA=0 Test Chi-Square DF Pr > ChiSq Likelihood Ratio 20.2651 1 <.0001 Score 15.3270 1 <.0001 Wald 10.1930 1 0.0014 Luke Neff: Refractory Burn Shock Data Logistic Regression and ROC Curve Analysis Analysis of Maximum Likelihood Estimates Parameter DF Estimate Standard Error Wald Pr > ChiSq Chi-Square Intercept 1 -3.0649 0.9514 10.3771 0.0013 Admission Lactate 1 0.8436 0.2642 10.1930 0.0014 Odds Ratio Estimates Effect Admission Lactate Point Estimate 2.325 95% Wald Confidence Limits 1.385 3.902 Luke Neff: Refractory Burn Shock Data Logistic Regression and ROC Curve Analysis Area 0.8489 Standard Error 0.0633 95% Wald Confidence Limits 0.7249 0.9729 Point that Maximizes sum of sensitivity and specificity. Corresponds to lactate value of about 3.0 Pred Prob True Pos True Neg False Pos 0.9995 1 22 0 0.9863 2 22 0 0.9838 3 22 0 0.96 4 22 0 0.9402 6 22 0 0.9353 7 22 0 0.9182 8 22 0 0.889 9 22 0 0.8401 10 22 0 0.8284 11 22 0 0.7894 12 22 0 0.675 12 21 1 0.637 12 20 2 0.5767 12 18 4 0.5351 13 17 5 0.493 14 17 5 0.4302 14 16 6 0.4096 15 16 6 0.3894 16 16 6 0.3695 17 16 6 0.3312 18 15 7 0.3127 18 14 8 0.2611 18 13 9 0.2299 18 12 10 0.1881 19 10 12 0.1637 19 8 14 0.1525 19 7 15 0.1419 19 5 17 0.1226 19 4 18 0.1056 19 2 20 0.0907 19 1 21 0.0718 20 0 22 False Neg 19 18 17 16 14 13 12 11 10 9 8 8 8 8 7 6 6 5 4 3 2 2 2 2 1 1 1 1 1 1 1 0 Se 0.05 0.1 0.15 0.2 0.3 0.35 0.4 0.45 0.5 0.55 0.6 0.6 0.6 0.6 0.65 0.7 0.7 0.75 0.8 0.85 0.9 0.9 0.9 0.9 0.95 0.95 0.95 0.95 0.95 0.95 0.95 1 1 - Sp 0 0 0 0 0 0 0 0 0 0 0 0.05 0.09 0.18 0.23 0.23 0.27 0.27 0.27 0.27 0.32 0.36 0.41 0.45 0.55 0.64 0.68 0.77 0.82 0.91 0.95 1