Curva ROC
figuras esquemáticas
Prof. Ivan Balducci
FOSJC / Unesp
Receiver Operating Characteristic
(ROC) Curve
DR
1
DR0
c
c  co
c
Each threshold c corresponds to a
point (FPR, DR) on the X-Y plane.
A ROC curve is obtained as c
sweeps from -  to + .
0
FPR0
FPR
1
2
Lab Tests: What is
“Abnormal”?
Desempenho
True
Positive
True
Negative
B
False
Negative
A
False
Positive
TP – Classe é A e classificamos como A
TN – Classe é B e classificamos como B
FP – Classe é B e classificamos como A
FN – Classe é A e classificamos como B
The Cut-off Value Trade off
• Sensitivity and specificity depend on the cut off
value between what we define as normal and
abnormal
• Assume high test values are abnormal; then,
moving the cut-off value to a higher one
increases FN results and decreases FP results
(i.e. more specific) and vice versa
• There is always a trade off in setting the cut-off
point
Receiver Operating Characteristic
(ROC) Curve
DR
1
DR0
c
c  co
c
Each threshold c corresponds to a
point (FPR, DR) on the X-Y plane.
A ROC curve is obtained as c
sweeps from -  to + .
0
FPR0
FPR
1
6
ROC Curve
Receiver Operating Characteristic (ROC)
Curves
Goodness-Of-Fit: Other Measures of Model
Performance
• ROC (Receiver Operating Characteristic) Curve
• Sensitivity and Specificity are dependent on a given cut-point c.
• An ROC curve is obtained by plotting sensitivity against (1specificity) for an entire range of possible cut-points.
• The area under the ROC curve is a measure of the model’s
ability to discriminate between event and non-event in the
following fashion:
» Among all possible pairs (event, non-event), the proportion of pairs
for which the event has higher probability than the corresponding
non-event is equal to the area under ROC.
ROC Curve
• Interpretation: Area Under ROC Curve
– If randomly selected pairs of subjects (one with event and one
with non-event) are classified in such a way that the subject with
higher estimated probability of the event belongs to the event
group and the other subject to non-event group, then the
proportion of correctly classified such pairs of subjects would be
equal to the area under ROC
• Generally Accepted Rule:
• ROC = 0.5:
no discrimination (no better than coin toss)
• 0.7 <= ROC < 0.8: acceptable discrimination
• 0.8 <= ROC < 0.9: excellent discrimination
• ROC > 0.9:
outstanding discrimination
• Roc area is often used to compare predictive ability of
different models;
The ROC
H1
PTP
PTP
H0
PFP
1
Decision threshold
• The ROC shows
the tradeoff
between PFP and
PTP as the threshold
is varied
AZ
PFP
0
0
1
Developmental characteristics: Cut-points and
Receiver Operating Characteristic (ROC)
Healthy
Developmental characteristics: Cut-points and Receiver
Operating Characteristic (ROC)
Healthy
Sick
Developmental characteristics: Cut-points and Receiver
Operating Characteristic (ROC)
Fals pos= 20% True pos=82%
Developmental characteristics: Cut-points and Receiver
Operating Characteristic (ROC)
Fals pos= 9% True pos=70%
Developmental characteristics: Cut-points and
Receiver Operating Characteristic (ROC)
Healthy
Developmental characteristics: Cut-points and Receiver
Operating Characteristic (ROC)
Healthy
Sick
Developmental characteristics: Cut-points and Receiver
Operating Characteristic (ROC)
Fals pos= 20% True pos=82%
Developmental characteristics: Cut-points and Receiver
Operating Characteristic (ROC)
Fals pos= 9% True pos=70%
Evaluating the results
1
How can we measure
the performance of a feature
matcher?
0.7
true
# true positives
# matching features (positives) positive
rate
0
0.1
false positive rate
# false positives
# unmatched features (negatives)
1
ROC curve
(“Receiver Operator Characteristic”)
How can we measure the performance of a feature
matcher?
1
0.7
true
# true positives
# matching features (positives) positive
rate
0
0.1
false positive rate
1
# false positives
# unmatched features (negatives)
ROC Curves
•
•
•
•
Generated by counting # current/incorrect matches, for different threholds
Want to maximize area under the curve (AUC)
Useful for comparing different feature matching methods
For more info: http://en.wikipedia.org/wiki/Receiver_operating_characteristic
Curva ROC
Cut-off
especificidade
Área sob curva
sensibiidade