SIGNAL-DETECTION THEORY AND RECEIVER
OPERATING CHARACTERISTIC (ROC) ANALYSIS
Psych 218
(Week 1)
“In the early 1950s, a well-established mathematical theory of statistical decision
was used by electrical engineers as the basis for a theory of an ideal detector—that
is, a machine that would yield the best possible performance at detecting faint
signals in communication networks (W. W. Peterson & Birdsall, 1953).”
“Soon after success of this effort had been demonstrated, an engineer, Wilson
P. Tanner, Jr., and a psychologist, John A. Swets, proposed that human
performance in perceiving near-threshold stimuli might be described by the
signal detection (henceforth, SD) model.”
Over ensuing decades, the SD model, with only technical modifications to
accommodate particular applications, has become almost universally
accepted as a theoretical account of decision making in research on
perceptual detection and recognition and in numerous extensions to applied
domains (Swets, 1988; Swets, Dawes, & Monahan, 2000).
This development may well be regarded as the most towering achievement
of basic psychological research of the last half century.
Green & Swets (1966)
Macmillan & Creelman (2005)
Eyewitness Memory and
Wrongful Convictions
• Since the 1990s, DNA testing has overturned 318 wrongful convictions
• Eyewitness misidentifications – which were invariably made with high
confidence in a court of law – played a role in 75% of these cases
• One of the most famous cases involved the misidentification of a man
named Ronald Cotton
The Case of Ronald Cotton
• In 1984, a college student named Jennifer Thompson was raped
• Shortly thereafter, she picked Ronald Cotton out of a photo lineup
• "I was absolutely, positively, without-a-doubt certain he was the man who
raped me when I got on that witness stand and testified against him. And
nobody was going to tell me any different."
• Cotton was sentenced to life in prison plus 54 years, and he served almost 11
years in jail before being exonerated by DNA testing
Eyewitness Identification Procedures
Simultaneous Lineup
Suspect:
Innocent or Guilty?
Fillers:
All are known to be innocent
Eyewitness Identification Procedures
Simultaneous Lineup
Sequential Lineup
Suspect:
Innocent or Guilty?
Lindsay & Wells (1985)
Simultaneous lineup
 Correct ID rate = 0.58
 False ID rate = 0.43
Sequential lineup
 Correct ID rate = 0.50
 False ID rate = 0.17
.58 / .43 = 1.35
Diagnosticity Ratio
.50 / .17 = 2.94
When does signal detection theory apply?
1. There are two true states of the world
• An enemy plane is either present or absent in the sky
• A disease is either present or absent in a patient
• A guilty suspect is either present or absent in a lineup
2. An imperfect diagnostic procedure is used to make a
decision (the target is "present" or "absent")
• An air-defense radar system
• A medical test
• An eyewitness presented with a lineup
2 X 2 Table
Diagnostic
Decision
Present
Absent
Present
Hit
(Correct ID)
Miss
Absent
False Alarm
(False ID)
Correct
Rejection
True
State
2 X 2 Table
Diagnostic
Decision
Present
Absent
Present
Hit
(Correct ID)
Miss
Absent
False Alarm
(False ID)
Correct
Rejection
True
State
2 X 2 Table
Diagnostic
Decision
Present
Absent
Present
Hit
(Correct ID)
Miss
Absent
False Alarm
(False ID)
Correct
Rejection
True
State
2 X 2 Table
Diagnostic
Decision
Present
Absent
Present
Hit
(Correct ID)
Miss
Absent
False Alarm
(False ID)
Correct
Rejection
True
State
2 X 2 Table
Diagnostic
Decision
Present
Absent
Present
Hit
(Correct ID)
Miss
Absent
False Alarm
(False ID)
Correct
Rejection
True
State
2 X 2 Table
Diagnostic
Decision
Present
Absent
Present
Hit
(Correct ID)
Miss
Absent
False Alarm
(False ID)
Correct
Rejection
True
State
2 X 2 Table
Ted Bundy
Diagnostic
Decision
Absent
Present
Present
Hit
(Correct ID)
Miss
Absent
False Alarm
(False ID)
Correct
Rejection
True
State
Innocence Project
Ronald Cotton
2 X 2 Table
Ted Bundy
Diagnostic
Decision
Absent
Present
Present
Hit
(Correct ID)
Miss
Absent
False Alarm
(False ID)
Correct
Rejection
True
State
Innocence Project
Ronald Cotton
2 X 2 Table
Diagnostic
Decision
Present
Absent
Present
Hit
(Correct ID)
Miss
Absent
False Alarm
(False ID)
Correct
Rejection
True
State
Signal Detection Theory
Continuous diagnostic signal
Weak Signal
Strong Signal
Power of
the reflected
radio signal
Blood
Memory
glucose
strength
level
Signal Detection Theory: Response Bias
“the guilty suspect is
probably in the lineup”
Weak Signal
Strong Signal
Signal Detection Theory: Response Bias
“the guilty suspect is
probably in the lineup”
“absent”
“present”
Weak Signal
Liberal response bias:
Identify even if confidence is low
Strong Signal
Signal Detection Theory: Response Bias
“the guilty suspect may or
may not be in the lineup”
Weak Signal
Strong Signal
Signal Detection Theory: Response Bias
“the guilty suspect may or
may not be in the lineup”
“absent”
Weak Signal
“present”
Strong Signal
Neutral response bias:
Identify if confidence is fairly high
Signal Detection Theory: Response Bias
“too many innocent suspects
have been misidentified”
Weak Signal
Strong Signal
Signal Detection Theory: Response Bias
“too many innocent suspects
have been misidentified”
“absent”
Weak Signal
“present”
Strong Signal
Conservative response bias:
Identify only if confidence is very high
Signal Detection Theory: Discriminability
Weak Signal
Strong Signal
Signal Detection Theory: Discriminability
Target Absent
(Innocent suspects)
Weak Signal
Strong Signal
Signal Detection Theory: Discriminability
Target Absent
(Innocent suspects)
Weak Signal
Strong Signal
Signal Detection Theory: Discriminability
Target Absent
(Innocent suspects)
Weak Signal
Target Present
(Guilty suspects)
Strong Signal
Signal Detection Theory: Discriminability
Discriminability
Target Absent
(Innocent suspects)
Weak Signal
The degree to which the
memory signals associated
with innocent and guilty
suspects are separated using a
particular diagnostic procedure
Target Present
(Guilty suspects)
Strong Signal
"Absent"
Liberal
"Present"
Target Absent
Target Present
Strong Signal
Weak Signal
"Absent"
Neutral
Target Absent
Target Present
Weak Signal
"Absent"
Conservative
"Present"
Target Absent
Weak Signal
Strong Signal
"Present"
Target Present
Strong Signal
"Absent"
Liberal: “the guilty
suspect is probably in
the lineup”
"Present"
Target Absent
Target Present
Strong Signal
Weak Signal
"Absent"
Target Absent
"Present"
Target Present
Weak Signal
"Absent"
Target Absent
Weak Signal
Strong Signal
"Present"
Target Present
Strong Signal
"Absent"
Liberal: “the guilty
suspect is probably in
the lineup”
"Present"
Target Absent
Target Present
Strong Signal
Weak Signal
"Absent"
Target Absent
"Present"
Target Present
Weak Signal
"Absent"
Target Absent
Weak Signal
Strong Signal
"Present"
Target Present
Strong Signal
Correct ID Rate = 0.98
"Absent"
Liberal: “the guilty
suspect is probably in
the lineup”
"Present"
Target Absent
Target Present
Correct ID Rate = 0.98
False ID Rate = 0.50
Strong Signal
Weak Signal
"Absent"
Target Absent
"Present"
Target Present
Weak Signal
"Absent"
Target Absent
Weak Signal
Strong Signal
"Present"
Target Present
Strong Signal
"Absent"
"Present"
Target Absent
Target Present
Strong Signal
Weak Signal
"Absent"
Neutral: “the guilty
suspect may or may not be
the lineup”
Target Absent
"Present"
Target Present
Weak Signal
"Absent"
Target Absent
Weak Signal
Strong Signal
"Present"
Target Present
Strong Signal
"Absent"
"Present"
Target Absent
Target Present
Strong Signal
Weak Signal
"Absent"
Neutral: “the guilty
suspect may or may not be
the lineup”
Target Absent
"Present"
Target Present
Weak Signal
"Absent"
Target Absent
Weak Signal
Strong Signal
"Present"
Target Present
Strong Signal
Correct ID Rate = 0.84
"Absent"
"Present"
Target Absent
Target Present
Strong Signal
Weak Signal
"Absent"
Neutral: “the guilty
suspect may or may not be
the lineup”
Target Absent
"Present"
Target Present
Correct ID Rate = 0.84
False ID Rate = 0.16
Weak Signal
"Absent"
Target Absent
Weak Signal
Strong Signal
"Present"
Target Present
Strong Signal
"Absent"
"Present"
Target Absent
Target Present
Strong Signal
Weak Signal
"Absent"
Target Absent
Target Present
Weak Signal
"Absent"
Conservative: “do not make an
ID unless you are certain of
being correct”
"Present"
Target Absent
Weak Signal
Strong Signal
"Present"
Target Present
Strong Signal
"Absent"
"Present"
Target Absent
Target Present
Strong Signal
Weak Signal
"Absent"
Target Absent
Target Present
Weak Signal
"Absent"
Conservative: “do not make an
ID unless you are certain of
being correct”
"Present"
Target Absent
Weak Signal
Strong Signal
"Present"
Target Present
Strong Signal
Correct ID Rate = 0.50
"Absent"
"Present"
Target Absent
Target Present
Strong Signal
Weak Signal
"Absent"
Target Absent
Target Present
Weak Signal
"Absent"
Conservative: “do not make an
ID unless you are certain of
being correct”
"Present"
Target Absent
Strong Signal
"Present"
Target Present
Correct ID Rate = 0.50
False ID Rate = 0.02
Weak Signal
Strong Signal
"Absent"
"Present"
Correct ID Rate = 0.98
Target Absent
Target Present
Weak Signal
False ID Rate = 0.50
Receiver Operating
Characteristic (ROC)
Strong Signal
"Absent"
Target Absent
"Present"
Target Present
Correct ID Rate
1.0
0.8
0.6
Correct ID Rate = 0.84
0.4
False ID Rate = 0.16
Weak Signal
Strong Signal
0.2
0.0
0.0
0.2
0.4
0.6
0.8
False ID Rate
"Absent"
Target Absent
"Present"
Target Present
Correct ID Rate = 0.50
False ID Rate = 0.02
Weak Signal
Strong Signal
1.0
Receiver Operating Characteristic Analysis
Target Absent
Weak Signal
Target Present
Correct ID Rate
1.0
0.8
High
discriminability
0.6
0.4
0.2
Strong Signal
0.0
0.0
0.2
0.4
0.6
0.8
1.0
False ID Rate
Target Absent
Target Present
Weak Signal
Strong Signal
Correct ID Rate
1.0
0.8
Low
discriminability
0.6
0.4
0.2
0.0
0.0
0.2
0.4
0.6
0.8
False ID Rate
1.0