10 / 31 Outline
• Perception workshop groups
• Signal detection theory
• Scheduling meetings
Detection experiment
• Question
– How sensitive is an observer to a sensory
stimulus; for example, light?
Detection experiment
• Question
– How sensitive is an observer to (for example)
light?
• Classic experiment
– Yes/No task
Detection experiment
• Question
– How sensitive is an observer to (for example)
light?
• Classic experiment
– Yes/No task
– Measure threshold intensity needed to have
50% hits
1
0.9
0.8
P ropo rtion "Y es " respo nse s
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
0
10
20
30
Intensity
40
50
60
1
0.9
0.8
P ropo rtion "Y es " respo nse s
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
0
5
10
15
20
25
Intensity
30
35
40
45
50
1
0.9
0.8
P ropo rtion "Y es " respo nse s
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
0
5
10
15
20
25
Intensity
30
35
40
45
50
1
0.9
0.8
P ropo rtion "Y es " respo nse s
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
0
5
10
15
20
25
Intensity
30
35
40
45
50
1
0.9
0.8
P ropo rtion "Y es " respo nse s
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
0
5
10
15
20
25
Intensity
30
35
40
45
50
1
0.9
0.8
P ropo rtion "Y es " respo nse s
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
0
5
10
15
20
25
Intensity
30
35
40
45
50
1
0.9
0.8
P ropo rtion "Y es " respo nse s
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
0
5
10
15
20
25
Intensity
30
35
40
45
50
1
0.9
0.8
P ropo rtion "Y es " respo nse s
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
0
5
10
15
20
25
Intensity
30
35
40
45
50
1
0.9
0.8
P ropo rtion "Y es " respo nse s
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
0
5
10
15
20
25
Intensity
30
35
40
45
50
1
0.9
0.8
P ropo rtion "Y es " respo nse s
0.7
0.6
0.5
Threshold
0.4
0.3
0.2
0.1
0
0
5
10
15
20
25
Intensity
30
35
40
45
50
1
0.9
0.8
P ropo rtion "Y es " respo nse s
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
0
5
10
15
20
25
Intensity
30
35
40
45
50
1
0.9
0.8
P ropo rtion "Y es " respo nse s
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
0
5
10
15
20
25
Intensity
30
35
40
45
50
1
0.9
0.8
P ropo rtion "Y es " respo nse s
0.7
0.6
0.5
Threshold
0.4
0.3
0.2
0.1
0
0
5
10
15
20
25
Intensity
30
35
40
45
50
1
0.9
0.8
P ropo rtion "Y es " respo nse s
0.7
Jane
Nancy
0.6
0.5
0.4
0.3
0.2
0.1
0
0
5
10
15
20
25
Intensity
30
35
40
45
50
Summary of results
• Thresholds
– Jane = 20
– Nancy = 25
Summary of results
• Thresholds
– Jane = 20
– Nancy = 25
• False alarm rates
– Jane = 51%
– Nancy = 18.7%
Look at one intensity level
• I = 25
1
0.9
0.8
P ropo rtion "Y es " respo nse s
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
0
5
10
15
20
25
Intensity
30
35
40
45
50
1
0.9
0.8
P ropo rtion "Y es " respo nse s
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
0
5
10
15
20
25
Intensity
30
35
40
45
50
Jane’s Hit Rate
1
P(H) = .84
0.9
0.8
P ropo rtion "Y es " respo nse s
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
0
5
10
15
20
25
Intensity
30
35
40
45
50
1
0.9
0.8
Nancy’s Hit Rate
P(H) = .5
P ropo rtion "Y es " respo nse s
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
0
5
10
15
20
25
Intensity
30
35
40
45
50
Look at one intensity level
• I = 25
– Jane
• Hit rate: P(H) = .84
Look at one intensity level
• I = 25
– Jane
• Hit rate: P(H) = .84
• False alarm rate: P(FA) = .51
Look at one intensity level
• I = 25
– Jane
• Hit rate: P(H) = .84
• False alarm rate: P(FA) = .51
– Nancy
• Hit rate: P(H) = .5
Look at one intensity level
• I = 25
– Jane
• Hit rate: P(H) = .84
• False alarm rate: P(FA) = .51
– Nancy
• Hit rate: P(H) = .5
• False alarm rate: P(FA) = .187
Signal detection theory terms
• Hits - p(H)
– Proportion of “yes” responses when signal is present
Signal detection theory terms
• Hits - p(H)
– Proportion of “yes” responses when signal is present
• Misses - p(M)
– Proportion of “no” responses when signal is present
Signal detection theory terms
• Hits - p(H)
– Proportion of “yes” responses when signal is present
• Misses - p(M)
– Proportion of “no” responses when signal is present
• False alarms - p(FA)
– Proportion of “yes” responses when signal is not present
Signal detection theory terms
• Hits - p(H)
– Proportion of “yes” responses when signal is present
• Misses - p(M)
– Proportion of “no” responses when signal is present
• False alarms - p(FA)
– Proportion of “yes” responses when signal is not present
• Correct rejections - p(CR)
– Proportion of “no” responses when signal is not present
Relationships between terms
• P(H) + P(M) = 1
Relationships between terms
• P(H) + P(M) = 1
• P(FA) + P(CR) = 1
Relationships between terms
• P(H) + P(M) = 1
• P(FA) + P(CR) = 1
• Only need to specify P(H) and P(FA)
Extreme detection strategies
• Most liberal (always say yes)
Extreme detection strategies
• Most liberal (always say yes)
– P(H) = 1, P(FA) = 1
Extreme detection strategies
• Most liberal (always say yes)
– P(H) = 1, P(FA) = 1
• Most conservative (always say no)
Extreme detection strategies
• Most liberal (always say yes)
– P(H) = 1, P(FA) = 1
• Most conservative (always say no)
– P(H) = 0, P(FA) = 0
Signal Detection Theory
Signal Detection Theory
• Assume an internal measure of signal
strength.
Signal Detection Theory
• Assume an internal measure of signal
strength (X).
– E.g. firing rate of ganglion cell
Signal Detection Theory
• Assume an internal measure of signal
strength (X).
– E.g. firing rate of ganglion cell
• X is corrupted by noise
Signal Detection Theory
• Assume an internal measure of signal
strength (X).
– E.g. firing rate of ganglion cell
• X is corrupted by noise
– E.g. random variations in firing rate
Signal Detection Theory
• Assume an internal measure of signal
strength (X).
– E.g. firing rate of ganglion cell
• X is corrupted by noise
– E.g. random variations in firing rate
• When signal is not present, X = X0 + N
40
35
30
firi ng ra te
25
20
15
10
5
0
0
10
20
30
40
50
trial number
60
70
80
90
100
Signal Detection Theory
• Assume an internal measure of signal
strength (X).
– E.g. firing rate of ganglion cell
• X is corrupted by noise
– E.g. random variations in firing rate
• When signal is not present, X = X0 + N
• When signal is present, X = XS + N
40
35
30
firi ng ra te
25
20
15
10
5
0
0
10
20
30
40
50
trial number
60
70
80
90
100
o Firing rate when signal is present
o Firing rate when signal is not present
40
35
30
firi ng ra te
25
20
15
10
5
0
0
10
20
30
40
50
trial number
60
70
80
90
100
Criterion
• Set a criterion level, C
Criterion
• Set a criterion level, C
• If X > C
– Report a signal
Criterion
• Set a criterion level, C
• If X > C
– Report a signal
• If X < C
– Report no signal
o Firing rate when signal is present
o Firing rate when signal is not present
40
35
30
firi ng ra te
25
20
15
10
C=20, Liberal criterion
5
0
0
10
20
30
40
50
trial number
60
70
80
90
100
Liberal criterion = High hit rate
40
35
30
firi ng ra te
25
20
15
10
5
0
0
10
20
30
40
50
trial number
60
70
80
90
100
Liberal criterion = High false alarm rate
40
35
30
firi ng ra te
25
20
15
10
5
0
0
10
20
30
40
50
trial number
60
70
80
90
100
o Firing rate when signal is present
o Firing rate when signal is not present
40
C=30, Conservative criterion
35
30
firi ng ra te
25
20
15
10
5
0
0
10
20
30
40
50
trial number
60
70
80
90
100
Conservative criterion = Low hit rate
40
35
30
firi ng ra te
25
20
15
10
5
0
0
10
20
30
40
50
trial number
60
70
80
90
100
Conservative criterion = Low false alarm rate
40
35
30
firi ng ra te
25
20
15
10
5
0
0
10
20
30
40
50
trial number
60
70
80
90
100
Probability distribution on X (no signal)
0.14
0.12
0.1
pro bab ility
0.08
0.06
0.04
0.02
0
-5
0
5
10
15
20
X
25
30
35
40
45
Probability distribution on X (signal)
0.14
0.12
0.1
0.08
0.06
0.04
0.02
0
-5
0
5
10
15
20
25
30
35
40
45
Liberal criterion
0.14
0.12
0.1
pro bab ility
0.08
0.06
0.04
0.02
0
-5
0
5
10
15
20
X
25
30
0
5
10
15
20
25
30
35
40
45
0.14
0.12
0.1
0.08
0.06
0.04
0.02
0
-5
35
40
45
Conservative criterion
0.14
0.12
0.1
pro bab ility
0.08
0.06
0.04
0.02
0
-5
0
5
10
15
20
X
25
30
0
5
10
15
20
25
30
35
40
45
0.14
0.12
0.1
0.08
0.06
0.04
0.02
0
-5
35
40
45
ROC curve
0.15
1
0.1
H its
P rob ab ility
0.8
0.6
0.4
0.05
0.2
0
0
0
10
20
30
40
Signal Strength
P rob ab ility
0.1
0.05
0
10
20
Signal Strength
0.2
0.4
0.6
False alarms
0.15
0
0
30
40
0.8
1
ROC curve
0.15
1
0.1
H its
P rob ab ility
0.8
0.6
0.4
0.05
0.2
0
0
0
10
20
30
40
Signal Strength
P rob ab ility
0.1
0.05
0
10
20
Signal Strength
0.2
0.4
0.6
False alarms
0.15
0
0
30
40
0.8
1
ROC curve
0.15
1
0.1
H its
P rob ab ility
0.8
0.6
0.4
0.05
0.2
0
0
0
10
20
30
40
Signal Strength
P rob ab ility
0.1
0.05
0
10
20
Signal Strength
0.2
0.4
0.6
False alarms
0.15
0
0
30
40
0.8
1
ROC curve
0.15
1
0.1
H its
P rob ab ility
0.8
0.6
0.4
0.05
0.2
0
0
0
10
20
30
40
Signal Strength
P rob ab ility
0.1
0.05
0
10
20
Signal Strength
0.2
0.4
0.6
False alarms
0.15
0
0
30
40
0.8
1
ROC curve
0.15
1
0.1
H its
P rob ab ility
0.8
0.6
0.4
0.05
0.2
0
0
0
10
20
30
40
Signal Strength
P rob ab ility
0.1
0.05
0
10
20
Signal Strength
0.2
0.4
0.6
False alarms
0.15
0
0
30
40
0.8
1
ROC curve
0.15
1
0.1
H its
P rob ab ility
0.8
0.6
0.4
0.05
0.2
0
0
0
10
20
30
40
Signal Strength
P rob ab ility
0.1
0.05
0
10
20
Signal Strength
0.2
0.4
0.6
False alarms
0.15
0
0
30
40
0.8
1
0.1
0.05
0
P ro b ab il i ty
P ro b ab ilit y
P ro b ab il i t y
0.15
0.15
0.15
No
signal
0.1
0.05
10
20
30
40
0
10
Signal Strength
30
0
40
0.1
0.05
0
20
Signal Strength
10
30
40
20
30
40
30
40
Signal Strength
0.15
P ro b ab il i ty
P ro b ab ilit y
P ro b ab il i t y
20
0.15
10
0.05
Signal Strength
0.15
0
0.1
0
0
0
Signal
C
B
A
0.1
0.05
0.1
0.05
0
0
0
10
20
Signal Strength
30
40
0
10
20
Signal Strength
1
0.9
0.8
A
0.7
B
H its
0.6
C
0.5
0.4
0.3
0.2
0.1
0
0
0.1
0.2
0.3
0.4
0.5
False alarms
0.6
0.7
0.8
0.9
1
1
C
0.9
0.8
B
0.7
H its
0.6
0.5
A
0.4
0.3
0.2
0.1
0
0
0.1
0.2
0.3
0.4
0.5
False alarms
0.6
0.7
0.8
0.9
1
0.2
0.2
0.15
0.15
0.15
0.1
0.05
0
10
20
30
P ro b ab il i t y
0.2
0
0.1
0.05
0
0.05
10
20
30
0
40
0.2
0.15
0.15
0.15
0.05
0
10
20
Signal Strength
30
40
P ro b ab il i t y
0.2
0.1
20
30
40
30
40
Signal Strength
0.2
0
10
Signal Strength
P ro b ab il i ty
P ro b ab il i ty
0.1
0
0
40
Signal Strength
Signal
C
B
P ro b ab il i ty
No
signal
P ro b ab il i ty
A
0.1
0.05
0
0.1
0.05
0
0
10
20
Signal Strength
30
40
0
10
20
Signal Strength
1
0.9
0.8
A
0.7
B
H its
0.6
C
0.5
0.4
0.3
0.2
0.1
0
0
0.1
0.2
0.3
0.4
0.5
False alarms
0.6
0.7
0.8
0.9
1
1
0.9
A
0.8
0.7
B
H its
0.6
C
0.5
0.4
0.3
0.2
0.1
0
0
0.1
0.2
0.3
0.4
0.5
False alarms
0.6
0.7
0.8
0.9
1
Determinants of performance
0.2
No
signal
P rob ab ility
0.15
0.1
0.05
0
0
10
20
30
40
30
40
Signal Strength
0.2
Signal
P rob ab ility
0.15
0.1
0.05
0
0
10
20
Signal Strength
Determinants of performance
0.2
No
signal
P rob ab ility
0.15
0.1
0.05
0
0
10
20
30
40
30
40
Signal Strength
XN
0.2
Signal
P rob ab ility
0.15
0.1
0.05
0
0
XS
10
20
Signal Strength
Determinants of performance
∆X
0.2
No
signal
P rob ab ility
0.15
0.1
0.05
0
0
10
20
30
40
30
40
Signal Strength
XN
0.2
Signal
P rob ab ility
0.15
0.1
0.05
0
0
XS
10
20
Signal Strength
Determinants of performance
∆X
1. Difference in average strength of
Signal measure
0.2
No
signal
P rob ab ility
0.15
0.1
∆X = XS - XN
0.05
0
0
10
20
30
40
30
40
Signal Strength
XN
0.2
Signal
P rob ab ility
0.15
0.1
0.05
0
0
XS
10
20
Signal Strength
Determinants of performance
∆X
1. Difference in average strength of
Signal measure
0.2
No
signal
P rob ab ility
0.15
0.1
∆X = XS - XN
0.05
0
0
10
20
30
40
2. Amount of noise
Signal Strength
0.2
s
Signal
P rob ab ility
0.15
s
0.1
0.05
0
0
10
20
Signal Strength
30
40
Determinants of performance
∆X
1. Difference in average strength of
Signal measure
0.2
No
signal
P rob ab ility
0.15
0.1
∆X = XS - XN
0.05
0
0
10
20
30
40
2. Amount of noise
Signal Strength
0.2
s
Signal
P rob ab ility
0.15
s
0.1
0.05
0
0
10
20
30
40
3. Sensitivity
Signal Strength
d’ = ∆X / s
D’ determines which ROC curve your data will fall on
1
0.9
0.8
0.7
H its
0.6
0.5
0.4
0.3
0.2
0.1
0
0
0.1
0.2
0.3
0.4
0.5
False alarms
0.6
0.7
0.8
0.9
1
D’ determines which ROC curve your data will fall on
1
0.9
d’ = 2.5
0.8
d’ = 1.2
0.7
d’ = .83
H its
0.6
0.5
0.4
0.3
0.2
0.1
0
0
0.1
0.2
0.3
0.4
0.5
False alarms
0.6
0.7
0.8
0.9
1
Criterion determines where your data will sit on an ROC curve
1
0.9
0.8
0.7
H its
0.6
0.5
0.4
0.3
0.2
0.1
0
0
0.1
0.2
0.3
0.4
0.5
False alarms
0.6
0.7
0.8
0.9
1
Criterion determines where your data will sit on an ROC curve
1
0.9
0.8
0.7
Liberal criterion
H its
0.6
0.5
0.4
0.3
Conservative criterion
0.2
0.1
0
0
0.1
0.2
0.3
0.4
0.5
False alarms
0.6
0.7
0.8
0.9
1
Measuring sensitivity
Measuring sensitivity
• Pick a stimulus level for a yes / no task
Measuring sensitivity
• Pick a stimulus level for a yes / no task
• Measure hit rate and false alarm rate
Measuring sensitivity
• Pick a stimulus level for a yes / no task
• Measure hit rate and false alarm rate
• Use p(H) and p(FA) to calculate d’
Measuring sensitivity
•
•
•
•
Pick a stimulus level for a yes / no task
Measure hit rate and false alarm rate
Use p(H) and p(FA) to calculate d’
d’ = absolute measure of sensitivity
Blood test example
• Get a blood test for level of protein A.
Blood test example
• Get a blood test for level of protein A.
• Doctor says that test is positive for liver
cancer.
Blood test example
• Get a blood test for level of protein A.
• Doctor says that test is positive for liver
cancer.
• Doctor recommends surgery to collect
tissue sample for biopsy.
Blood test example
• Get a blood test for level of protein A.
• Doctor says that test is positive for liver
cancer.
• Doctor recommends surgery to collect
tissue sample for biopsy.
• What should you ask the doctor about the
blood test?
0.2
P rob ab ility
0.15
No cancer
0.1
0.05
0
0
10
20
30
40
Signal Strength
0.2
P rob ab ility
0.15
Cancer
0.1
0.05
0
0
10
20
Signal Strength
30
40
Liberal criterion
0.2
P rob ab ility
0.15
No cancer
0.1
0.05
0
0
10
20
30
40
Signal Strength
0.2
P rob ab ility
0.15
Cancer
0.1
0.05
0
0
10
20
Signal Strength
30
40
Conservative criterion
0.2
P rob ab ility
0.15
No cancer
0.1
0.05
0
0
10
20
30
40
Signal Strength
0.2
P rob ab ility
0.15
Cancer
0.1
0.05
0
0
10
20
Signal Strength
30
40