Different types of upper limits
in case of a non-detection
Christian Röver1
Chris Messenger1,2
Reinhard Prix1
1Max-Planck-Institut für Gravitationsphysik, Hannover, Germany
2School of Physics & Astronomy, Cardiff University, Cardiff, UK
·· Upper limits
·· The loudest event upper limit
·· Upper limits’ behaviour
Signal detection is commonly based on generalized Neyman-Pearson test statistics, as in the case
of the matched filter. In case of a non-detection,
the detection statistic may be used to derive a loudest-event upper limit on the signal magnitude.
Alternatively, within the Bayesian framework, the
signal amplitude my be bounded via a posterior
upper limit. We use a toy example to investigate
the differences between the two kinds of limits.
Detection significance and upper limit determination are based on the detection statistic’s distribution under H0 and H1, as shown below.
Having the detection statistic’s distribution (under
H0, H1) plus the mapping to upper limits allows to
derive the upper limits’ distributions for different
scenarios.
detection / significance
P(D 2|H0)
99% quantile
p−value: 21.3%
90%
10
10
median
P(D 2|H1, A=0.659)
20
detection statistic D 2
30
40
·· Example problem
Example of detection and 90% upper limit construction (here: detection statistic d2 = 11).
s(t) = A sin(2πf t + φ)
Interpretation of the upper limit then is “Had the
amplitude been A⋆ or larger, a larger detection statistic value would have
with 90% prob been observed
ability (P D ≥ d A ≥ A⋆ ≥ 90%).”
1% quantile
straight line !
0
For illustration, we consider the example of detection of a sinusoidal signal
10% quantile
8
(90%) upper limit
P(D 2|H0)
0
40
6
30
4
20
detection statistic D 2
90% quantile
2
10
90% upper limit’s quantiles
0
posterior limit
frequentist limit
in white Gaussian noise. This example shares many
similarities with commonly encountered problems,
e.g. nuisance parameters, partly analytical, partly
numerical likelihood maximization,...
·· ML detection
A Monte Carlo study shows that the posterior limit
is essentially identical when based on the complete
data y or loudest event d2 only. Assuming a uniform prior, the “loudest event” posterior limit is
then computed based on integration over the likelihood function as shown below.
likelihood function P(D 2|A )
2
4
6
true amplitude
8
N
2σ2
A
The 90% upper limits’ behaviour for different
amplitudes.
Both limits behave the same for high SNR, but very
differently for (the interesting case of) low SNR.
The frequentist upper 90% upper limit is essentially a random variable that has its 10% quantile
at the true value. Consequently, 10% of limits will
be zero under H0 (!).
Bayesian integral
99% quantile
8
posterior limit
frequentist limit
90%
90% quantile
m = 10
m = 1000
m = 1e+05
m = 1e+07
distribution under H0
90%
0
1
P(D2|A)
2
frequentist integral
10
15
20
25
detection statistic D
30
35
5
2
10
20
30
10
20
30
40
6
detection statistic D 2
Integrations for both kinds of limit are very
different.
10% quantile
1e+01
1e+03
1e+06
1e+09
"trials factor" m
Interpretation of the upper limit then is “If there is a
signal, then its amplitude is less than A⋆ with 90%
probability (P(A < A⋆ | y, H1) = 90%).”
·· Comparison
0
90% quantile
median
0
5
median
10% quantile
1% quantile
99% quantile
4
7
3
5 6
N
A
2σ2
2
2 3 4
amplitude
4
1
90% upper limit’s quantiles
0
N
2σ2
amplitude
where ỹ is the DFT of the data y. Under the null
hypothesis H0 of “noise only”, the (normalized)
power in all frequency bins N2σ2 |ỹj |2 follows a χ22distribution. Under the alternative signal hypothesis H1, the power in the (single) “true” frequency
bin is noncentral-χ22-distributed with noncentrality
N A2 (=“SNR”).
parameter λ = 2σ
2
The distribution of the maximum d2 is illustrated
below; it depends on signal amplitude A and the
trials factor m, the number of Fourier frequency
bins considered.
A
5
6
d2 = N2σ2 maxj |ỹj |2
P(D2|A)
7
In case of the sinusoidal signal model, the maximum likelihood detection statistic (matched filter)
is the maximum of the periodogram,
·· The posterior upper limit
0
The 90% upper limits’ behaviour under H0 for
increasing trials factor.
Under H0, the posterior limit is more sensitive, especially for large trials factors.
50
0
10
20
A =5
N
2σ2
30
40
detection statistic D
m = 1e+05
A =6
50
2
40
frequentist limit
posterior limit
distribution under H1 (
5
m = 1e+07
10
10
20
15
20
25
0.5
0.0
2
4
true amplitude
6
N
2σ2
8
A
Different 90% sensitivity limits.
10% quantile
2
P(D |H0)
m = 10
0
P(false alarm)=0.1
P(false alarm)=0.01
P(false alarm)=0.001
observed loudest event D 2
A = 4)
m = 1000
1.0
0
0
N
2σ2
detection probability
N
2σ2
30
A =4
20
N
2σ2
10
A =0
0
distribution under H1 (m = 1000)
90% limit on squared amplitude 2σN2A 2
detection statistic D 2
30
40
50
detection statistic D 2
0
5
10
15
20
25
observed loudest event D 2
Distribution of the detection statistic under Hypotheses H0 and H1.
The mapping from detection statistic to limit.
The closely related sensitivity statement is actually very different, as it is inseparably connected to
the detection problem, based on some false alarm
probability, and is independent of the observed detection statistic value.
LIGO-G1100094