Tests
Jean-Yves Le Boudec
Contents
1. The Neyman Pearson framework
2. Likelihood Ratio Tests
3. ANOVA
4. Asymptotic Results
5. Other Tests
1
Tests
Tests are used to give a binary answer to hypotheses of a statistical nature
Ex: is A better than B?
Ex: does this data come from a normal distribution ?
Ex: does factor n influence the result ?
2
Example: Non Paired Data
Is red better than blue ?
For data set (a) answer is clear (by inspection of confidence interval) no test
required
3
Is this data normal ?
4
5.1 The Neyman-Pearson Framework
5
Example: Non Paired Data
 Is red better than blue ?
6
Critical Region, Size and Power
7
Example : Paired Data
8
Power
9
Grey Zone
10
11
p-value of a test
12
13
Tests are just tests
14
Test versus Confidence Intervals
If you can have a confidence interval, use it instead of a test
15
2. Likelihood Ratio Test
A special case of Neyman-Pearson
A Systematic Method to define tests, of general applicability
16
17
18
A Classical Test: Student Test
The model :
The hypotheses :
19
20
21
Here it is the same as a Conf. Interval
22
The “Simple Goodness of Fit” Test
Model
Hypotheses
23
1. compute likelihood ratio statistic
24
2. compute p-value
25
Mendel’s Peas
P= 0.92 ± 0.05 => Accept H0
26
3 ANOVA
Often used as “Magic Tool”
Important to understand the underlying assumptions
Model
Data comes from iid normal sample with unknown means and same variance
Hypotheses
27
28
29
The ANOVA Theorem
We build a likelihood ratio statistic test
The assumption that data is normal and variance is the same allows an
explicit computation
it becomes a least square problem
= a geometrical problem
we need to compute orthogonal projections on M and M0
30
The ANOVA Theorem
31
Geometrical Interpretation
Accept H0 if SS2 is small
The theorem tells us what “small”
means in a statistical sense
32
33
ANOVA Output: Network Monitoring
34
The Fisher-F distribution
35
36
Compare Test to Confidence Intervals
For non paired data, we cannot simply compute the difference
However CI is sufficient for parameter set 1
Tests disambiguate parameter sets 2 and 3
37
Test the assumptions of the test…
Need to test the assumptions
Normal
In each group: qqplot…
Same variance
38
39
4 Asymptotic Results
2 x Likelihood ratio statistic
40
41
The chi-square distribution
42
Asymptotic Result
Applicable when central limit theorem holds
If applicable, radically simple
Compute likelihood ratio statistic
Inspect and find the order p (nb of dimensions that H1 adds to H0)
This is equivalent to 2 optimization subproblems
lrs
=
= max likelihood under H1 - max likelihood under H0
The p-value is
43
Composite Goodness of Fit Test
We want to test the hypothesis that an iid sample has a distribution that
comes from a given parametric family
44
Apply the Generic Method
Compute likelihood ratio statistic
Compute p-value
Either use MC or the large n asymptotic
45
46
Is it
normal ?
47
48
49
Mendel’s Peas
P= 0.92 ± 0.05 => Accept H0
50
Test of Independence
Model
Hypotheses
51
Apply the generic method
52
53
5 Other Tests
Simple Goodness of Fit
Model: iid data
Hypotheses: H0 common distrib has cdf F()
H1 common distrib is anything
Kolmogorov-Smirnov: under H0, the distribution of
is independent of F()
54
55
Anderson-Darling
An alternative to K-S, less sensitive to “outliers”
56
57
58
Jarque Bera test of normality (Chapter 4)
Based on Kurtosis and Skewness
Should be 0 for normal distribution
59
60
Robust Tests
Median Test
Model : iid sample
Hypotheses
61
Median Test
62
Wilcoxon Signed
Rank Test
63
Wilcoxon Rank Sum Test
Model: Xi and Yj independent samples, each is iid
Hypotheses:
H0 both have same distribution
H1 the distributions differ by a location shift
64
Wilcoxon Rank Sum
Test
65
Turning Point
66
Questions
What is the critical region of a test ?
What is a type 1 error ? Type 2 ? The size of a test ?
What is the p-value of a test ?
67
Questions
What are the hypotheses for ANOVA ?
How do you compute a p-value by Monte Carlo simulation ?
A Monte Carlo simulation returns p = 0; what can we conclude ?
What is a likelihood ratio statistic test ? What can we say about its p-value ?
68
We have data X_1,…,X_m
and Y_1, …,Y_m. Explain
how we can compute the
p-value of a test that
compares the variance of
the two samples ?
We have a collection of
random variables X[i,j]
that corresponds to the
result of the ith simulation
when the machine uses
configuration j. How can
you test whether the
configuration plays a role
or not ?
69