Likelihood Ratio, Wald, and
Lagrange Multiplier (Score) Tests
Soccer Goals in European Premier
Leagues - 2004
Statistical Testing Principles
• Goal: Test a Hypothesis concerning parameter
value(s) in a larger population (or nature), based on
observed sample data
• Data – Identified with respect to a (possibly
hypothesized) probability distribution that is indexed
by one or more unknown parameters
• Notation:
Data: y1 ,..., yn
Parameter(s): 1 ,...,  k
Joint Density Function: f  y1 ,..., yn 1 ,...,  k 
Example – English League – Total Goals/Match
• Suppose we wish to test whether the mean number
of goals (in a hypothetically infinite population) of
games is equal to 3. Note: all games of equal length
(no overtime in regular season games)
• Data: Y=Total # of goals in a randomly selected game
• Distribution: Assume Poisson with parameter 
• Null Hypothesis: H0:  = 3
• Alternative Hypothesis: HA:  ≠ 3
• Joint Probability Density Function:
n

 n
 yi
e 
e  i1
f  y1 ,..., yn    
 n
yi !
i 1
 yi !
n
yi
i 1
yi  0,1, 2,...
Likelihood Function
• Another term for joint probability density/mass
function. Common Notation: L() or L(,y) or L(|y)
• Considered as a function of both the (observed) data
and the (unknown) parameter values
• Used in estimation and testing parameter value(s)
• Goal is to choose parameter value(s) that maximize
likelihood function given the observed data.
• Typically work with the log of the likelihood, as it is
often easier to differentiate to solve for maximum
likelihood (ML) estimators for many families of
probability distributions
ML Estimation of Poisson Mean
n
L  , y  
 yi
 n
e  i1
n
y !
i
i 1
 n

 n 
l  ln  L  , y    n    yi  ln    ln   yi !
 i 1 
 i 1

Taking derivative (wrt  ) and setting to zero for maximum:
n
y
dl
 n  i 1
d

i
n
set
0  0  n
y
i 1
^

i
n
^
0   
y
i 1
n
i
y
Total Goals Data
Goals
0
1
2
3
4
5
6
7
8
9
10+
Total
Frequency
30
79
99
67
61
24
11
6
2
1
0
380
Frequency of Total Goals
120
100
80
60
40
380
y
i
975


 2.57
380 380
^
i 1
20
0
0
1
2
3
4
5
6
7
8
9
10+
ln(L) versus theta (Ignoring constant term)
0
-50
-100
-150
ln(L)
-200
-250
ln(L)
-300
-350
-400
-450
-500
1
1.5
2
2.5
3
3.5
theta
4
4.5
5
5.5
6
Likelihood Ratio Test
•
•
•
•
•
•
•
Identify the parameter space: W  {:>0}
Identify the parameter space under H0: W0  {:0}
Evaluate the maximum log-Likelihood
Evaluate the log-Likelihood under H0
Any terms not involving parameter can be ignored
Take -2 times difference (H0 – maximum)
Under null hypothesis (and large samples), statistic is
approximately chi-square with 1 degree of freedom (number
of constraints under H0)
X
2
LR

  ^  
 2 ln  L 0 , y    ln  L   , y   
 
 

Soccer Goals Example
 380 
 380 
ln  L  , y    380    yi  ln    ln   yi !
 i 1 
 i 1

 380 
Under H 0 :   3 (Ignoring ln   yi !) :
 i 1

ln  L   3, y    380(3)  975  ln(3)   1140  1071.15  68.85
^
Maximum Value @   2.57 :
  ^ 
ln  L   , y    380(2.57)  975  ln(2.57)   976.6  920.31  56.29

 
Test Statistic:
X
2
LR

  ^  
2
 2 ln  L   3, y    ln  L   , y     2  68.85  (56.29)   25.12 > .05,1
 3.84
 
 

We have strong evidence to conclude the “true” mean total number of goals is below 3.
Wald Test - I
• By Central Limit Theorem arguments, many
estimators have sampling distributions that are
approximately normal in large samples
• Then, if we have an estimate of the variance of the
estimator, we can obtain a chi-square statistic by
taking the square of the distance between the ML
estimate and the value under H0 divided by the
estimated variance
• The estimated variance can be obtained from the
second derivative of the log-Likelihood
Wald Test - II
^
V     I 1  
 
  2 ln( L) 
where: I     E 

2
 

2
^




2

0 
^
^




Wald Chi-Square Statistic: X W2   ^ ^   I        0 
 
 

V  
 
 n

 n

Poisson Model:
ln  L  , y    n    yi  ln    ln   yi !
 i 1 
 i 1

n


ln  L  , y  

 n 
y
i 1

i
n

 2 ln  L  , y  

2

y
i 1
i
2
  2 ln( L) 
 n  n
I     E 


 2  

2


   


2
 X W2
^

^




n



2


0 
0 
 ^  ^





 ^ ^
 I     0  
^
 
 


V  
 
2
380  2.57  3
 27.34
2.57
2

Lagrange Multiplier (Score) Test
• Obtain the first derivative of the log-Likelihood
evaluated at the parameter under H0 (This is the
slope of the log-Likelihood, evaluated at 0 and is
called the score)
• Multiply the square of the score by the variance of
the ML estimate, evaluated at 0 . This is the inverse
of the variance of the score.
• Then chi-square test statistic is computed as follows:
X
2
LM
s  0 , y 

nI  0 
2
where s  , y  
 ln  L  , y  

Soccer Goals Example
 n

 n 
ln  L  , y    n    yi  ln    ln   yi !
 i 1 
 i 1 
n
 s  , y  
I   
1

2
 X LM
 ln  L  , y  

 n 
y
i 1

i
 s  0 , y   380 
1
1
 I  0   
0 3
s  , y  


55 

0

 23.57
nI  0 
385 1 3
2
2
2
2
Note that: X W2  27.34 > X LR
 25.12 > X LM
 23.57
975
 55
3
Log-Likelihood versus Theta (Ignoring Constant Term)
0
-20
Log(Likelihood) - Ignoring Constant Term
-40
LM Test
-60
LR Test
-80
ln(L)
Wald Test
Wald/LR1
-100
Wald/LR2
LM
-120
-140
-160
-180
2
2.2
2.4
2.6
2.8
3
Theta
3.2
3.4
3.6
3.8
4
Generalization to Tests of Multiple Parameters
1 
Parameter Vector:    
 k 
H 0 : R  r
 R11

R
 Rg1

 r1 
R1k 

 
r


  rank  R   g
 rg 
Rgk 
 
^
Maximum Likelihood Estimator over entire parameter space: 
~
Maximum Likelihood Estimator over constraint under H 0 : 
Likelihood Ratio Statistic: X
2
LR
   ~ 
  ^  
 2 ln  L   , y    ln  L   , y   

 
 
  
1
 ^   1  ^  T   ^ 
2
Wald statistic: X W  n  R   r   RI    R   R   r 

 
   

T
Lagrange Multiplier (Score) Statistic: X
2
LM
1

n

1  2
where: I ij     E 
ln  L  , y   
n  i  j

1
 ~    ~   ~ 
s  , y   I    s  , y 

    

T
si  , y  

ln  L  , y  
i
Soccer Goals Example
• Premier League Games in 2004 for k=5 European
Countries:





England
France
Germany
Italy
Spain
n1 = 380, Y1• = 975
n2 = 380, Y2• = 826
n3 = 306, Y3• = 890
n4 = 380, Y4• = 960
n5 = 380, Y5• = 980
 5
 5 yi
exp   nii  i
 i 1
 i 1
L  , y  
ni
5
 yij !
i 1
j 1
ni
yi   yij
j 1
Testing Equality of Mean Goals Among Countries - I
1 1 0 0 0 
1 0 1 0 0 

H 0 : 1   2  3   4   5  R  r
R
1 0 0 1 0 


1
0
0
0

1


5
5
 5 ni

ln  L  , y     nii    yi  ln i   ln   yij !
i 1
i 1
 i 1 j 1

 5 ni

Under H 0 : ln  L  , y     n   y  ln    ln   yij !
 i 1 j 1

^
 ln  L  , y  
yi
y
  ni 
  i  i  y i
i
i
ni
Under H 0 :
 ln  L  , y  

  n 
y

~
 
0 
0 
r 
0 
 
0 
y
 y 
n
^
^
^
^
975
826
890
960
980
 1 
 2.57  2 
 2.17  3 
 2.91  4 
 2.53  5 
 2.58
380
380
306
380
380
~
975  826  890  960  980 4631


 2.54
380  380  306  380  380 1826
  2 ln  L  , y   
 2 ln  L  , y  
 2 ln  L  , y  
yi
nii
ni



0
E






i2
i2
i  j
i2
i2
i


^
Testing Equality of Mean Goals Among Countries - II

  n1 


  n2 


s  , y     n3 


  n4 


  n5 

y1 
1 

y 2 
 2 
y3 

3 
y 4 

4 
y5 

5 
 380
 1826
1


 0


I  , y    0


 0


 0

0
0
0
380
1826 2
0
0
0
306
18263
0
0
0
380
1826 4
0
0
0




0 


0 


0 

380 

18265 
0
Likelihood Ratio Test
5
5
^
 5 ni

  ^ 
^ 
ln  L   , y     ni  i    yi  ln   i   ln   yij !

 
 
i 1
i 1
 i 1 j 1

5
 
  y    yi  ln y i
i 1
 5 ni

 ln   yij ! 
 i 1 j 1

 5 ni

 4631   918.71  641.33  950.20  889.69  928.43   ln   yij !
 i 1 j 1

 5 ni

 5 ni

 4631  4328.36  ln   yij !  302.64  ln   yij !
 i 1 j 1

 i 1 j 1

 5 ni

  ~ 
~
ln  L   , y     y  ln    y  ln   yij ! 

 
 
 i 1 j 1

 5 ni

 5 ni

 4631  4309.82  ln   yij !  321.18  ln   yij !
 i 1 j 1

 i 1 j 1

2
2
 X LR
 2  321.18  (302.64)   37.08
 4,.05
 9.49
Evidence that the true population means differ (in particular: France lower,
Germany higher than the others)
Wald Test
1
  ^
 ^
 
^

Wald statistic: X W2  n  R   r   RI 1    RT   R   r 

 
 

 
 2.57 
0
0 
1 1 0
 0   2.57  2.17  0   0.40 
2.17    



^
1 0 1 0
0
0
2.57  2.91  0   0.34 
 2.91     
R  r  

1 0
0 1 0  
  0   2.57  2.53  0   0.04 

 2.53    
 

0
0 1 
0   2.57  2.58  0   0.01
1 0

 2.58 
1826(2.57)

0
0
0
0


380


1826(2.17)

 1
1
1
1
0
0
0
0
0
0 
1 1 0


380
1 0
0
0 
1 0 1 0




^
0
1826(2.91)
 

  0 1 0
RI 1    RT  
0
0
0
0
0
1 0
0 1 0  

306
 

0
0 1 0 




1826(2.53)
0
0 1 
1 0
 0
0
0
0
0
0
0 1
380



1826(2.58) 
0
0
0
0


380


1
1
1
1
0057
0
0
0 
 .0068
.0125 .0068 .0068 .0068 


1
0
0
0
 .0068
.0068 .0163 .0068 .0068 

0
.0095
0
0  

 0 1 0
 1826 
0   1826 
 .0068


0
0
.0067
0 
.0068
.0068
.0135
.0068

0 1 0 

 0


0
0
0
.0068 
.0068 .0068 .0068 .0136 
 .0068

 0
0
0 1
132.72 25.34 36.23 35.49 
1


1  25.34 89.96 21.80 21.36 
 1  ^  T 
  RI    R  
 X W2  38.33
1826  36.23 21.80 119.25 30.53 
 




 35.49 21.36 30.53 117.44 
T
Lagrange Multiplier (Score) Test
Lagrange Multiplier (Score) Statistic: X
2
LM
1

n
1
 ~    ~ 
~ 
s  , y   I    s  , y 

   


T
4631
 2.5361
1826
975 

 380

380

0
0
0
~ 
~


 

1826 


826 
380
0
0
 380  ~   4.42 
 0
~


  
1826 


54.31




~
306
 ~   306  890  


0
0
s  , y  
 44.93 
I  , y    0
~
~










1826 

  1.47 

380
 380  960   6.41 
 0
0
0


~
~



 
1826 




980 
0
0
0
 380  ~ 
 0
 


0
0
0
0 
 4.42 
12.19
 54.31
 0
12.19
0
0
0 
1



~
~




 s   , y    44.93   I   , y    0
0
15.13
0
0 

 





0
0
12.19
0 
 1.47 
 0
 6.41 
0
0
0
12.19 
 0
2
 X LM
 36.83
~
  y  




0 


0 


0 


380 
~ 
1826  
0
Testing Goodness of Fit to Poisson Distribution
• All estimation and testing has assumed that number of
goals follow Poisson distributions
• To test whether that assumption is reasonable, we
compare the observed distributions of goals with what
we would expect under the Poisson model
• We can check whether the observed mean and variance
are similar (under Poisson model they are equal)
• We can also obtain a chi-square statistic by summing
over range of goals: (observed#-expected#)2/expected#
which under hypothesis of model fits is approximately
chi-square with (# in range)-1 degrees of freedom
Distributions of Goals
Goals
0
1
2
3
4
5
6
7
8
9
10
Total Games
Total Goals
Average
Observed
England France
30
54
79
82
99
110
67
57
61
51
24
15
11
4
6
6
2
1
1
0
0
0
380
380
975
826
2.5658
2.1737
Germany
18
43
66
77
54
29
13
4
1
1
0
306
890
2.9085
Italy
36
85
85
78
49
20
20
4
1
1
1
380
960
2.5263
7
X
2
obs

i 0
Spain
29
73
96
79
60
28
8
6
1
0
0
380
980
2.5789
Expected (Truncated at 7)
England France Germany
29.2062 43.2279 16.6947
74.9370 93.9639 48.5563
96.1363 102.1239 70.6130
82.2218 73.9950 68.4592
52.7410 40.2105 49.7783
27.0645 17.4810 28.9560
11.5736 6.3330 14.0364
6.1196
2.6648
8.9060
#N/A
#N/A
#N/A
#N/A
#N/A
#N/A
#N/A
#N/A
#N/A
 obsi  expi 
expi
Italy
30.3822
76.7549
96.9536
81.6451
51.5653
26.0541
10.9701
5.6747
#N/A
#N/A
#N/A
2
Spain
28.8244
74.3367
95.8553
82.4019
53.1275
27.4026
11.7783
6.2732
#N/A
#N/A
#N/A
Chi-square
CritVal
P-Value
approx
Chi-Square Statistic
England France Germany
0.0216
2.6843
0.1021
0.2203
1.5233
0.6358
0.0853
0.6074
0.3014
2.8180
3.9034
1.0655
1.2933
2.8951
0.3580
0.3470
0.3521
0.0001
0.0284
0.8595
0.0765
1.3558
7.0529
0.9482
6.1697
14.0671
0.5201
~ 
19.8780
14.0671
0.0058
3.4876
14.0671
0.8365
2
7
All leagues, except France, appear to be well described by the Poisson
distribution. Especially England, Germany, and Spain
Italy
1.0388
0.8857
1.4738
0.1627
0.1276
1.4068
7.4328
0.3095
Spain
0.0011
0.0240
0.0002
0.1404
0.8890
0.0130
1.2120
0.0842
12.8377
14.0671
0.0762
2.3640
14.0671
0.9370