Handout #10: Additional Tests of Significance
Example 10.1: Consider the following data regarding bone density. In this study, T scores are being
used to measure of bone density.
A snip-it of the dataset is provided here.
Linear Regression Setup



Response Variable: Forearm T-Score
Predictor Variable: AP Spine T-Score, i.e. the gold standard in measuring t-scores
Assume the following structure for mean and variance functions
o
o
𝐸(𝐹𝑜𝑟𝑒𝑎𝑟𝑚 | 𝑆𝑝𝑖𝑛𝑒 ) = 𝛽0 + 𝛽1 ∗ 𝑆𝑝𝑖𝑛𝑒
𝑉𝑎𝑟(𝐹𝑜𝑟𝑒𝑎𝑟𝑚|𝑆𝑝𝑖𝑛𝑒) = 𝜎 2
1
Regression Output
Scatterplot of Forearm T Score vs. AP Spine T Score
Residual plot for this regression model
Regression output from JMP.
2
Overall F Test
The F test identified above is commonly given with standard regression output. The purpose of this test
is to determine whether or not the regression model, using these predictor(s), is useful in reducing the
unexplained variation in the response. Using language presented earlier in this course, this test
determines whether or not considering the conditional distribution is useful.
𝐻0 : 𝐸(𝐹𝑜𝑟𝑒𝑎𝑟𝑚|𝑆𝑝𝑖𝑛𝑒) = 𝛽0
𝐻𝐴 : 𝐸(𝐹𝑜𝑟𝑒𝑎𝑟𝑚|𝑆𝑝𝑖𝑛𝑒) = 𝛽0 + 𝛽1 ∗ 𝑆𝑝𝑖𝑛𝑒
Comment: The model under HO is a linear model that contains only an intercept term. In this situation,
the best estimate for 𝛽0 , 𝛽̂0 , is simply the average forearm measurement because the spine
measurements are not being considered under the null hypothesis.
The test statistic for this test is given by
(𝑆𝑆𝐸𝐻𝑜 − 𝑆𝑆𝐸𝐻𝑎 )
⁄(𝑑𝑓 − 𝑑𝑓 )
𝐻𝑜
𝐻𝑎
𝐹 𝑅𝑎𝑡𝑖𝑜 =
𝑆𝑆𝐸𝐻𝑎
⁄𝑑𝑓
𝐻𝑎
The p-value for this test is obtained from a F-distribution with 𝑑𝑓𝑛𝑢𝑚𝑒𝑟𝑎𝑡𝑜𝑟 = (𝑑𝑓𝐻𝑜 − 𝑑𝑓𝐻𝑎 ), and
𝑑𝑓𝑑𝑒𝑛𝑜𝑚𝑖𝑛𝑎𝑡𝑜𝑟 = 𝑑𝑓𝐻𝑎
In Excel,
3
Wiki Entry for F-Distribution
http://en.wikipedia.org/wiki/F-distribution
Showing the computations for our example.
𝐹 𝑅𝑎𝑡𝑖𝑜
(𝑆𝑆𝐸𝐻𝑜 − 𝑆𝑆𝐸𝐻𝑎 )
⁄(𝑑𝑓 − 𝑑𝑓 )
𝐻𝑜
𝐻𝑎
=
𝑆𝑆𝐸𝐻𝑎
⁄𝑑𝑓
𝐻𝑎
(30.498 − 15.591)
⁄(39 − 38)
=
15.591⁄
38
= 36.33
Sketch the test statistic and p-value on the following F-distribution graph.
Getting the p-value in Excel
Task:
Write a conclusion for this test in context.
𝐻0 : 𝐸(𝐹𝑜𝑟𝑒𝑎𝑟𝑚|𝑆𝑝𝑖𝑛𝑒) = 𝛽0
𝐻𝐴 : 𝐸(𝐹𝑜𝑟𝑒𝑎𝑟𝑚|𝑆𝑝𝑖𝑛𝑒) = 𝛽0 + 𝛽1 ∗ 𝑆𝑝𝑖𝑛𝑒
Conclusion:
4
Test for Comparing Models –
The “BIG” TEST
The overall F Test presented above can be generalized. For example, in the T-Scores investigation, the
goal was to establish whether or not the AP Spine T Score could be used to reasonably predict the
Forearm T Score. If this were the case, then the y=x line would suffice. The begs the question of
whether or not our regression line statistically close enough to the y=x line.
The formal hypothesis would be set up as follows.
𝐻0 : 𝐸(𝐹𝑜𝑟𝑒𝑎𝑟𝑚|𝑆𝑝𝑖𝑛𝑒) = 𝑆𝑝𝑖𝑛𝑒
𝐻𝐴 : 𝐸(𝐹𝑜𝑟𝑒𝑎𝑟𝑚|𝑆𝑝𝑖𝑛𝑒) = 𝛽0 + 𝛽1 ∗ 𝑆𝑝𝑖𝑛𝑒
Comments:


The model under HO is often referred to as the restricted model; whereas, the model under HA is
identified as the unrestricted model.
The restricted model, i.e. the model under HO, must be a subset of the unrestricted model, i.e
the model under HA.
5
Fitting these model in JMP
Fitting the unrestricted model
Fitting the restricted model
Showing the computations for our example.
𝐹 𝑅𝑎𝑡𝑖𝑜
(𝑆𝑆𝐸𝐻𝑜 − 𝑆𝑆𝐸𝐻𝑎 )
⁄(𝑑𝑓 − 𝑑𝑓 )
𝐻𝑜
𝐻𝑎
=
𝑆𝑆𝐸𝐻𝑎
⁄𝑑𝑓
𝐻𝑎
(_______________ − _______________)
⁄(_______
− _______)
=
_______________⁄____________
= 11.28
6
Sketch the test statistic and p-value on the following F-distribution graph.
Getting the p-value in Excel
Task:
Write a conclusion for this test in context.
𝐻0 : 𝐸(𝐹𝑜𝑟𝑒𝑎𝑟𝑚|𝑆𝑝𝑖𝑛𝑒) = 𝑆𝑝𝑖𝑛𝑒
𝐻𝐴 : 𝐸(𝐹𝑜𝑟𝑒𝑎𝑟𝑚|𝑆𝑝𝑖𝑛𝑒) = 𝛽0 + 𝛽1 ∗ 𝑆𝑝𝑖𝑛𝑒
Conclusion:
Using Matrix Construction to Accomplish the “BIG” Test
Recall, the hypothesis above was written as
𝐻0 : 𝐸(𝐹𝑜𝑟𝑒𝑎𝑟𝑚|𝑆𝑝𝑖𝑛𝑒) = 𝑆𝑝𝑖𝑛𝑒
𝐻𝐴 : 𝐸(𝐹𝑜𝑟𝑒𝑎𝑟𝑚|𝑆𝑝𝑖𝑛𝑒) = 𝛽0 + 𝛽1 ∗ 𝑆𝑝𝑖𝑛𝑒
Realize, this is equivalent to the following
𝐻0 : 𝐸(𝐹𝑜𝑟𝑒𝑎𝑟𝑚|𝑆𝑝𝑖𝑛𝑒) = 0 + 1 ∗ 𝑆𝑝𝑖𝑛𝑒
𝐻𝐴 : 𝐸(𝐹𝑜𝑟𝑒𝑎𝑟𝑚|𝑆𝑝𝑖𝑛𝑒) = 𝛽0 + 𝛽1 ∗ 𝑆𝑝𝑖𝑛𝑒
If the model parameter representation for our model is used instead the above equations, we obtain
1
𝐻0 : [
0
1
𝐻𝐴 : [
0
0 𝛽0
0
][ ] = [ ]
1 𝛽1
1
𝛽
0 0
0
][ ] ≠ [ ]
1 𝛽1
1
which more generally can be written as
𝐻0 : 𝑪𝜷 = 𝒄
𝐻𝐴 : 𝑪𝜷 ≠ 𝒄
7
Now, it can be shown, that the numerator of the test statistic from the test above is
̂ − 𝒄)′ [𝑪(𝑿′ 𝑿)−𝟏 𝑪′ ]−1 (𝑪𝜷
̂ − 𝒄)
(𝑆𝑆𝐸𝐻𝑜 − 𝑆𝑆𝐸𝐻𝑎 ) = (𝑪𝜷
This quantity can be easily computed in R. This is shown here.
> SSE.ho_minus_SSE.Ha=t(beta.hat-littlec) %*% solve(xprimex.inv,diag(2)) %*%
(beta.hat-littlec)
> SSE.ho_minus_SSE.Ha
[,1]
[1,] 9.258655
Or, R can compute the F Ratio value directly as follow.
> (t(beta.hat-littlec) %*% solve(xprimex.inv,diag(2)) %*% (beta.hatlittlec)/2)/(15.591/38)
[,1]
[1,] 11.28308
Getting the p-value can be accomplished in R as well.
> 1-pf(11.283,2,38)
[1] 0.0001424096
Task:
Write a conclusion for this test in context.
𝐻0 : 𝐸(𝐹𝑜𝑟𝑒𝑎𝑟𝑚|𝑆𝑝𝑖𝑛𝑒) = 0 + 1 ∗ 𝑆𝑝𝑖𝑛𝑒
𝐻𝐴 : 𝐸(𝐹𝑜𝑟𝑒𝑎𝑟𝑚|𝑆𝑝𝑖𝑛𝑒) = 𝛽0 + 𝛽1 ∗ 𝑆𝑝𝑖𝑛𝑒
↔
1
𝐻0 : [
0
1
𝐻𝐴 : [
0
0 𝛽0
0
][ ] = [ ]
1 𝛽1
1
0 𝛽0
0
][ ] ≠ [ ]
1 𝛽1
1
Conclusion:
8
The linear combination of the parameter test can be accomplished in JMP as well.
Select Estimates > Custom Test
Setting the C Matrix
Output provided by JMP
9
Tasks: The test statistics, i.e. F. Ratio value, can be used more to measure how similarity of two models.
A large F Ratio value implies the two models are dissimilar and vise versa.
1. Obtain the F Ratio value for comparing the response, Forearm T Score, against Hip T Score.
2.
Obtain the F Ratio value for comparing the response, Forearm T Score, against Neck T Score.
3. For which type (AP Spine, Hip, or Neck) does Forearm T Score agree most? You should use the F
Ratio to make these comparisons.
4. Suppose an individual with a T-Score less than -1 should seek treatment. If the Forearm T-score
is used instead of the AP Spine, Hip, or Neck, then what is likely to happen to those that need
treatment? Discuss.
10