Homework 8 – Due 12 am CST, 23 April 2012
The total points on this assignment is 100. The dataset is posted on the class website. All problems are from
the book and are worth the points indicated in the parenthesis. Datasets are also all available on the class
website.
1. Let X 1 , X 2 , . . . , X n ∼ Np (µ, Σ). We are interested in testing the null hypothesis that the variancecovariance matrix Σ is diagonal against the alternative that it is general structured (with no constraint
other than the fact that it is a positive-definite matrix). From the results in the second exam, we know
that there are two asymptotic null distributions for testing the above: these are the χ2 -test and the
normal test (assuming that each log-eigenvalue of the correlation matrix is normally distributed with
zero mean and variance n2 ). In this exercise, we will evaluate which of the two tests perform better
and under what circumstances.
Our analysis will comprise of calculating the receiver operating characteristic (ROC) curve for both
tests for different sample sizes n = 100, 500, 1000. The ROC curve is a plot of the true positive
rate against the false positive rate (i.e., the power (or sensitivity) against the Type I error (or 1 specificity) rate. As Type I error increases, the power of any test should approach unity. Therefore
the real evaluation is to decide which of the two curves hit unity at a faster rate. This is evaluated
graphically and numerically by means of the area under the ROC curve (AUC). The higher the AUC,
the better the testing methodology.
We will evaluate the tests for p = 3 only and for a very spatial case of correlation matrix with the same
correlation coefficient ρ for all variables. We assume that µ = 0 for the simulation (but unknown for
purposes of estimation) and that the variances are unity (but unknown for purposes of estimation). ρ
varies in the range (-0.5, -0.3, -0.1, 0, 0.1, 0.3, 0.5). We will evaluate the two tests for different n and
ρ.
Perform a bootstrap analysis of the tests and report the results in terms of a table of AUC measures.
Is there any conclusion we can arrive at? [40 points]
2. Consider the breast cancer dataset. Perform tree-based classification on the dataset using only the
means and the 30 variables. Display the two trees and compare them after pruning each following
leave-one-out cross-validation. [20 points]
3. The vowel dataset has both training and test sets: our objective here is to evaluate the different methods
(LDA, QDA, k-nearest neighbors, CART) on the training set and come up with a classification rule.
Then you are asked to evaluate performance of the classification rule on the test set. For fun, also
report the performance of the other classification rules on the test dataset. [20 points]
4. Derive the E and the M steps for the problem and the dataset in Example 5.13 in the book. [20 points]