10 – Discriminant Analysis (LDA, QDA, and RDA)
10.1 - Discriminant Analysis in JMP
Example 10.1 – Flea Beetles: The genus of flea beetle Chaetocnema contains three species
that are difficult to distinguish from one another and, indeed, were confused for a long
time. These data comprise the six different measured characteristics made on these
three species of flea beetles.
The variables in this data set are:

Species - species of flea beetle (1= Chaetocnema concinna, 2 = Chaetocnema
heikertingeri, or 3 = Chaetocnema heptapotamica)






Width 1 - a numeric vector giving the width of the first joint of the first tarsus in
microns (the sum of measurements for both tarsi)
Width 2 - a numeric vector giving the width of the second joint of the first tarsus
in microns (the sum of measurements for both tarsi)
Maxwid 1 - a numeric vector giving the maximal width of the head between the
external edges of the eyes in 0.01 mm
Maxwid 2 - a numeric vector giving the maximal width of the aedeagus in the
fore-part in microns
Angle - a numeric vector giving the front angle of the aedeagus (1 unit = 7.5
degrees)
Width 3 - a numeric vector giving the aedeagus width from the side in microns
The main questions of interest are:
Are there significant differences in the two measured characteristics for these three
species of flea beetle?
Can a classification rule based on these six measurements be developed? A
classification rule should allow us to predict the species of flea beetle based on these six
values measurements. Statistical methods for developing classification rules include
discriminant analysis, classification trees, logistic regression, nearest-neighbors, naïve
Bayes classifiers, support vector machines, neural networks, etc.
287
Analysis in JMP
First we use graphical techniques such as histograms, comparative boxplots and
scatterplots with color coding in an attempt to derive a classification rule for the three
species of flea beetle. This can done as follows:
Color coding - From the Rows menu select Color/Marker by Col... then check box
labeled Marker (color will already be checked by default). Now highlight Species with
the mouse and click OK. In subsequent plots the different species of flea beetle will
color coded and a different plotting symbol will be used for each species. Look at the
spreadsheet to see the correspondence.
Histograms - Select Distribution of Y from the Analyze menu and add Species and all
six of the measurements to the right hand box, then click OK. Now use the mouse to
click on the bars corresponding to the different species of beetle in the bar graph for
Species, carefully observing what happens in the histograms for the two measured
characteristics. This is an example of linked viewing - data that is highlighted or
selected in one plot becomes highlighted in all other plots.
What can we conclude from this preliminary histogram analysis?
288
From the above displays we can see that Heptapot flea beetles has width measurements
between 125 and 150. On the angle variable we can see that this species of flea beetle
tends to have much smaller angle measurements than the other two species. At this
point might conjecture that if we observed a width value between 125-150 and an angle
measurement less than 12 for an unclassified flea beetle that it would be classified as a
Heptapot flea beetle. Similar types of observations can be made by clicking on the bars
for the other two species in the bar graph for species.
Comparative Boxplots - Select Fit Y by X from the Analyze menu and add Species in
the X box and all six measurements in the Y box. Add boxplots and/or means diamonds
to the resulting plots by selecting these options in the Display menu which is located
below the plot.
289
Comparative Displays for Width 1
The plot at the top to the left gives
comparative boxplots for the three
flea beetle genus for the width
variable. We clearly see that the
Heikert. genus have the lowest
width measurements in general,
while Concinna generally have the
highest.
The Compare density option gives
smoothed histograms which also can
be used to show univariate
differences between the three genus
of flea beetle.
290
Comparative Displays for Angle
These graphs again show that the genus Heptapot have the smallest angle measurements
in general. Another interesting plot to consider when looking at multivariate data
where there are potentially different populations involved is the parallel coordinate
plot. In the parallel coordinates plot, a set of parallel axes are drawn for each variable.
Then a given row of data is represented by drawing a line that connects the value of
that row on each corresponding axis. When you have predefined populations
represented in your data, these plots can be particularly useful when looking for
distinguishing characteristics of these populations.
To obtain a parallel coordinate plot in JMP select Parallel Plot from the Graph menu
and place the variables of interest in the Y response box. You can place a grouping
variable in the X, Grouping box if you wish to see a parallel coordinate display for each
level of the grouping variable X. The results from both approaches are shown on the
following page.
291
At this point we can see our classification rule could be stated as follows... If the angle is
less than 12 classify a flea beetle as genus Heptapot. If angle is greater than 12 then the
beetles is most likely from genus Concinna or Heikert. To distinguish genus Concinna
from Heikert the comparative boxplots for Maxwid 2 suggest that we could classify a
flea beetle with angle greater than 12 as being from genus Concinna if the maximal
width exceeded 133 or 134, otherwise classify it as being from genus Heikert.
292
To formally compare the genus types on these six characteristics we could perform a
one-way analysis of variance (ANOVA) for each. This is can be done by selecting
Means/ANOVA from the Analysis menu below the plot. Pairwise multiple
comparisons can be done using Tukey’s method by selecting Compare All Pairs from
the Oneway Analysis pull-down menu.
ANOVA TABLE for Maximal Width 2
The p-value for the F-test for testing the null hypothesis that the population means for
width are all equal is less than <.0001 indicating that at least two means are significantly
different. To decide which means are significantly different we can use Tukey’s
multiple comparison procedure examining all pairwise comparisons of the populations
means. The results are shown below.
Results of Tukey's Multiple Comparisons
Here we can see that the mean maximal widths differ for all species.
293
To aid in the development of a simple classification rule we could also examine a
scatterplot for Maxwid 2 vs. Angle making sure that the different species are still color
coded (see above). This can be done by choosing Fit Y by X from the Analyze menu and
placing Width in the X box and Angle in the Y box. We can add a density ellipse for
each group to the plot by choosing Grouping Variable from the Analysis menu below
the plot and highlighting Species as the variable to group on. Then select Density
Ellipses with the desired percent confidence from the Analysis menu. The ellipses
should contain the specified percent of the points for each group. Notice the nice
separation of the density ellipses.
Scatterplot of Angle vs. Width
We can use this scatterplot to state a classification rule that can be used to identify
species on the basis of these measurements. Clearly an angle less than 12 would
indicate that the flea beetle was genus Heptapot. Concinna and Heikert both appear
to have angle measurements exceeding 12. However by using the maximal width
measurement we can distinguish between these two species. Concinna flea beetles have
widths exceeding 134, while the Heikert beetles have widths less than 134. To
statistically formalize the procedure above we could perform discriminant analysis,
which we will consider later in this tutorial.
294
Rather than consider each of the six characteristics individually using ANOVA.
MANOVA allows us to determine if the three species differ on the six characteristics
measured simultaneously. This is achieved by looking at the multivariate response
consisting of all six measurements rather than each characteristic individually which
was done above through the use of one-way ANOVA. To perform MANOVA in JMP
first select the Fit Model option in the Analyze menu. Place all six measurements in
the Y box and place Species in the Effects in Model box, then select MANOVA from the
Personality pull-down menu and select Run. When the analysis is completed, choose
Identity from the menu below where it says Choose Response and click Done. Below
you will see a variety of boxes with many different statistics and statistical tests. To test
the main hypotheses that these measured characteristics differ from species to species
you will want to examine the results of the tests in the Species box. This box contains
four different test statistics which all the answer the question - - do the means for these
six measurements differ in any way from species to species? If the p-value is small for
any of them there is evidence that the mean of width and angle differ significantly from
species to species. You can examine a profile of the mean values of width and angle for
each species by examining the plot for species in the Least Square Means box in the
output. The table on the following page shows the results of the MANOVA for species.
MANOVA Results for Species Comparison
Here we can see that the p-values associated with each test statistic is less than .0001,
which provides compelling evidence that the three species differ significantly on the six
measured characteristics. To see the nature of these differences select Centroid Plot
from the pull-down menu at the top of the Species box. The centroid plot for these data
is shown on the following page.
295
Canonical Centroid Plot
A 2-D canonical centroid plot is a plot of the first
two discriminants from Fisher’s Discriminant
Analysis. Fisher’s discriminant analysis is a
method where linear combinations of X1, …, Xp
are found that maximally separate the groups.
The first linear combination maximally separates
the groups in 1-D dimension, the second linear
combination maximally separates the groups
subject to the constraint that the linear
combination is orthogonal to the first. Thus
when the resulting linear combinations are plotted
for each observation we obtain a scatterplot
exhibiting zero correlation and hopefully good
group separation. We can also visualize the
results in 3-D considering a third linear
combination, provided there are more than three
groups/populations.
The above plot confirms things we have already seen. Notice that the Concinna and
Heikert centroid circles lie in the direction of the Angle ray indicating that these two
genus types have large angle measurements relative to genus Heptapot. The circle for
genus Heikert lies in the direction of the Width 1 ray indicating that these flea beetles
have relatively large width measurements. In total, we see that a nice species
separation is achieved.
The canonical centroid plot displays the results of discriminant analysis. Discriminant
analysis, though related to MANOVA, is really a standalone method. There are two
main approaches to classic discriminant analysis: Fisher’s method which is discussed
above and a Bayesian approach where the posterior probability of group membership is
calculated assuming X’s have an approximate multivariate normal distribution. In
Fisher’s approach to discriminating between g groups, (g – 1) orthogonal linear
combinations are found that maximally separate the groups, with the first linear
combination doing largest degree of separation and so. Future observations are
classified to the group they are closest to in the lower dimensional space created by
these linear combinations.
296
In LDA/QDA we classify observations to groups based on their posterior
“probability” of group membership. The probabilities are calculated assuming the
populations have a multivariate normal distribution. In linear discriminant analysis
(LDA) we assume each population, while having different mean vectors, have the same
variance-covariance structure. In quadratic discriminant analysis (QDA) we assume
that the variance-covariance structure of the populations is different. QDA requires
more observations per group, (𝑛𝑖 > 𝑝 )𝑓𝑜𝑟 𝑎𝑙𝑙 𝑖, as the variance-covariance matrix is
estimated separately for each of the groups, whereas LDA uses a pooled estimate of the
common variance-covariance structure. Because QDA is effectively estimating more
“parameters” it should provide a better discrimination between than LDA. Regularized
discriminant analysis (RDA) is a balance between the two extremes by essentially
taking a weighted average of variance-covariance structure of the two approaches. You
can think of it as a shrunken version of QDA, where the shrinkage is towards LDA.
To classify future observations a Bayesian approach is used where the posterior
probability of group membership for each group is computed as
𝑃(𝐺𝑟𝑜𝑢𝑝 = 𝑘|𝒙) =
2
exp(−.5 𝐷𝑘∗ (𝒙))
∑𝑔𝑖=1 exp(−.5 𝐷𝑖∗ 2 (𝒙))
𝑓𝑜𝑟 𝑘 = 1, … , 𝑔
where,
2
̅𝒊 )′ 𝑺−𝟏
̅𝒊 ) + 𝒍𝒏|𝑺𝒊 | − 𝟐𝒍𝒏𝒑𝒊 for QDA with unequal priors
𝐷𝑖∗ (𝒙) = (𝒙 − 𝒙
𝒊 (𝒙 − 𝒙
2
̅𝒊 )′ 𝑺−𝟏
̅𝒊 ) + 𝒍𝒏|𝑺𝒊 |
𝐷𝑖∗ (𝒙) = (𝒙 − 𝒙
𝒊 (𝒙 − 𝒙
for QDA with equal priors
2
̅𝒊 )′ 𝑺−𝟏
̅𝒊 ) − 𝟐𝒍𝒏𝒑𝒊
𝐷𝑖∗ (𝒙) = (𝒙 − 𝒙
𝒑 (𝒙 − 𝒙
for LDA with unequal priors
2
̅𝒊 )′ 𝑺−𝟏
̅𝒊 )
𝐷𝑖∗ (𝒙) = (𝒙 − 𝒙
𝒑 (𝒙 − 𝒙
for LDA with equal priors
and
𝑆𝑝−1 =
𝑔
∑𝑖=1(𝑛𝑖 −1)𝑆𝑖−1
𝑔
∑𝑖=1(𝑛𝑖 −1)
 pooled estimate of the common variance-covariance matrix in LDA
and 𝑝𝑖 = 𝑝𝑟𝑖𝑜𝑟 𝑝𝑟𝑜𝑏𝑎𝑏𝑖𝑙𝑖𝑡𝑦 𝑜𝑓 𝑏𝑒𝑖𝑛𝑔 𝑓𝑟𝑜𝑚 𝑝𝑜𝑝𝑢𝑙𝑎𝑡𝑖𝑜𝑛 𝑜𝑟 𝑔𝑟𝑜𝑢𝑝 𝑖. For RDA we use the
same formula as QDA with the sample variance-covariance matrix replaced by
𝑆𝑖∗ = 𝜆𝑆𝑖 + (1 − 𝜆)𝑆𝑝
297
If the  parameter is different from 0, then another convex combination is formed based
on 𝑆𝑖∗ .
𝛾
𝑆𝛾 = (1 − 𝛾)𝑆𝑖∗ + 𝑝 𝑡𝑟(𝑆𝑖∗ )𝐼 where 𝐼 is the pxp identity matrix.
More on RDA: The values for the two regularization parameters, 0 ≤ 𝜆 ≤ 1 and 0 ≤
𝛾 ≤ 1, are chosen to minimize jointly an unbiased estimate of future misclassification
risk which we could obtain using some form of cross-validation. Regularized
discriminant analysis provides for a fairly rich class of regularization alternatives. The
four corners defining the extremes of the (𝜆, 𝛾) plane represent the well-known
classification procedures outlined above. The lower left corner (𝜆 = 0, 𝛾 = 0) represents
QDA. The lower right corner (𝜆 = 1, 𝛾 = 0) represents LDA. The upper right corner (𝜆 = 1, 𝛾 = 1)
corresponds to the nearest-means classifier where an observation is assigned to the class
with the closest (Euclidean distance) mean, but this could be changed to statistical
distance if we first standardize the 𝑋𝑖 ′𝑠. The upper left corner of the plane represents a
weighted nearest-means classifier, with the class weights inversely proportional to the
average variance of the measurement variables within the class. Holding 𝛾 fixed at 0
and varying 𝜆 produces models somewhere between QDA and LDA. Holding 𝜆 fixed at
0 and increasing 𝛾 attempts to un-bias the sample-based eigenvalue estimates. Holding
𝜆 fixed at 1 and increasing 𝛾 gives rise to a ridge-regression analog for LDA. A good
pair of values for 𝜆 and 𝛾 is not likely to be known in advance. We therefore must use
cross-validation methods to find optimal values. The resulting misclassification loss
averaged over the training/test samples are then used as an estimate of future
misclassification risk. We can choose a grid of points in the (𝜆, 𝛾) plane (0 ≤ 𝜆 ≤ 1, 0 ≤
𝛾 ≤ 1) and evaluate the cross-validated estimate of misclassification risk at each
prescribed points on the grid, and then choose the point with the smallest estimated
misclassification rate.
With all methods (LDA, QDA, and RDA) the posterior probabilities for each
observation in the training data are computed and group membership is predicted. The
performance of the discriminant analysis is the reported the number/percent
misclassified in the training data and in the test data if it is available. In JMP future
observations can be classified by adding them to the data table.
298
LDA/QDA Details
299
LDA/QDA Details (continued)
300
To perform LDA/QDA/RDA in JMP, choose Discriminant from the Multivariate
Methods option within the Analyze menu as shown below.
Put all six measurements
in the Y, Covariates box
and Species in the X,
Categories.
The results are shown below:
Here we can see that linear discriminant analysis misclassifies none of the flea beetles in
these training data.
A contingency table showing the classification results is displayed below the table of
posterior genus probabilities for each observation. This table is sometimes referred to as
a confusion matrix.
301
Confusion Matrix for Species Classification
None of the species are misclassified giving us an apparent error rate (APER) of .000 or
0%. The apparent error rate is analogous to the RSS for the training data fit in a
regression problem. Applying either QDA (which actually might be recommended
given that the variability of some of the characteristics differs across species) or RDA
results in perfect classification of the species as well.
We can save the distances to each group along with the posterior probabilities for each
species to our spreadsheet by selecting Save Formulas from Score Options pull-out
menu as shown below.
Having saved these formulae to the data spreadsheet we can use the results of our
discriminant analysis to classify the species of new observations. For example, adding 2
rows to our spreadsheet and entering the measurements for two yet to be classified
beetles will obtain their predicted species based on our model.
302
The predictions are shown below for the two new beetles, the first is classified as
Heikert and the second as Concinna.
Below is a visualization of the classification of the new flea beetles using the first two
discriminants.
303
Example 2: Italian Olive Oils: The goal here is to classify olive oils in terms of the area of Italy
they are from based upon amino acid composition. There are two geographic classifications in
these data. The first classification is nine individual growing areas in Italy (Area Name) – East
Liguria, West Liguria, Umbria, North-Apulia, South-Apulia, Sicily, Coastal Sardinia, InlandSardinia, and Calabria. A broader classification is the growing region in Italy (Region Name) –
Northern, Southern, and Sardinia. The map below should help in your understanding of where
these areas/regions are located in Italy.
Puglia = Apulia
Sardegna = Sardinia
Sicilia = Sicily
The bar graph above shows the number of olive oils
in these data from each area. The plots on the left
show a parallel coordinate plot and comparative
boxplots for palmitic acid across areas.
304
The scatterplot matrix on the right above clearly shows that LDA is NOT appropriate for these
data. Why? Thus we will consider QDA first.
We can see the overall misclassification rate on
the entire training data set is 11/572 = 1.923%.
We can randomly choose some cases to use as
test data.
.
The end result of doing this will be a column
with 0 for training and 1 for test cases. We can
then Hide and Exclude the test cases and when
we run a discriminant analysis (LDA, QDA, or
RDA) it will use the Excluded cases as test cases
and report the misclassification rate on those
cases that were not used to perform the
discriminant analysis.
305
We can create training/test cases in JMP using the Initialize Data feature and then we
can use a bar graph of this column to Hide and Exclude the test cases.
The results from a QDA now contain a misclassification rate (APER) for the training
cases and the test cases (Excluded).
The APER = 1.312% and the Test Case Error Rate = 4.712%. Next we consider using
regularized discriminant analysis for these data. We will again use the training and test
cases to obtain a measure of prediction error.
306
Results from RDA with 𝜆 = .6667 and 𝛾 = 0 (giving a test case error rate = 8.38%)
and using 𝜆 = 0.1 and 𝛾 = 0 (giving a test case error rate = 4.19%)
There is no easy method for finding “optimal” values 𝜆 & 𝛾 in JMP, it is pretty much
trial and error.
307
10.2 - LDA, QDA, and RDA in R
Example 10.2: Italian Olive Oils (revisited)
> dim(Olives)
[1] 572 12
> names(Olives)
[1] "Region.name" "Area.name"
[6] "palmitoleic" "strearic"
[11] "linolenic"
"eicosenoic"
"Region"
"oleic"
"Area"
"linoleic"
"palmitic"
"eicosanoic"
> sam = sample(1:572,floor(.3333*572),replace=F)
> OA.test = OliveOils[sam,-c(1,3,4,12)]  remove Region identifiers
> OA.train = OliveOils[-sam,-c(1,3,4,12)]
> dim(OA.test)
[1] 190
8
> dim(OA.train)
[1] 382
8
> names(OA.train)
[1] "Area.name"
"palmitic"
"palmitoleic" "strearic"
"oleic"
[6] "linoleic"
"eicosanoic" "linolenic"
> table(OA.test$Area.name)  make sure all areas are in test cases!
Calabria Coastal-Sardinia
17
12
North-Apulia
Sicily
10
8
West-Liguria
15
> table(OA.train$Area.name)
Calabria Coastal-Sardinia
39
21
North-Apulia
Sicily
15
28
West-Liguria
35
East-Liguria
17
South-Apulia
70
Inland-Sardinia
23
Umbria
18
East-Liguria
33
South-Apulia
136
Inland-Sardinia
42
Umbria
33
> area.lda = lda(Area.name~.,data=OA.train)
> pred.lda = predict(area.lda,newdata=OA.test)
> misclassification(pred.lda$class,OA.test$Area.name)
Table of Misclassification (row = predicted, col = actual)
y
fit
Calabria Coastal-Sardinia East-Liguria Inland-Sardinia
Calabria
16
0
0
0
Coastal-Sardinia
0
11
0
0
East-Liguria
0
0
10
0
Inland-Sardinia
0
1
0
21
North-Apulia
0
0
4
0
Sicily
1
0
0
1
South-Apulia
0
0
0
1
Umbria
0
0
0
0
West-Liguria
0
0
3
0
y
fit
North-Apulia Sicily South-Apulia Umbria West-Liguria
Calabria
0
1
0
0
0
Coastal-Sardinia
0
0
0
0
0
East-Liguria
2
0
0
1
0
Inland-Sardinia
0
0
0
0
0
North-Apulia
7
1
0
0
0
Sicily
0
6
4
0
0
South-Apulia
0
0
66
0
0
Umbria
1
0
0
17
0
West-Liguria
0
0
0
0
15
Misclassification Rate = 0.111
> area.qda = qda(Area.name~.,data=OA.train)
308
> pred.qda = predict(area.qda,newdata=OA.test)
> misclass(pred.qda$class,OA.test$Area.name)
> misclassification(pred.qda$class,OA.test$Area.name)
Table of Misclassification (row = predicted, col = actual)
y
fit
Calabria Coastal-Sardinia East-Liguria Inland-Sardinia
Calabria
14
0
0
0
Coastal-Sardinia
0
11
0
1
East-Liguria
0
0
16
0
Inland-Sardinia
0
0
0
21
North-Apulia
0
0
0
0
Sicily
2
0
0
0
South-Apulia
1
1
0
1
Umbria
0
0
0
0
West-Liguria
0
0
1
0
y
fit
North-Apulia Sicily South-Apulia Umbria West-Liguria
Calabria
0
1
0
0
0
Coastal-Sardinia
0
0
0
0
0
East-Liguria
2
0
0
0
0
Inland-Sardinia
0
0
0
0
0
North-Apulia
6
0
0
0
0
Sicily
1
7
3
0
0
South-Apulia
1
0
67
0
0
Umbria
0
0
0
18
0
West-Liguria
0
0
0
0
15
Misclassification Rate = 0.0789
> area.rda = rda(Area.name~.,data=OA.train)
> pred.rda = predict(area.rda,newdata=OA.test)
> misclassification(pred.qda$class,OA.test$Area.name)
Table of Misclassification (row = predicted, col = actual)
y
fit
Calabria Coastal-Sardinia East-Liguria Inland-Sardinia
Calabria
14
0
0
0
Coastal-Sardinia
0
11
0
1
East-Liguria
0
0
16
0
Inland-Sardinia
0
0
0
21
North-Apulia
0
0
0
0
Sicily
2
0
0
0
South-Apulia
1
1
0
1
Umbria
0
0
0
0
West-Liguria
0
0
1
0
y
fit
North-Apulia Sicily South-Apulia Umbria West-Liguria
Calabria
0
1
0
0
0
Coastal-Sardinia
0
0
0
0
0
East-Liguria
2
0
0
0
0
Inland-Sardinia
0
0
0
0
0
North-Apulia
6
0
0
0
0
Sicily
1
7
3
0
0
South-Apulia
1
0
67
0
0
Umbria
0
0
0
18
0
West-Liguria
0
0
0
0
15
Misclassification Rate = 0.0789
 Same as QDA!
RDA help file from the package klaR
309
We clearly need to consider changing the regularization parameters  if we are going to see
any benefit from using RDA vs. LDA/QDA.
> area.rda = rda(Area.name~.,data=OA.train,lambda=.1,gamma=0)
> pred.rda = predict(area.rda,newdata=OA.test)
> misclassification(pred.rda$class,OA.test$Area.name)
Table of Misclassification
(row = predicted, col = actual)
y
fit
Calabria Coastal-Sardinia East-Liguria Inland-Sardinia
Calabria
16
0
0
0
Coastal-Sardinia
0
12
0
1
East-Liguria
0
0
16
0
Inland-Sardinia
0
0
0
21
North-Apulia
0
0
0
0
Sicily
0
0
0
0
South-Apulia
1
0
0
1
Umbria
0
0
0
0
West-Liguria
0
0
1
0
y
fit
North-Apulia Sicily South-Apulia Umbria West-Liguria
Calabria
0
1
0
0
0
Coastal-Sardinia
0
0
0
0
0
East-Liguria
2
0
0
0
0
Inland-Sardinia
0
0
0
0
0
North-Apulia
6
0
0
0
0
Sicily
2
7
4
0
0
South-Apulia
0
0
66
0
0
Umbria
0
0
0
18
0
West-Liguria
0
0
0
0
15
Misclassification Rate = 0.0684
We can write a simple function for looking through a 𝑘 × 𝑘 grid of (𝜆, 𝛾) of
combinations in order to find “optimal” values for these tuning parameters. The
function assumes the response is in the first column of both the training and test data
frames.
find.lamgam = function(formula,train,test,k=5){
lambda = seq(0,1,length=k)
gamma = seq(0,1,length=k)
mcr = rep(0,as.integer(k^2))
ntest = dim(test)[1]
lg.grid = expand.grid(lambda=lambda,gamma=gamma)
for (i in 1:as.integer(k^2)){
temp = rda(formula,data=train,lambda=lg.grid[i,1],gamma=lg.grid[i,2])
pred = predict(temp,newdata=test)$class
numinc = ntest - sum(diag(table(pred,test[,1])))
mcr[i] = numinc/ntest
}
cbind(lg.grid,mcr)
}
310
> find.lamgam(formula=Area.name~.,train=OA.train,test=OA.test,k=10)
lambda
gamma
mcr
1
0.0000000 0.0000000 0.07894737
2
0.1111111 0.0000000 0.06842105
3
0.2222222 0.0000000 0.07894737
4
0.3333333 0.0000000 0.07368421
5
0.4444444 0.0000000 0.06842105
6
0.5555556 0.0000000 0.06842105
7
0.6666667 0.0000000 0.06315789
8
0.7777778 0.0000000 0.06842105
9
0.8888889 0.0000000 0.08421053
10 1.0000000 0.0000000 0.11052632
11 0.0000000 0.1111111 0.08421053
12 0.1111111 0.1111111 0.08947368
. . .
. . .
. . .
. . .
96 0.5555556 1.0000000 0.18421053
97 0.6666667 1.0000000 0.18421053
98 0.7777778 1.0000000 0.18421053
99 0.8888889 1.0000000 0.17894737
100 1.0000000 1.0000000 0.17894737
It appears that 𝜆 = .6667 and  is approximately optimal for this train/test combination.
We can use the partimat function to visualize the classification boundaries in a pairwise basis
using two of the 𝑋𝑖′ 𝑠 at a time. The classification error rates are for discriminant analysis using
the two displayed variables only, thus the usefulness of these displays is questionable.
> partimat(Area.name~.,data=OA.train,method="lda",nplots.hor=4,nplots.ver=4)
Individual plot from LDA
311
> partimat(Area.name~.,data=OA.train,method="qda",nplots.hor=4,nplots.ver=4)
Individual plot from QDA
312
> partimat(Area.name~.,data=OA.train,method="rda",lambda =
.6667,gamma=0,nplots.hor=4,nplots.ver=4)
An individual plot from RDA
> partimat(Area.name~.,data=OA.train,method=”sknn”,nplots.hor=4,nplots.ver=4)
313
> partimat(Area.name~.,data=OA.train,method=”naiveBayes”,nplots.hor=4,nplots.ver=4)
It is interesting to contrast the decision boundaries between the different methods of
classification we have discussed so far.
314