advertisement

Classification by nearest neighbor strategy. Lazy Learning via kNN algorithm. Nearest Neighbor Algorithm. Given a dataset of cancer biopsies, each with several features, we want to train a classifier so that it can figure out whether a specific sample(s) represent a benign or cancerous case. Classification via nearest neighbor approach has following characteristics. Pros: (a) simple and effective, (b) makes no assumption of any distribution, and (c) fast training phase Cons: (a) Does not produce any model (no abstraction, hence Lazy Learning), (b) classification process is slow, (c) needs a lot of memory, (d) nominal features and missing data need additional scrutiny. Training dataset: Examples of classification into several categories. For each record in the test dataset, kNN identifies 𝑘 known samples nearest in similarities. Unlabeled test is assigned the category label of the majority of the k nearest neighbors. 𝑘 = 5 kNN environment The distance function between the target object and an accepted vector sample (the ‘geographical’ distance) could be measured in many ways. By default, though, it is Euclidean distance metric. Distance between (𝑥1 , 𝑦1 ) and (𝑥2 , 𝑦2 ): Euclidean: √(𝑥1 − 𝑥2 )2 + (𝑦1 − 𝑦2 )2 Manhattan: |𝑥1 − 𝑥2 | + |𝑦1 − 𝑦2 | Minkowski: (|𝑥1 − 𝑥2 |𝑞 + |𝑦1 − 𝑦2 |𝑞 )1/𝑞 We are going to use Wisconsin breast-cancer data available in our website. R uses Euclidean distance metric. Major issue in kNN classification approach is the value of 𝑘. If 𝑘 is large, one is basically focusing on majority decision, regardless of which clusters are nearest to it. If 𝑘 is too small, noisy data or outliers may control the situation. This is not acceptable either. We need 𝑘 that avoids two extremes. Cluster separation boundary Low k. For any k, we could encounter misclassifications: False Positive, and False Negatives. 𝑘 needs to be chosen so that these misclassifications tend to be as low as possible. First we read the data as a csv file from our website, and figure out its structure. > md=read.csv("http://web.cs.sunyit.edu/~sengupta/num_maths/wisc_bc_data.csv"). Now let’s look at its structure. We use str() operation to reveal the data structure md. > str(md) It seems there are 32 variables (10 × 3 + 2) over 569 samples. ID we do not need. The Diagnosis is our focus, and it is a factor with 2 levels: B (Benign), M (Malignant). Note that all features are numeric. We now filter the data, and need to prepare it further. > table(md$diagnosis) B M 357 212 Majority is Benign. Let us use proportion on the diagnosis factor to find what proportion leads to benign. > > round(prop.table(table(md$diagnosis))*100, digits=1) B M 62.7 37.3 We are going to focus on three features of our data. You have to try all of them or some other combination of features. >summary(md[c("radius_mean","area_mean","smoothness_mean")]) radius_mean Min. : 6.981 1st Qu.:11.700 Median :13.370 Mean :14.127 3rd Qu.:15.780 Max. :28.110 > area_mean Min. : 143.5 1st Qu.: 420.3 Median : 551.1 Mean : 654.9 3rd Qu.: 782.7 Max. :2501.0 smoothness_mean Min. :0.05263 1st Qu.:0.08637 Median :0.09587 Mean :0.09636 3rd Qu.:0.10530 Max. :0.16340 Everything is okay, except our data need to be polished further. kNN is very sensitive to measurement scale of the input features. In order to make sure our classifier is not comparing apples with oranges, we are going to rescale each of our features (except the diagnosis, that is a factor) with a function > normalize <- function(x) { + return((x-min(x))/(max(x)-min(x))) +} > > > > # lets try on a vector q=c(12,56,3,2.7, 9.8,12.6,-5.3) q=normalize(q) q [1] 0.2822186 1.0000000 0.1353997 0.1305057 0.2463295 0.2920065 0.0000000 The minimum and the maximum are at 0.0 and at 1.0. We are going to normalize these numerical data, and make a data frame with the normalized attributes. To apply to our data, we are going to call lapply on column 2:31 using normalize function. > #redefine our data frame md by normalizing first all the numerical > # columns in our dataframe. We might need them later. > #apply normalize function to all our data frame columns. We need to apply lapply. > md=as.data.frame(lapply(md[,2:31], normalize)) Anytime you want to create a new data frame that might contain mixed data you need to make the call as.data.frame. Let’s pick the three attributes of our interest and put them in a new data frame qq. qq=data.frame(md$area_mean,md$radius_mean,md$smoothness_mean) > summary(qq) Check every attribute is bounded between 0 and 1. We are now ready to define our training set and our testing set. We are going to split the data into two portions: mdn_train, and mdn_test. The first 469 rows (samples) would comprise the training set, and the rest 100 rows the mdn_test > > > > train=qq[1:469,] # First 469 rows for training test=qq[470:569,] # Last 100 rows for our testbed train_labels=q1[1:469,1] #Our training classification factors test_labels=q1[470:569] # Our target sample factors The classifier would change test_labels column if necessary. The kNN package is in “class” that we need to install. You may need to make install.packages(“class”) first from a mirror site. And then call library(class) > library(class) > p=knn(train, test, cl=train_labels, k=3) > Thank Heaven. It worked. Now, we need to do some tidying it up. For this we need a package called “gmodels” , which will provide us with cross tabulation results. > library("gmodels") > CrossTable(x=test_labels, y=p, prop.chisq=FALSE) The final output emerges now. We don’t want Chi square results. There were 100 test data. The top-left data (True Negative) score is 56/61 (we suggested 61 of them “benign”, the classifier picked 56 of them benign). The top-right data is False Positive, meaning 5 of the 61 were misclassified under our scheme. The bottom left box implies total number of False Negatives, 10 out of 39 were wrongly classified as Malignant. The bottom right box shows True Positive stats: 29 out of 39 were correctly labelled. Perhaps we would get different result using different k values. Find out how our misclassification results change by selecting different k values. We try with k=7 > p=knn(train, test, cl=train_labels,k=7) > CrossTable(x=test_labels, y=p, prop.chisq=FALSE) Now we are making it slightly worse. Look at the cross tabulation figures. The top row didn’t change, but now we made 1 more misclassification in the False Negative area. Now you know how you have to approach the problem.