Classification Modeling
K-Nearest Neighborhood
By
Subhasis Dasgupta
Asst Professor
Praxis Business School, Kolkata
Why KNN
It is the simplest machine learning algorithm
Very few parameters are required to be specified
Quite fast in operation
No statistical assumptions are required
Concept
K-NN classification classifies instances on the basis of their similarity to
instances in the training data
Usually some distance measure is required to measure similarity
There are many distance measures available
Euclidian distance
Squared Euclidian distance
Mahalobonis distance
Cosine similarity measure
1-correlation coefficient etc
It uses simple “voting” concept to classify an instance to either one class or
other
Algorithm
Have a N-dimensional feature space from the training dataset
Put the new instance in that feature space
Measure the distance of the unclassified instance from K instances in the
trained feature space
The closer the new instance to existing classified instances, the more
similar it is to the existing classified instances
Algorithm
If K=1, only one classified instance decides the class of unclassified new
instance
If K > 1, “voting” occurs among K classified instances
The new instance is put to that class where maximum voting occurs
Example
Classified Instances
Example
Classified Instances with one unclassified instance
Unclassified instance
Example
Example
K=1
New instance is classified
as class RED
Example
K=4
The new instance is
closer to 3 RED
classes and 1 BLUE
class. Hence, voting
takes place and there
are 3 RED votes and
1 BLUE vote. Thus
the new instance is
classified as RED
Voronoi Diagram
Given a set of points, Voronoi
diagram describe the areas that
are nearest to any given point
These areas act as zone of control
Voronoi Diagram
Decision boundaries are formed by
the subset of the Voronoi
diagram of the training data
Each line segment is equidistant
between two points of opposite
class
The more examples that are stored
the more fragmented and
complex the decision
boundaries can be
Comments
Advantages
❑
Learning is vary simple and intuitive
❑
Flexible decision boundaries
❑
With properly chosen K and distance measures, accuracy improves
greatly
Disadvantages
❑
Inclusion of irrelevant variable lowers accuracy of classification
❑
Primarily, variables should be numeric in nature (though categorical
variables, after some form of transformations can be used)
❑
Gets affected by the scales of variable (normalization might be
helpful in such situations)
❑
Choosing optimum value of K is always an issue
❑
K-NN algorithm gets affected by high dimensional space (50 or 100
different variables)
❑
Correlated features have high impact on accuracy
0
