Data Mining
Principal Component Analysis
Clustering1
CS 584 :: Fall 2025
Ziwei Zhu
Department of Computer Science
George Mason University
Part of slides is from Drs. Tan, Steinbach and Kumar.
Part of slides is from Dr. Theodora Chaspari.
1
• HW3 is due on 11/16, start now!
• Will explain the solution of Midterm exam next
week.
2
Outline
• Introduction
• Partitional Clustering: K-means
• Hierarchical Clustering
• Density-based Clustering
3
Classification
• Supervised learning (using data with labels)
• Predictive task (predicting labels)
𝒟 !"#$% = { 𝒙𝟏 , 𝑦' , … , (𝒙𝑵 , 𝑦) )}
𝒟 !*+! = { 𝒙𝟏 , 𝑦' , … , (𝒙𝑴 , 𝑦- )}
4
What is Cluster Analysis
Given a set of objects, place them in groups such that the
objects in the same group are similar to each other, and
different from the objects in other groups. (without labels)
5
What is Cluster Analysis
Given a set of objects, place them in groups such that the
objects in the same group are similar to each other, and
different from the objects in other groups. (without labels)
• Unsupervised learning (using data without labels)
• Descriptive task
6
Applications of Clustering
• Understanding: finding
patterns/structure/subpopulations in data
Cluster documents and
discover topics
• Summarization: reducing
the size of datasets
User portrait
7
Notion of a Cluster can be Ambiguous
How many clusters?
8
Notion of a Cluster can be Ambiguous
9
Types of Clustering
10
Types of Clustering
• Partitional Clustering
• Hierarchical Clustering
• Density-based Clustering
• And more …
11
Outline
• Introduction
• Partitional Clustering: K-means
• Hierarchical Clustering
• Density-based Clustering
12
Types of Clustering
• Partitional Clustering
•
A division of data objects into non-overlapping subsets
(clusters)
13
K-means Clustering
• A division of data objects into non-overlapping
subsets (clusters)
14
K-means Clustering
• A division of data objects into non-overlapping subsets
(clusters)
• Number of clusters, K, must be specified
(hyperparameter)
15
K-means Clustering
• A division of data objects into non-overlapping subsets
(clusters)
• Number of clusters, K, must be specified (hyperparameter)
• Each cluster is associated with a centroid (center
point)
16
K-means Clustering
• A division of data objects into non-overlapping subsets
(clusters)
• Number of clusters, K, must be specified (hyperparameter)
• Each cluster is associated with a centroid (center point)
• Each point is assigned to the cluster with the closest
centroid
17
K-means Clustering
• A division of data objects into non-overlapping subsets
(clusters)
• Number of clusters, K, must be specified (hyperparameter)
• Each cluster is associated with a centroid (center point)
• Each point is assigned to the cluster with the closest
centroid
• A simple iterative algorithm
18
K-means Clustering
Each centroid represents a cluster
For each data point i:
calculate distances to K centroids: 𝑑!, 𝑑", …, 𝑑#
assign point i to the cluster j with minimum distance 𝑑$
19
K-means Clustering
For each cluster:
calculate the new centroid by averaging over all points in it
20
Example of K-means Clustering
Iteration 1
3
2.5
2
y
1.5
1
0.5
0
-2
-1.5
-1
-0.5
0
0.5
1
1.5
2
x
21
Example of K-means Clustering
Iteration 1
3
2.5
2
y
1.5
1
0.5
0
-2
-1.5
-1
-0.5
0
0.5
1
1.5
2
x
22
Example of K-means Clustering
Iteration 2
3
2.5
2
y
1.5
1
0.5
0
-2
-1.5
-1
-0.5
0
0.5
1
1.5
2
x
23
Example of K-means Clustering
Iteration 2
3
2.5
2
y
1.5
1
0.5
0
-2
-1.5
-1
-0.5
0
0.5
1
1.5
2
x
24
Example of K-means Clustering
Iteration 3
3
2.5
2
y
1.5
1
0.5
0
-2
-1.5
-1
-0.5
0
0.5
1
1.5
2
x
25
Example of K-means Clustering
Iteration 4
3
2.5
2
y
1.5
1
0.5
0
-2
-1.5
-1
-0.5
0
0.5
1
1.5
2
x
26
Example of K-means Clustering
Iteration 5
3
2.5
2
y
1.5
1
0.5
0
-2
-1.5
-1
-0.5
0
0.5
1
1.5
2
x
27
Example of K-means Clustering
Iteration 6
3
2.5
2
y
1.5
1
0.5
0
-2
-1.5
-1
-0.5
0
0.5
1
1.5
2
x
28
K-means Clustering: Formalization
Sum of Squared Error (SSE)
SSE
)
SSE = - 𝒙% − 𝜇/(𝒙!)
%.'
3
3
29
K-means Clustering: Formalization
)
SSE = - 𝒙% − 𝜇/(𝒙!)
%.'
3
3
𝑥'
𝑥3
𝑥6
𝜇'
𝑥4
𝑥5
𝜇3
30
K-means Clustering: Formalization
)
min 𝐽 = - 𝒙% − 𝜇/(𝒙!)
/
•
•
%.'
3
3
SSE
Step 0: initialize 𝜇 to some values
Step 1: Assume the current value of 𝜇 fixed, minimize 𝐽 over 𝐴,
which leads to the following cluster assignment rule:
𝐴 𝒙% = argmin 𝒙% − 𝜇& ""
&
Step 2: Assume the current value of 𝐴 fixed, minimize 𝐽 over 𝜇,
which leads to the following rule to update the prototypes of the
clusters:
∑'! ∈)" 𝑥$
𝑋& = 𝑥$ 𝐴 𝑥$ = 𝜇& , 𝜇& =
|𝑋& |
• Step 3: Determine whether to stop or return to Step 1.
•
32
K-means Clustering
• The centroid 𝜇7 is the mean of data points assigned
to the cluster k, hence the name K-means clustering.
33
K-means Clustering
• The centroid 𝜇7 is the mean of data points assigned
to the cluster k, hence the name K-means clustering.
• Most of the convergence happens in the first few
iterations.
•
Often the stopping condition is changed to ‘Until relatively
few points change clusters’
34
K-means Clustering
• The centroid 𝜇7 is the mean of data points assigned
to the cluster k, hence the name K-means clustering.
• Most of the convergence happens in the first few
iterations.
•
Often the stopping condition is changed to ‘Until relatively
few points change clusters’
• (Crucial limitation) Initial centroids are often chosen
randomly.
• Clusters produced can vary from one run to another.
•
No global optimum of SSE is guaranteed.
35
K-means Clustering
36
Importance of Choosing Initial Centroids
Iteration 1
Iteration 2
Iteration 3
2.5
2.5
2.5
2
2
2
1.5
1.5
1.5
y
3
y
3
y
3
1
1
1
0.5
0.5
0.5
0
0
0
-2
-1.5
-1
-0.5
0
0.5
1
1.5
2
-2
-1.5
-1
-0.5
x
0
0.5
1
1.5
2
-2
Iteration 4
Iteration 5
2.5
2
2
2
1.5
1.5
1.5
1
1
1
0.5
0.5
0.5
0
0
0
-0.5
0
x
0.5
1
1.5
2
0
0.5
1
1.5
1
1.5
2
y
2.5
y
2.5
y
3
-1
-0.5
Iteration 6
3
-1.5
-1
x
3
-2
-1.5
x
-2
-1.5
-1
-0.5
0
x
0.5
1
1.5
2
-2
-1.5
-1
-0.5
0
0.5
x
37
2
Importance of Choosing Initial Centroids
38
Solutions to Initial Centroids Problem
• Multiple runs
• Helps, but random
• Use some strategy to select the k initial centroids
• Select most widely separated (e.g., K-means++ )
• Use hierarchical clustering to determine initial
centroids
39
K-means++
Select most widely separated points as initial centroids
40
K-means++
• Assume a set of five points: 𝒙' , 𝒙3 , 𝒙4 , 𝒙5 , 𝒙6
• We want to pick K=3 initial centroids
41
K-means++
1st iteration
•
•
Selected centroid set: 𝒄! (randomly selected before the loop,
assume 𝒙*)
Remaining points: 𝒙!, 𝒙", 𝒙+, 𝒙,
42
K-means++
1st iteration
•
•
•
Selected centroid set: 𝒄! (randomly selected before the loop,
assume 𝒙*)
Remaining points: 𝒙!, 𝒙", 𝒙+, 𝒙,
Calculate distances:
𝑑 𝒙!, 𝒄! ,
𝑑 𝒙", 𝒄! ,
𝑑 𝒙+, 𝒄! ,
𝑑(𝒙,, 𝒄!)
,
Σ=7
𝑑 𝒙& , 𝒄!
&-!
43
K-means++
1st iteration
•
•
•
Selected centroid set: 𝒄! (randomly selected before the loop,
assume 𝒙*)
Remaining points: 𝒙!, 𝒙", 𝒙+, 𝒙,
Calculate distances:
𝑑 𝒙!, 𝒄! ,
𝑑 𝒙", 𝒄! ,
𝑑 𝒙+, 𝒄! ,
𝑑(𝒙,, 𝒄!)
,
Σ=7
•
𝑑 𝒙& , 𝒄!
&-!
Calculate the probability distribution, and randomly select a
new centroid based on the distribution:
. 𝒙# ,𝒄# . 𝒙$ ,𝒄# . 𝒙% ,𝒄# . 𝒙& ,𝒄#
[ 2 , 2 , 2 , 2 ]
Assume we select 𝒙, as new centroid 𝒄"
44
K-means++
2nd iteration
•
•
Selected centroid set: 𝒄!, 𝒄"
Remaining points: 𝒙!, 𝒙", 𝒙+
45
K-means++
2nd iteration
•
•
•
Selected centroid set: 𝒄!, 𝒄"
Remaining points: 𝒙!, 𝒙", 𝒙+
Calculate distances:
𝑑 𝒙!, 𝒄! ,
𝑑 𝒙", 𝒄! ,
𝑑 𝒙!, 𝒄" ,
𝑑 𝒙 " , 𝒄" ,
𝑑 𝒙+, 𝒄!
𝑑 𝒙 + , 𝒄"
46
K-means++
2nd iteration
•
•
•
Selected centroid set: 𝒄!, 𝒄"
Remaining points: 𝒙!, 𝒙", 𝒙+
Calculate distances:
𝑑 𝒙!, 𝒄! ,
𝑑 𝒙", 𝒄! ,
𝑑 𝒙!, 𝒄" ,
𝑑 𝒙 " , 𝒄" ,
𝑑 𝒙+, 𝒄!
𝑑 𝒙 + , 𝒄"
47
K-means++
2nd iteration
Selected centroid set: 𝒄!, 𝒄"
Remaining points: 𝒙!, 𝒙", 𝒙+
Calculate distances:
𝑑 𝒙!, 𝒄! ,
𝑑 𝒙", 𝒄! ,
𝑑 𝒙+, 𝒄!
𝑑 𝒙!, 𝒄" ,
𝑑 𝒙 " , 𝒄" ,
𝑑 𝒙 + , 𝒄"
assume 𝑑 𝒙!, 𝒄! < 𝑑 𝒙!, 𝒄" , 𝑑 𝒙", 𝒄! > 𝑑 𝒙", 𝒄" ,
𝑑 𝒙#, 𝒄! < 𝑑 𝒙#, 𝒄"
Σ = 𝑑 𝒙!, 𝒄! + 𝑑 𝒙", 𝒄" + 𝑑 𝒙+, 𝒄!
•
•
•
48
K-means++
2nd iteration
Selected centroid set: 𝒄!, 𝒄"
Remaining points: 𝒙!, 𝒙", 𝒙+
Calculate distances:
𝑑 𝒙!, 𝒄! ,
𝑑 𝒙", 𝒄! ,
𝑑 𝒙+, 𝒄!
𝑑 𝒙!, 𝒄" ,
𝑑 𝒙 " , 𝒄" ,
𝑑 𝒙 + , 𝒄"
assume 𝑑 𝒙!, 𝒄! < 𝑑 𝒙!, 𝒄" , 𝑑 𝒙", 𝒄! > 𝑑 𝒙", 𝒄" ,
𝑑 𝒙#, 𝒄! < 𝑑 𝒙#, 𝒄"
Σ = 𝑑 𝒙!, 𝒄! + 𝑑 𝒙", 𝒄" + 𝑑 𝒙+, 𝒄!
• Calculate the probability distribution, and randomly select a
new centroid based on the distribution:
•
•
•
. 𝒙 ,𝒄
. 𝒙 ,𝒄
. 𝒙 ,𝒄
[ 2# # , 2$ $ , 2% # ]
Assume we select 𝒙" as new centroid 𝒄+
49
K-means++
• Assume a set of five points: 𝒙' , 𝒙3 , 𝒙4 , 𝒙5 , 𝒙6
• We want to pick K=3 initial centroids
Now, we have 𝒄' = 𝒙6 , 𝒄3 = 𝒙5 , 𝒄4 = 𝒙3
50
K-means++
Select most widely separated points as initial centroids
51
How to choose k?
52
Limitations of K-means
• K-means has problems when clusters are of differing
•
•
•
Sizes
Densities
Non-globular shapes
• K-means has problems when the data contains
outliers and noise.
•
Mean calculation is sensitive to outliers and noise
53
Limitations of K-means: Differing Sizes
Original Points
K-means (3 Clusters)
54
Limitations of K-means: Differing Density
Original Points
K-means (3 Clusters)
55
Limitations: Non-globular Shapes
Original Points
K-means (2 Clusters)
56
HW3 – regarding clustering evaluation
All three metrics should be >0.48
58
Outline
• Introduction
• Partitional Clustering: K-means
• Hierarchical Clustering
• Density-based Clustering
59
Hierarchical Clustering
• Produces a set of nested clusters organized as a
hierarchical tree
• Can be visualized as a dendrogram
A tree like diagram that records the sequences of
merges or splits
•
5
6
0.2
4
3
4
2
0.15
5
2
0.1
1
0.05
0
3
1
3
2
5
4
1
6
60
Advantages of Hierarchical Clustering
• Do not have to assume any particular number of
clusters
Any desired number of clusters can be obtained by
‘cutting’ the dendrogram at the proper level
•
5
6
0.2
4
3
4
2
0.15
5
2
0.1
1
0.05
0
3
1
3
2
5
4
1
6
61
Hierarchical Clustering
Two main types of hierarchical clustering
• Agglomerative (bottom-up):
•
•
Start with the points as individual clusters
At each step, merge the closest pair of clusters until only one
cluster left
• Divisive (top-down):
62
Hierarchical Clustering
Two main types of hierarchical clustering
• Agglomerative (bottom-up):
• Divisive (top-down):
• Start with one, all-inclusive cluster
• At each step, split a cluster until each cluster
contains an individual point
• E.g., apply K-means with K=2 recursively to split
existing clusters into smaller ones
63
Bottom-up Clustering Algorithm
Key idea: successively merge closest clusters
64
Bottom-up Clustering Algorithm
Initialization: Start with each sample being a cluster
1
5
1
3
2
4
2
3
4
5
D(1,2)
D(1,3)
D(1,4)
D(1,5)
1
D(2,3)
D(2,4)
D(2,5)
2
D(3,4)
D(3,5)
3
D(4,5)
4
5
Assume among these 10 distances, D(2,3) is
the minimum
1
2
3
4
5
65
Bottom-up Clustering Algorithm
Initialization: Start with each sample being a cluster
1
5
1
3
2
4
{2,3}
4
5
D(1,{2,3})
D(1,4)
D(1,5)
1
D({2,3},4) D({2,3},5)
{2, 3}
D(4,5)
4
5
Assume among these 6 distances, D(4,5) is the
minimum
1
2
3
4
5
66
Bottom-up Clustering Algorithm
Among current clusters, find the two with minimum distance
1
3
2
{4,5}
D(1,{2,3})
D(1,{4,5})
1
D({2,3},{4,5})
{2,3}
{4,5}
Assume among these 3 distances, D(1,{2,3}) is
the minimum
5
1
{2,3}
4
1
2
3
4
5
67
Bottom-up Clustering Algorithm
Among current clusters, find the two with minimum distance
{1,2,3}
{1,2,3}
{4,5}
D({1,2,3},{4,5})
5
1
3
{4,5}
2
4
1
2
3
4
5
68
Bottom-up Clustering Algorithm
Among current clusters, find the two with minimum distance
{1,2,3,4,5}
{1,2,3,4,5}
5
1
3
2
4
1
2
3
4
5
69
Bottom-up Clustering Algorithm
The question is “How do we calculate the proximity
matrix?”
5
1
3
2
{1,2,3}
4
{1,2,3}
{4,5}
{4,5}
D({1,2,3},{4,5})
70
How to Define Inter-Cluster Distance
71
MIN
Distance of two clusters is based on the two closest
points in the different clusters.
Example:
Proximity/Distance Matrix:
0.16
0.16
72
MIN
1
3
5
5
0.2
2
1
2
3
4
4
Nested Clusters
0.15
6
0.1
0.05
0
3
6
2
5
4
1
Dendrogram
73
Advantage of MIN
• Can handle non-globular shapes
Original Points
Six Clusters
75
Limitation of MIN
• Sensitive to noise
Original Points
Two Clusters
76
MAX
Distance of two clusters is based on the two most
distant points in the different clusters.
Example:
Proximity/Distance Matrix:
0.16
0.16
77
MAX
1
3
5
5
0.2
2
1
2
3
4
4
Nested Clusters
0.15
6
0.1
0.05
0
3
6
2
5
4
1
Dendrogram
78
Advantage of MAX
• Less susceptible to noise
Original Points
Two Clusters
79
Limitation of MAX
• prefer globular clusters
• Tends to break large clusters
Original Points
Two Clusters
80
Group Average
Distance of two clusters is the average of pairwise distance
between points in the two clusters.
• Compromise between MIN and MAX
• Advantage: less susceptible to noise
• Limitation: prefer globular clusters
81
Hierarchical Clustering: Comparison
1
3
5
5
4
2
2
1
2
1
3
6
MIN
MAX
5
5
2
3
3
4
4
6
1
4
1
5
2
5
4
2
3
3
6
Group Average
1
4
82
Outline
• Introduction
• Partitional Clustering: K-means
• Hierarchical Clustering
• Density-based Clustering
83
Density Based Clustering
Clusters are regions of high density that are separated from
regions of low density.
Original Points
Clusters (dark blue points
are noise)
84
DBSCAN: Density-based spatial clustering of applications
with noise.
• Density = number of points within a specified radius
(denoted as Eps)
• A point is a core point if it has at least a specified number
of points (denoted as MinPts) within Eps
•
•
These are points that are at the interior of a cluster
Counts the point itself
85
DBSCAN: Density-based spatial clustering of applications
with noise.
• Density = number of points within a specified radius
(denoted as Eps)
• A point is a core point if it has at least a specified number
of points (denoted as MinPts) within Eps
•
•
These are points that are at the interior of a cluster
Counts the point itself
• A border point is not a core point, but is in the
neighborhood of a core point (i.e., within Eps)
86
DBSCAN: Density-based spatial clustering of applications
with noise.
• Density = number of points within a specified radius
(denoted as Eps)
• A point is a core point if it has at least a specified number
of points (denoted as MinPts) within Eps
•
•
These are points that are at the interior of a cluster
Counts the point itself
• A border point is not a core point, but is in the
neighborhood of a core point (i.e., within Eps)
• A noise point is any point that is not a core point or a
border point
87
DBSCAN
MinPts=7
C
B
A
Eps
88
DBSCAN
MinPts=7
89
DBSCAN
Original Points
Point types: core, border,
and noise
Eps = 10, MinPts = 4
90
DBSCAN
Form clusters using core points, and assign border
points to one of its neighboring clusters
1: Label all points as core, border, or noise points.
91
DBSCAN
Form clusters using core points, and assign border
points to one of its neighboring clusters
2: Eliminate noise points.
92
DBSCAN
Form clusters using core points, and assign border
points to one of its neighboring clusters
3: Link all core points within a distance Eps of each
other.
93
DBSCAN
Form clusters using core points, and assign border
points to one of its neighboring clusters
4: Make each group of connected core points into a
separate cluster.
94
DBSCAN
Form clusters using core points, and assign border
points to one of its neighboring clusters
5: Assign each border point to one of the clusters of
its associated core points
95
When DBSCAN Works Well
• Can handle clusters of different shapes and sizes
• Resistant to noise
Original Points
Clusters (dark blue points
are noise)
96
When DBSCAN Does NOT Work Well
• Varying densities
97
Scikit-learn: many options
98
What we have learnt so far
• Partitional Clustering: K-means
• Hierarchical Clustering: bottom-up approach
• Density-based Clustering: DBSCAN
Next big question: How to rigorously
evaluate clustering results?
99