Session 2: Clustering
Prof. Yash Babar
Business Analytics for
Technology Management
(BATM), Term 3,
Hyderabad, Co27
Why Cluster?
Cluster Analysis
(“data
segmentation”) is an
exploratory method
for identifying
homogenous groups
(“clusters”) of records
Similar records
should belong to the
same cluster
Dissimilar records
should belong to
different clusters
Let’s talk about
John Snow
Example:
Geographical
Clustering
Why is this a clustering
problem?
● Treat coordinates as
dimensions
● Spatially cluster disease
occurence
● Identify clusters of
deaths → Identify the
source
Clustering: Basic Ideas
● Organizing data points/objects (e.g., customers) into homogeneous (and,
hopefully, meaningful) groups/clusters
○ “Unsupervised” learning technique
● Desired properties of clustering result:
○ High intra-similarity, i.e., any two data points / objects that are assigned into
the same cluster should exhibit similarity to each other
○ Low inter-similarity, i.e., any two data points / objects that are assigned into
different clusters should not be very similar to each other, otherwise they
should have been assigned into the same cluster
● Helps to gain insights into your data
○ Dimensionality Reduction: Instead of trying to look at the entire dataset (e.g., a
huge number of customers), you can inspect the representative data
groups/clusters (e.g., a small number of groups, into which your data can be
arranged most naturally)
○ Usually a useful precursor for additional, deeper analyses
○ Many applications!
Example: Fitting the Troops (from Data
Mining Techniques by Berry & Linoff)
● The US army recently commissioned a study on how
to redesign the uniforms of female soldiers. The
army’s goal is to reduce the number of different
uniform sizes that have to be kept in inventory while
still providing each soldier with well-fitting khakis.
● Researchers Ashdown and Paal @ Cornell University
designed a new set of sizes based on the actual
shapes of women in the army. Unlike traditional
clothing size systems, the new sizes are not an
ordered set of graduated sizes where all dimensions
increase together.
● Instead, they came up with sizes that fit particular
body types (e.g., short-legged, small-waisted,
large-busted women with long torsos, average arms,
broad shoulders, and skinny necks).
When you cluster, remember
Objective: align the cluster analysis
with the business objective(s).
● What entities you are clustering?
● Based on what attributes?
● In order to achieve what?
How to cluster?
● Since clustering is based on the features available in the data, if the data is 3D or less, clustering
would be very straightforward
● In low-dimensional spaces, clusters can even emerge from simple plots
Similarity with
multi-dimensional
data
UG Business Programs Universities
Clustering.xls
●
●
Data for 25
undergraduate
programs at business
schools in US
universities in 1995.
○ Average SAT
scores,
percentage of the
incoming class in
the top 10% of
their high school
class, acceptance
rate, student to
faculty ratio,
annual expenses,
and graduation
rate.
This dataset excludes
university image
variables (student
satisfaction, employer
satisfaction, deans’
opinions)
Similarity across one
Dimension
Distance
across Two
Dimensions
Dist(UMich, UCB) =
|Accept1 - Accept2|
((Accept1- Accept2)2 +
(SAT1 - SAT2)2)1/2
|SAT1 - SAT2|
A particular kind of distance:
Euclidean
Distance across all
dimensions
Euclidean Distance
● 6-dimensional Euclidean distance between Caltech and
Cornell:
Sqrt [ (1415-1280)2 + (100-83)2 + (25-33)2 + (6-13)2 +
(63575-21864)2 + (81-90)2] = 41,711.22
Standardize when there are multiple
variables
Euclidean distance is influenced by the units of the different
measurements
Solution: standardize (or normalize) each variable before
measuring distances
Standardize vs Normalize
Standardize
Normalize
Distance Matrix
(Normalized
Euclidean)
Lots of other distance metrics
Statistical (Mahalanobis) distance
uses correlation matrix
Manhattan distance
Matching-type metrics for categorical data
(next slide)
Distances for Binary Data
● Similarity-based metrics based on 2x2 table of counts
● Binary Euclidean Distance: (b+c)/(a+b+c+d)
● Simple matching Coefficient: (a+d)/(a+b+c+d)
● Jaquard’s coefficient: d/(b+c+d)
Why would we want to do this?
● How can clustering help a
prospective applicant?
● How can clustering help a
business school dean?
● How do the different
dimensions relate to the
different interest groups.
K-Means Clustering
● Gives Predetermined number (K) of
non-overlapping clusters
○ Requires specifying # clusters
● Assign records to each of the clusters in
order to improve homogeneity within
group
○ Clusters are homogeneous yet
dissimilar to other clusters
● Need measures of within-cluster
similarity (homogeneity) and
between-cluster similarity
● No hierarchy (no dendrogram)!
End-product is final cluster memberships
● Computationally cheap
○ Useful for large datasets
K-means clustering
Algorithm minimizes within-cluster variance
(heterogeneity)
For a user-specified value of K, partition dataset into K initial
clusters (next slide).
For each record, assign it to cluster with closest centroid
Re-calculate centroids for the
“losing” and “receiving” clusters.
Can be done
after reassignment of each record, or
after one complete pass through all
records (cheaper)
Repeat Steps 2-3 until no more reassignment is necessary
Initial partition into K clusters
Initial partitions can be obtained by either
● User-specified initial partitions, - initial
belief per observation
● User-specified initial centroids, - initial
belief by average characteristics
● Random partitions - No initial belief
A
B
Cluster A
Item 1
Item 2
Item 3
Item 4
Item 5
Cluster B
K-Means Clustering:
Illustration
1.
Example: let us say k = 2. The algorithm picks 2 data-points at random; they will represent the initial
cluster centroids
2.
Next, it will assign the other data-points to the cluster centroid to which they are closest, based on a
specified distance measure (usually, squared Euclidian is used)
3.
Then, it will update (re-calculate) the clusters centroids based on the new clusters
K-Means Clustering:
Illustration
4.
Repeat step 2: re-assign points to the cluster’s centroid to which they are closest
5.
Recalculate the centroids for the updated clusters
6.
Keep repeating step 2 until convergence: that is, until the points re-assignment
does not change anymore
•
In particular: further rearranging the points does not improve the within-clusters
variance
•
The algorithm stops when the within-clusters variance is minimized
K-Means Clustering:
Illustration
Simple demo: http://syskall.com/kmeans.js/
Why are
multiple start
points (initial
partitions)
necessary?
Convergence/robustness of
K-means
● The procedure might oscillate indefinitely
● Convergence criterion:
○ Stop when a cluster centroid moves less
than x % of smallest distance between any
of the centroids
○ Specify the maximum number of iterations
(Solver)
Selecting K - Revision
● Re-run algorithm for different values of K
● Tradeoff: simplicity (interpretation) vs. adequacy
(within-cluster homogeneity)
● Plot cluster variability (total within-cluster sum of
squares) vs. K
● Choice is subjective!
Elbow Curve/Scree plot - Inertia
A subjective measure of when
there is the biggest dip in the Avg
Within Sum of Squared Distances
in cluster - the largest gain in
cluster cohesion.
Silhouette vs Inertia
Inertia
●
●
Look for the sharpest
drop in WSS with K
Subjective and
increasingly less popular
Silhouette
Score
●
●
Larger values, better
clusters
You do still look for the
sharpest drop to decide
when the marginal gain of
clustering is minimal
Evaluating usefulness of clustering
● What characterizes each cluster?
○
Look at the centroid means.
● Can you give a “name” to each cluster?
● Does this give us any insight?
● How do you want the clusters to evolve as you gather more data?
Mall of America
Example
QUESTION
APPROACH
RESULT
What can we learn from
a newly available data
source?
Analysis of Wi-Fi usage
mapped to physical
space.
Location- and
behavior-based visitor
segments.
Image (cc) flickr/ Will
Various Attributes were used for the
Visit Type Segmentation
● ATTRIBUTES
●
●
●
●
●
●
●
Visit Duration
Number of Sections Visited
Number of Levels Visited
Count of Access Points
Average Time Per Access Point
Time Spent in each section
Time Spent in Areas of Interest (6): Nordstrom, Macy’s, Sears,
Restaurants, South Food Court, North Food Court
Four Visit Types Were Identified From
our Clustering Algorithm
● EXPLORER
● EMPLOYEE
● LEISURE
● SMALL
SHOPPER
Case Study:
Customer
Segmentation for
Regional Airline
Case Study:
Customer Segmentation for
Regional Airline
• Goal: break down a large data set into small similar groups based on customer
attributes.
• Customer attributes considered
in this situation included:
• Travel frequency
• Average days booked in
advance
• Number of flights per trip
• Percentage of round trip
• Percentage of group trip
• Booking channels
Case Study (cont.):
Customer Segments Identified
Case Study (cont.): Segmentation
Summary