Principal Component Analysis (PCA)
Lecture Series: Mathematics for AI/ML
1. Story: Why PCA Emerged After LDA?
In Linear Discriminant Analysis (LDA), we searched for directions that separate classes
as much as possible. However, what if labels are not available? We may have a large
dataset with thousands of features, many being redundant, noisy, or highly correlated.
The goal becomes understanding the geometric structure of the data itself.
Imagine standing at the top of a hill and observing a city. From some angles, the
view looks cluttered, but from another angle the entire structure becomes clear. PCA is
the process of finding that viewing angle—the perspective from which the data becomes
maximally informative.
Thus, PCA helps us find the direction that preserves the most important structure of
data without needing class labels.
2. Intuition: PCA Searches for Directions of Maximum Spread
When we project data onto some direction p, we obtain scalars
zi = pT xi .
If the projection direction is poorly chosen, the points clump together and the variance is small. If chosen well, the spread is large and the projection captures the major
structural information.
Core idea: PCA chooses the direction along which the data varies the most.
From LDA viewpoint:
• LDA finds directions that maximize between-class separation.
• PCA finds directions that maximize overall variance.
No labels, no classes — pure geometry.
1
3. Theory: Mathematical Setup
We are given data points
X = {xi ∈ Rd : i = 1, . . . , n}.
We project each vector onto a unit direction p:
zi = pT xi ,
pT p = 1.
Our objective is to maximize the variance of the projected values:
n
1X
v=
(zi − γ)2 ,
n i=1
where
γ=
n
1X
zi = pT µ,
n i=1
and
µ=
n
1X
xi .
n i=1
4. Derivation: Variance of the Projected Scalars
Step 1: Zero-mean transformation
Define centered data:
yi = xi − µ.
Then
zi − γ = pT yi .
Step 2: Express variance in matrix form
n
1X
(pT yi )2 = pT
v=
n i=1
n
1X
yi y T p.
n i=1 i
Define covariance matrix:
C=
n
1X
(xi − µ)(xi − µ)T .
n i=1
Thus the variance becomes:
v = pT Cp .
2
!
5. Optimization Using KKT
We aim to solve
max
pT Cp
p
s.t.
pT p = 1.
Construct the Lagrangian:
g(p, α) = −pT Cp + α(pT p − 1).
KKT condition:
∇p g = −2Cp + 2αp = 0.
This gives the eigenvector equation:
Cp = αp.
Thus,
• The direction of maximum variance is an eigenvector of C.
• The maximum variance along that direction equals its eigenvalue.
The eigenvector corresponding to the largest eigenvalue is the first principal component.
6. Second Component: Orthogonality Constraint
After determining the first principal component e1 , the next direction q must satisfy:
q T q = 1,
q T e1 = 0,
and must maximize:
q T Cq.
Applying KKT conditions yields:
Cq = αq,
q ⊥ e1 .
Thus the second principal component is the eigenvector corresponding to the secondlargest eigenvalue. This extends similarly to all components.
3
7. PCA Transformation Matrix
Let the sorted eigenvectors be
E = [e1 , e2 , . . . , ed ].
The top-k eigenvectors form:
Q = [e1 , . . . , ek ].
PCA transformation:
v = QT (x − µ) .
This maps a d-dimensional sample to a k-dimensional reduced representation.
8. Practical Example (Intuitive)
Consider a 2D dataset shaped like an elongated ellipse. PCA identifies:
• PC1: the direction of maximum variance (long axis),
• PC2: the perpendicular direction with smaller variance.
Projecting onto PC1 alone retains the major structure of the data.
9. ML Connection: Face Recognition (Eigenfaces)
PCA is widely used in face recognition. Steps:
1. Vectorize each face image into a long vector.
2. Compute the mean face µ.
3. Compute covariance matrix C.
4. Compute eigenvectors (eigenfaces).
5. Select top-k eigenfaces.
6. Represent each new face as
v = QT (x − µ).
This approach drastically reduces dimensionality and isolates key distinguishing facial
features.
4
10. Bridge to Next Topic
PCA produces uncorrelated, variance-maximizing directions. Natural next topics include
Singular Value Decomposition (SVD), Whitening, or Kernel PCA depending on the course
structure.
5