Principle Component Analysis
and its use in MA clustering
Lecture 12
What is PCA?
• This is a MATHEMATICAL procedure that transforms a set of
correlated responses into a smaller set of uncorrelated variables called
PRINCIPAL COMPONENTS.
•
•
•
•
•
Uses:
Data screening
Clustering
Discriminant Analysis
Regression combating Multicollinearity
Objectives of PCA
• It is an exploratory technique meant to give researchers a
better FEEL for their data
• Reduce dimensionality, rather try to understand the TRUE
dimensionality of the data
• Identify “meaningful” variables
• If you have a VARIANCE-COVARIANCE MATRIX, S:
• PCA returns new variables called Principal Components
that are:
– Uncorrelated
– First component explains MOST of the variability
– The remaining PC explain decreasing amounts of variability
Idea of PCA
• Consider x to be a random variable with mean m and Variance given
by S.
• The first PC variable is defined by: y1= a1’(x-m), such that a1 is chosen
so that the VAR(a1’(x-m)) is maximized for all vectors a1 satisfying
a1’a1=1
• It can be shown that the maximum value of the variance of a1’(x-m)
among all vectors satisfying the condition is l1 (the first or largest
eigen value of the Matrix, S). This implies a1is the eigen vector
corresponding to the eigen value l1.
• The second PC is the eigen vector corresponding to the second largest
eigen value l2 and so on to the pth eigen value.
Supplementary Info:
• What are Eigen Values and Eigen Vectors?
• Also called characteristic root (latent root) eigen values are
the roots of the polynomial equation defined by:
• | S - l I| =0
• This leads to an equation of form:
• c1lp + c2lp-1 + … cpl + cp+1 = 0
• If S is symmetric then the eigen values are real numbers
and can be ordered.
Supplementary Info: II
• What are Eigen Vectors?
• Similarly, eigen vectors are the vectors satisfying the equation:
Sa - l a =0
• If S is symmetric then there will be p eigen vectors corresponding to
the p eigen values.
• Generally not unique and are normalized to aj’aj = 1
• Remarks: if two eigen values are NOT equal there eigen vectors will
be orthogonal to each other. When two eigen values are equal their
eigen vectors are CHOSEN orthogonal to each other (in this case these
are non-unique).
• Tr(S) = S li
• |S| = li
Idea of PCA contd…
• Hence the p principal components are a1, a2….ap, the eigen
vectors corresponding to the ordered eigen values of S.
• Here, l1  l2  …  lp.
• Result: two principal components are uncorrelated if and
only if their defining eigen vectors are orthogonal to each
other. Hence the PC are placed on a orthogonal axis
system where are the data fall.
Idea of PCA contd…
• The varaince of the jth component is lj, j=1,…,p.
• Remember: tr(S) = s11+s22+…+spp.
• Also, tr(S)=l1+l2+…+lp.
• Hence, often a measure of “importance” of the jth principal
component is given by lj/tr(S).
Comments
• To actually do PCA we need to compute the principal
component scores or the values of the principal component
variable for each unit in the data set.
• These scores provide locations of the observations in a
data set with respect to the principal component axis.
• Generally eigen vectors are normalized to length 1, aj’aj=1.
• Often to make comparison between eigen values each
element in the vector is multiplied by the square root of the
corresponding eigen value( called component vectors),
cj = (lj1/2 )aj.
Estimating PC
• Life would be easy if m and S were known. All we had to
do was to estimate the normalized eigen vectors and
corresponding eigen values.
• But, most of the time we DO NOT know m and S and we
need to estimate those and hence the PCA are the sample
values corresponding to the estimated m and S.
• Determining the # of PC:
– Look for the eigen values that are much smaller than the others.
Plots like SCREE plot (plot of eigen value versus the eigen
number)
Caveats
• The whole idea of PCA is to transform a set of correlated
variables to a set of uncorrelated variables, hence if the
data are already uncorrelated, not much additional
advantage of doing PCA.
• One can do PCA on correlation matrix or the Covariance
matrix.
• In the latter case, the component correlation vectors
cj = (lj1/2 )aj give the correlations between the original
variables and the jth principal component variable.
PCA and Multidimensional Scaling
• Essentially what PCA does is what is called SINGULAR VALUE
DECOMPOSITION(SVD) of a matrix
•
•
•
•
X=UDV’
Where X is n by p, with n<<p (in MA)
U is n by n
D is n by n, diagonal matrix with the diagonals decreasing, d1 d2… 
dn.
• V is a p by n matrix, which rotates X into a new set of co-ordinates.
such that XV=UD
SVD and MDS
• SVD is a VERY memory hungry procedure and especially
for MA data when there a large number of genes it is very
slow and often needs HUGE amounts of memory to work.
• Multidimensional Scaling (MDS): is a collection of
methods that do not use the full data matrix but rather the
distance matrix between the variables. This reduces the
computation from n by p to n by n (quite a reduction!).
Sammon Mapping
• A common method used in MA is SAMMON mapping
which aims to find the two-dimensional representation that
has the maximum dissimilarity matrix compared to the
original one.
• PCA has the advantage in the sense that it represents the
samples in a scatterplot whose axes are made up of a linear
combination of the most variable genes.
• Sammon mapping treats all genes equivalently and hence
is a bit “duller” than PCA based clustering.
PCA in Microarrays
• Useful technique to understand the TRUE dimensionality of the
data.
• Useful for clustering.
• In R under the MASS package you can use:
•
•
•
•
•
•
my.data1=read.table("cluster.csv",header=TRUE,sep=",")
princomp(my.data1)
myd.sam <- sammon(dist(my.data1))
plot(myd.sam$points, type = "n")
text(myd.sam$points, labels =as.character(1:nrow(my.data1)))