Ninness, C., Lauter, J. Coffee, M., Clary, L., Kelly, E., Rumph, M., Rumph, R., Kyle, R., & Ninness, S. (2012) Behavioral and Biological N
EPS 651 Multivariate Analysis
Factor Analysis,
Principal Components Analysis,
and
Neural Network Analysis
(Self-Organizing Maps)
For next week:
Continue with T&F Chapter 13
and please read the study below posted on our webpage:
T&F Chapter 13 --> 13.5.3 page 642
Several slides are based on material from the
UCLA SPSS Academic Technology Services
http://www.ats.ucla.edu/stat/spss/output/factor1.htm
Principal components analysis (PCA) and Factor Analysis are methods
of data reduction:
Suppose that you have a dozen variables that are correlated. You might use principal
components analysis to reduce your 12 measures to a few principal components. For
example, you may be most interested in obtaining the component scores (which are
variables that are added to your data set) and/or to look at the dimensionality of the
data. For example, if two components are extracted and those two components
accounted for 68% of the total variance, then we would say that two dimensions in the
component space account for 68% of the variance. Unlike factor analysis, principal
components analysis is not usually used to identify underlying latent variables.
[direct quote from below].
http://www.ats.ucla.edu/stat/spss/output/factor1.htm
FA and PCA: Data reduction methods
If raw data are used, the procedure will create the original correlation matrix or
covariance matrix, as specified by the user. If the correlation matrix is used, the
variables are standardized and the total variance will equal the number of variables
used in the analysis (because each standardized variable has a variance equal to
1). If the “covariance matrix” is used, the variables will remain in their original
metric. However, one must take care to use variables whose variances and scales
are similar. Unlike factor analysis, which analyzes the common variance, the
original matrix in a principal components analysis analyzes the total
variance. Also, principal components analysis assumes that each original measure
is collected without measurement error [direct quote].
http://www.ats.ucla.edu/stat/spss/output/factor1.htm
Spin Control
Factor analysis is a method of data reduction also – forgiving relative to PCA
Factor Analysis seeks to find underlying unobservable (latent) variables that are reflected in
the observed variables (manifest variables). There are many different methods that can be
used to conduct a factor analysis (such as principal axis factor, maximum likelihood,
generalized least squares, unweighted least squares). There are also many different types of
rotations that can be done after the initial extraction of factors, including orthogonal
rotations, such as varimax and equimax, which impose the restriction that the factors cannot
be correlated, and oblique rotations, such as promax, which allow the factors to be correlated
with one another. You also need to determine the number of factors that you want to
extract. Given the number of factor analytic techniques and options, it is not surprising that
different analysts could reach very different results analyzing the same data set. However, all
analysts are looking for a simple structure. A simple structure is pattern of results such that
each variable loads highly onto one and only one factor.
[direct quote]
http://www.ats.ucla.edu/stat/spss/output/factor1.htm
FA vs. PCA conceptually
FA produces factors
PCA produces components
FA
I1
I2
PCA
I3
I1
I2
I3
6
Kinds of Research Questions re PCA and FA
What does each factor mean? Interpretation? Your call
What is the percentage of variance in the data accounted for by the
factors? SPSS & psyNet will show you
Which factors account for the most variance? SPSS & psyNet
How well does the factor structure fit a given theory? Your call
What would each subject’s score be if they could be measured directly on the
factors? Excellent question!
7
Before you can even start to answer these questions using FA
should be
> .6
should be
< .05
Kaiser-Meyer-Olkin Measure of Sampling Adequacy - This measure varies between 0
and 1, and values closer to 1 are better. A value of .6 is a suggested minimum.
It answers the question: Is there enough data relative to the number of variables.
Bartlett's Test of Sphericity - This tests the null hypothesis that the correlation matrix is an identity
matrix. An identity matrix is a matrix in which all of the diagonal elements
are 1 and all off diagonal elements are 0.
Ostensibly, you want to reject this null hypothesis.
This, of course, is psychobabble.
Taken together, these two tests provide a minimum standard which should be passed before a factor
analysis (or a principal components analysis) should be conducted.
What is a Common Factor?
It is an abstraction, a “hypothetical construct” that relates to at least two
of our measurement variables into a factor
In FA, psychometricians / statisticians try to estimate the common factors
that contribute to the variance in a set of variables.
Is this an act of logical conclusion, a creation, or a figment of a
psychometrician’s imagination ? Depends on who you ask
What is a Unique Factor?
It is a factor that contributes to the variance in only one variable.
There is one unique factor for each variable.
The unique factors are unrelated to one another and unrelated to the
common factors.
We want to exclude these unique factors from our solution.
Seems reasonable … right?
Assumptions
Factor analysis needs large samples and it is one of
the only draw backs
• The more reliable the correlations are the
smaller the number of subjects needed
• Need enough subjects for stable estimates -How many is enough
11
Assumptions
Take home hint:
• 50 very poor, 100 poor, 200 fair, 300 good, 500 very good and
1000+ excellent
• Shoot for minimum of 300 usually
12
• More highly correlated markers fewer subjects
Assumptions
No outliers – obvious influence on correlations would bias
results
Multicollinearity
In PCA it is not problem; no inversions
In FA, if det(R) or any eigenvalue approaches 0 -> multicollinearity
is likely
The above Assumptions at Work:
Note that the metric for all these variables is the
same (since they employed a rating scale). So do we
do we run the FA as
correlation or covariance matrices /
does it matter?
Sample Data Set From Chapter 13 (p. 617)
Tabacknick and Fidell
Principal Components and Factor Analysis
Skiers
S1
S2
S3
S4
S5
Cost
32
61
59
36
62
Variables
Lift
Depth
64
65
37
62
40
45
62
34
46
43
Powder
67
65
43
35
40
Keep in mind, multivariate normality is assumed when
statistical inference is used to determine the number of factors.
The above dataset is far too small to fulfill the normality assumption. However,
even large datasets frequently violate this assumption and compromise the analysis.
Multivariate normality also implies that relationships among pairs of variables are linear. The analysis is
degraded when linearity fails, because correlation measures linear relationship and does not reflect
nonlinear relationship.
Linearity among variables is assessed through visual inspection of scatterplots.
Equations – Extractions - Components
Correlation matrix w/ 1s in the diag
Cost
Lift
Depth
Powder
Cost
Lift
Depth
Powder
1
-0.952990 -0.055276 -0.129999
-0.952990
1
-0.091107 -0.036248
-0.055276 -0.091107
1
0.990174
-0.129999 -0.036248 0.990174
1
Large correlation between Cost and Lift and another between Depth and Powder
Looks like two possible factors – why?
Are you sure about this?
L=V’RV => L =
V’
R
V
EigenValueMatrix = TransposeEigenVectorMatrix * CorMat * EigenVecMat
We are reducing to a few factors which duplicate the matrix?
Does this seem reasonable?
Equations – Extraction - Obtaining components
In a two-by-two matrix we derive eigenvalues
with two eigenvectors each containing two elements
In a four-by-four matrix we derive eigenvalues
with eigenvectors each containing four elements
L=V’RV It is important to know how L is constructed
Where
L is the eigenvalue matrix and V is the eigenvector matrix.
This diagonalized the R matrix and reorganized the variance into eigenvalues
A 4 x 4 matrix can be summarized by 4 numbers instead of 16.
5
4
Remember
this?
54
2
2-
1
1
2
-
(5- ) *
-5 + -2
+
2
0
0
(2-
(5 * 2)
)
-
-
(4 * 1)
(1 *4)
-7 + 6=0
With a two-by-two matrix we derive eigenvalues
with two eigenvectors each containing two elements
With a four-by-four matrix we derive eigenvalues
with eigenvectors each containing four elements
it simply becomes a longer polynomial
54
2
2-
1
(5- ) *
-5 + -2
+
2
(2-
-
(5 * 2)
)
(1 *4)
-7 + 6=0
-
(4 * 1)
= 0
Determinant
Where a = 1, b = -7 and c = 6
i
i
=
- (- 7 ) +
=
i
-b
1
b2 - 4 ac
2a
(-7)2 - 4 (1) * (6)
=6
2 (1)
- (- 7 ) -
=
-+
(-7)2 - 4 (1) * (6)
2 (1)
=6
2
an equation of the second
degree with two
roots [eigenvalues]
=1
=1
21
From Eigenvalues to Eigenvectors
R=VLV’
Equations – Extractions – Obtaining components
• SPSS matrix output
Skiers
S1
S2
S3
S4
S5
Cost
32
61
59
36
62
Variables
Lift
Depth Powder
64
65
67
37
62
65
40
45
43
62
34
35
46
43
40
Careful here. 1.91 is correct,
but it appears as a “2” in the text
Obtaining
L = the eigenvalue matrix
V’
Our original correlation matrix
V
Bartlett's Test of Sphericity - This tests the null hypothesis that the correlation
matrix is an identity matrix. An identity matrix is matrix in which all of the
diagonal elements are 1 and all off diagonal elements are 0.
Equations – Extraction – Obtaining Components
Other than the magic “2” below – this is a decent example
1.91
We have
“extracted” two
Factors from
four variables
Using a small
data set
Following SPSS Extraction and Rotation
and all that jazz… in this case, not much difference [others data sets show big change]
Cost
Lift
Depth
Powder
Factor 1 Factor 2
-0.401
0.907
0.251
-0.954
0.933
0.351
0.957
0.288
Here we see that Factor 1 is mostly Depth and Powder (Snow Condition Factor)
Factor 2 is mostly Cost and Lift, which is a Resort Factor
Both factors have complex loadings
Using SPSS 12, SPSS 20 and psyNet.SOM
This is a variation on your homework.
Just use your own numbers and replicate the process.
(we may use this hypothetical data as part of a study)
Skiers
S1
S2
S3
S4
S5
Cost
32
61
59
36
62
Variables
Lift
Depth
64
65
37
62
40
45
62
34
46
43
Powder
67
65
43
35
40
Here is an easier way than doing it by hand:
Arrange data in Excel Format as below: SPSS 20
Select Data Reduction: SPSS 12
Select Data Reduction: SPSS 20
Select Variables Descriptives: SPSS 12
Select Variables and Descriptives: SPSS 20
Start with a basic run using Principal Components: SPSS 12
Eigenvalues over 1
Start with a basic run using Principal Components: SPSS 12
Fixed number of factors
Select Varimax: SPSS 12
Select Varimax: SPSS 20
Under Options, select exclude cases likewise and sort by size: SPSS 12
Under Options, select exclude cases likewise and sort by size: SPSS 20
Under Scores, select “save variables” and “display matrix”: SPSS 20
Watch what pops out of your oven
A real time saver
Matching psyNet PCA correlation matrix with SPSS FA
This part is the same but the rest of PCA
goes in an entirely different direction
Kaiser's measure of sampling adequacy: Values of .6 and above are required for a good FA.
Remember
these guys?
An MSA of .9 is marvelous, .4 is not too impressive – Hey it was a small sample
Normally, variables with small MSAs should be deleted
Looks like two factors can be isolated/extracted
which ones? and what shall we call them?
Here they are again // they have eigenvalues > 1
We are reducing to a few factors which duplicate the matrix?
Fairly
Close
Rotations – Nice hints here
SPSS will provide an Orthogonal Rotation
without your help – look at the iterations
Extraction, Rotation, and Meaning of Factors
Orthogonal Rotation [assume no correlation among the factors]
Loading Matrix – correlation between each variable and the factor
Oblique Rotation [assumes possible correlations among the factors]
Factor Correlation Matrix – correlation between the factors
Structure Matrix – correlation between factors and variables
Oblique Rotations – Fun but not today
Factor extraction is usually followed by rotation in order to maximize
large correlation and minimize small correlations
Rotation usually increases simple structure and interpretability.
The most commonly used is the Varimax variance maximizing procedure
which maximizes factor loading variance
Rotating your axis “orthogonally” ~ sounds painfully chiropractic
Where are your
components
located on
these graphs?
What are the
upper
and
lower limits
on each of
these
axes?
Cost and Lift
may be a
factor,
but they are
polar opposites
Abbreviated Equations
Factor weight matrix [B] is found by dividing the loading matrix [A]
by the correlation matrix [R-1].
See matrix output
B  R
1
A
Factors scores [F] are found by multiplying the standardized scores
[Z] for each individual by the factor weight matrix [B]and adding
them up.
F  ZB
Abbreviated Equations
The specific goals of PCA or FA are to summarize patterns of correlations among
observed variables, to reduce a large number of observed variables to a smaller number
of factors,
to provide an operational definition
(a regression equation) for an underlying process by using observed
variables to test a theory about the nature of underlying processes.
Z  FA '
You can also estimate what each subject
would score on the “standardized variables.”
This is a revealing procedure—often overlooked.
Standardized
variables
as factors
Predictions based on Factor analysis: Standard-Scores
1.1447
Cost
Lift
Depth
Powder
0.96637
1
32
64
65
67
2
61
37
62
65
-0.41852
3
59
40
45
43
-1.11855
4
36
62
34
35
-0.574
5
62
46
43
40
Predictions based on Factor analysis: Standard-Scores
1.18534
Cost
Lift
Depth
Powder
-0.90355
1
32
64
65
67
- 0.70694
2
61
37
62
65
0.98342
3
59
40
45
43
4
36
62
34
35
Interesting stuff… what about cost?
-0.55827
5
62
46
43
40
Predictions based on Factor analysis: Standard-Scores
0.39393
-0.59481
- 0.73794
-0.64991
And this is supposed to represent ?
1.58873
SOM Classification of Ski Data
Variables
Skiers
S1
S2
S3
S4
S5
Skiers
Cost
Lift
Depth
Powder
Cost
32
61
59
36
62
S1
32
64
65
67
Variables
Lift
Depth
64
65
37
62
40
45
62
34
46
43
S2
61
37
62
65
S3
59
40
45
43
Powder
67
65
43
35
40
S4
36
62
34
35
S5
62
46
43
40
Transpose data before saving as a CSV file.
Transpose data to analyze by class/factors
4 rows 4 columns
In CSV format
SOM Classification of Ski Data
SOM classification 1:
Depth and Powder
across 5 SS
Nice match with FA 1
Cost
Lift
Powder
Depth
Cost
Lift
Depth
Powder
-1.36772 0.835832 0.683862 -1.06379 0.911816
1.27029 -1.14505 -0.87668 1.091376 -0.33994
1.285737 1.031973 -0.40602 -1.33649
-0.5752
1.275638 1.125563 -0.52526 -1.12556 -0.75038
1
32
64
65
67
2
61
37
62
65
3
59
40
45
43
4
36
62
34
35
5
62
46
43
40
Cost
Class/Factor ??
SOM classification 2:
Cost across 5 SS
SOM classification 3:
Lift across 5 SS
Lift
Class/Factor
Near match
with FA 2
C
ost
Lift
Depth
Powder
Cost
Lift
Powder
Depth
1
32
64
65
67
-1.36772
1.27029
1.285737
1.275638
2
61
37
62
65
0.835832
-1.14505
1.031973
1.125563
3
59
40
45
43
0.683862
-0.87668
-0.40602
-0.52526
4
36
62
34
35
-1.06379
1.091376
-1.33649
-1.12556
5
62
46
43
40
0.911816
-0.33994
-0.5752
-0.75038
Factor 1:
Appears to address
Depth and Powder
SOM classification 1:
Depth and Powder
across 5 SS
Nice match with FA 1
This could be placed into a logistic regression and predict with reasonable accuracy
Factor 2:
Appears to address Lift
SOM classification 3:
Lift across 5 SS
Predictions based on Factor analysis: Standard-Scores
Factor Analysis Factor 3: ??
SOM classification 2:
Cost across 5 SS
Center for Machine Learning and Intelligent Systems
Iris Setosa
Iris Versicolour
Iris Virginica
This dataset has provided the foundation for
multivariate statistics and machine learning
Transpose data before saving as a CSV file.
Transpose data to analyze by class/factors
4 rows 150 columns
In CSV format
Factor Analysis: Factor 1
Factor Analysis: Factor 2
petal width in cm
sepal length in cm
sepal width in cm
petal length in cm
SOM Neural Network: Class 1
SOM Neural Network: Class 2
sepal length in cm
sepal width in cm
petal length in cm
petal width in cm
Factor Analysis: Factor 1
SOM Neural Network: Class 1
sepal length in cm
sepal width in cm
petal length in cm
This could be placed into a logistic regression and predict with near perfect accuracy
Really ?? Look at the original
Everybody but psychologists seem to understand this