Factor Analysis and Principal Components Factor analysis with principal components presented as a subset of factor analysis techniques, which it is subset. Principal Components: (PC) Principal components is about explaining the variance-covariance structure, , of a set of variables through a few linear combinations of these variables. In general PC is used for either: 1. Data reduction or 2. Interpretation If you have p variables x x1 , , x p you need p components to capture all the variability, but often a smaller number, k, principal components can capture most of the variability. So the original data of n measurements on p variables can be reduced to a to a data set of n measurements on k principal components. PC tends to be a means to an end but not the end itself. That is PC is often not the final step. The PC may be used for multiple regression, cluster analysis, etc. Let x x1 , , x p with covariance matrix consider the linear combination Y1 a1x a11x1 a12 x 2 a1p x p p Y2 a 2 x a 2i x i i 1 p Yp a p x a pi x i i 1 Var Yi a i a i , i 1, 2, ,p Cov Yi , Yk a i a k , i, k 1, 2, ,p 1. First principal component linear combination a1x that maximizes Var a1x subject to a1a1 1 2. second principal component linear combination a 2 x that maximizes Var a 2 x subject to a 2a 2 1 and Cov a1x,a 2 x 0 i th . i th principal component linear combination a i x that maximizes Var a i x subject to a i a i 1 and Cov a i x,a k x 0 for k i. Find the principal components and the proportion of the total population of the total population variance explained by each when the covariance matrix is 2 2 0 1 1 2 2 2 , 2 2 0 2 2 To solve this you will have to go though your notes, but you can do this even though didn’t give you the formula. Hint : Re call : Maximization of Quadratic forms for points on the Unit Sphere. Let B be a positive definite matrix with eigenvalues pp 1 2 3 e1 ,e 2 , p and associated normalized eigenvectors ,e p . Then x Bx max 1 attained when x e1 x 0 x x x Bx min p attained when x e p x 0 x x Moreover x Bx max k 1 attained when x e k 1 k 1, 2, x e1 , ,e k x x , p 1 Answer : I 2 0 3 2 2 2 2 2 242 0 2 2 or 2 1 2 for 1 2 e1 1 1 2 2 0 1 2 2 2 1 2 e2 1 2 1 2 1 2 3 e3 1 2 1 2 1 2 2 2 Principal Components 1 Y1 X1 2 1 Y2 X1 2 1 Y3 X1 2 Var 2 1 X3 2 1 1 X 2 X3 2 1 2 2 2 1 1 X2 X3 2 2 2 1 2 % Total var 1 3 1 1 2 3 1 1 2 3 Let be the covariance matrix associated with the random vector x x1 , x 2 , , x p . Let have the eigenvalue eigenvector pairs 1 ,e1 , 2 ,e2 , , p ,e p where 1 2 Then the i th principal component is given by Yi ei x ei1x1 ei 2 x 2 eip x p i 1, 2, , p Var Yi ei ei i i 1, 2, ,p Cov Yi , Yk ei e k 0 i k 1. 11 22 p pp Var x i 1 2 i 1 p p Var Yi i 1 p 0. Proof : By definition tr 11 22 pp we can write as PP where is the diagonal matrix of eigenvalues and P e1 ,e 2 , ,e p PP PP I p tr tr PP tr PP tr i i 1 p p i 1 i 1 Var x i tr tr Yi p p i 1 i 1 Thus total population variance ii i Thus proportion of total pop.var. due to k th principal component is p k k 1, 2, , p. i i 1 Y1 e1x , Y2 e2 x , , Yp ep x are the principal components from the covariance matrix , then Yi ,x k eik i kk i, k 1, 2, are the correlation coefficients between Yi and the variables x k . Here 1 ,e1 , 2 ,e 2 , ,p , p ,e p are the eigenvalue eigenvector pair for . Show Yi ,x k eik i kk Proof : set a k 0,0, ,1,0, ,0 Cov X k , Yk Cov a k x,ei x a k ei x k a k x ei i ei So Cov X k , Yi a k i ei i eik Var Yi i Show earlier Var X k kk Yi ,Xk Cov Yi , X k Var Yi Var X k i eik i kk eik i kk i, k 1, 2, ,p Pricipal Components from Standard Variables Z1 X1 1 11 Z2 X 2 2 22 Zp X p p pp 1 In matrix notation Z V 11 0 1 22 0 2 where V 0 0 1 2 X 0 0 pp 1 1 E Z 0 Cov Z V V The PC of Z can be obtained from the eigenvectors of the correlation matrix of X. 1 2 1 2 For notation we shall continue to use Yi to refer to the i th PC and i ,ei as the eigenvalue eigenvector pair from either or . However, the i ,ei derived from are, in general, not the same as the ones derived from . The i th PC of the standardized variables Z Z1, Z2 , with Cov Z is given by Yi ei Z ei V 1 2 p p i 1 i 1 , Z p 1 X i 1, ,p Var Yi Var Zi The number of random variables, not rho Yi ,Zk eik i i,k 1,2, ,p where i ,ei are the eigenvalue eigenvector pair for , with 1 2 p 0. If S Sik is the p p sample covariance matrix with , ˆ ,eˆ eigenvalue eigenvector pairs ˆ 1 ,eˆ1 , ˆ 2 ,eˆ 2 , the i th sample principal component is given by yˆ i eˆ i x eˆ i1x1 eˆ i2 x 2 eˆ ip x p i 1, 2, , p. where ˆ 1 ˆ 2 ˆ p 0 and X is any observation on the var iables X1 , X 2 , , Xp. Also sample variance of yˆ k ˆ k k 1, 2, sample covariance of yˆ i , yˆ k 0 i k ryˆ i ,x k eˆ ik ˆ i Skk i, k 1, 2, ,p ,p p p Factor Analysis The main purpose of factor analysis is to try to describe the covariance relationships among many variables in term of a few underlying, but unobservable, random quantities called factors. The Orthogonal Factor Model: The observable random vector x, with p components, has mean and covariance matrix, . The factor model proposes that x is linearly dependent upon a few unobservable random variables, F1, F2 ,….,Fm , called common factors and p additional sources of variation 1, 2 ,….,p called error, or specific factors. X1 1 l11F1 l12 F2 l1m Fm 1 X 2 2 l21F1 l22 F2 l2m Fm 2 X p p lp1F1 lp2 F2 lpm Fm p in matrix notation : X L (p1) F (pm) (m1) (p1) lij is called the i th loading of the jth factor. L is called the matrix of factor loadings. The p deviations X1 1 , X 2 2 , , X p p are expressed in terms of F1 , p m random variables. , Fm , 1 , , p , that is Assumptions : E F 0 Cov F E FF I E 0 1 0 0 2 Cov E pp 0 0 (m1) (p1) (mm) 0 0 p F and are independent thus Cov , F E F 0 (pm) Solve for in terms of L and . Cov X E X X X X LF LF LF LF LF LF LF LF E X X E LF LF LF LF E LFFL FL LF LE FF L E F L LE F E indepenent 0 LIL LL independent 0 X F LF F LFF F Cov X, F E X F LE FF E F L. Also so So Var X i li12 2 lim i Cov X i , X k li1l k1 Cov X i , Fj lij lim l km i k The Principal Component and principal factor method One method for factor analysis Re call Spectral decomposition : Let a covariance matrix have eigenvalue eigenvector pair i ,ei with 1 2 1e1e1 2 e 2 e2 1 e1 | 2 e2 | p 0. Then p e p ep | 1 e1 2 e2 p e p p ep Thus for a factor analysis where m p #factors # var iables and i 0 for all i L L 0 LL (pp) (pp) (pp) (pp) (pp) Since we almost always want fewer factors than original var iables m p . One approach when the last p m eigenvalue are small is to neglect the contribution of m1e m1em1 p e p ep to . We use an approximation and now L L removing the last (pp) (pm) (pm) p m components. This assumes 1 error, can still be ignored. If we wish to allow for to be included LL 1 e1 2 e2 m where i ii lij2 j1 1 e1 1 0 2 e2 0 2 m em 0 0 m em for i 1, , p. 0 0 p When applying this approach it is typical to center the observations minus X . In the case where the units of the variables are not the same e.g. K.g. and cm. height weight it is usually desirable to work with the standardized variables, X j1 X1 X j2 X 2 Zj X jp X p S11 S22 Spp j 1, 2, ,n Maximum Likelihood Method for Factor Analysis If the common factors, F, and can be assumed to be normally distributed, then ML estimates of the factor loadings and variance may be obtained. When Fj and j are jointly normal, the observations X j LFj j are then normal. The likelihood is L , 2 np 2 n 2 1 1 n exp tr X j X X j X j1 2 n X X 1 1 n exp tr X j X X j X L , 2 j1 2 p 2 1 2 n exp X 1 X . 2 2 which depends on L and through LL . n 1 p 2 n 1 2 To make L will defined, a unique solution, impose the condition that L 1L a diagonal matrix Proportion of total sample th variance due to the j factor ˆl 2 ˆl 2 2j 1j ˆlpj2 S11 S22 Spp Factor Rotation All factor loadings obtained from the initial loadings by a orthogonal transformation have the same ability to reproduce the covariance (or correlation) matrix. ˆ where TT TT I Lˆ* LT ˆ ˆ ˆ Lˆ ˆ Lˆ*Lˆ* ˆ ˆ LTT so LL ˆ remains unchanged also. The Imagine if these were only m 2 factors Lˆ* Lˆ T p2 (p2) (22) cos sin clockwise where T rotation sin cos cos sin counter clockwise or T sin cos rotation is an angle which the factor loadings will be rotated through. A Reference: The following 13 slides comes from: Multivariate SPSS By Data Analysis Using John Zhang ARL, IUP Factor Analysis-1 The main goal of factor analysis is data reduction. A typical use of factor analysis is in survey research, where a researcher wishes to represent a number of questions with a smaller number of factors Two questions in factor analysis: How many factors are there and what they represent (interpretation) Two technical aids: Eigenvalues Percentage of variance accounted for Factor Analysis-2 Two types of factor analysis: Exploratory: introduce here Confirmatory: SPSS AMOS Theoretical basis: Correlations among variables are explained by underlying factors An example of mathematical 1 factor model for two variables: V1=L1*F1+E1 V2=L2*F1+E2 Factor Analysis-3 Each variable is composed of a common factor (F1) multiply by a loading coefficient (L1, L2 – the lambdas or factor loadings) plus a random component V1 and V2 correlate because the common factor and should relate to the factor loadings, thus, the factor loadings can be estimated by the correlations A set of correlations can derive different factor loadings (i.e. the solutions are not unique) One should pick the simplest solution Factor Analysis-4 That is the findings should not differ by methodology of analysis nor by sample A factor solution needs to confirm: By a different factor method By a different sample More on terminology Factor loading: interpreted as the Pearson correlation between the variable and the factor Communality: the proportion of variability for a given variable that is explained by the factor Extraction: the process by which the factors are determined from a large set of variables Factor Analysis-5 (Principle components) Principle component: one of the extraction methods A principle component is a linear combination of observed variables that is independent (orthogonal) of other components The first component accounts for the largest amount of variance in the input data; the second component accounts for the largest amount or the remaining variance… Components are orthogonal means they are uncorrelated Factor Analysis-6 (Principle components) Possible application of principle components: E.g. in a survey research, it is common to have many questions to address one issue (e.g. customer service). It is likely that these questions are highly correlated. It is problematic to use these variables in some statistical procedures (e.g. regression). One can use factor scores, computed from factor loadings on each orthogonal component Factor Analysis-7 (Principle components) Principle component vs. other extract methods: Principle component focus on accounting for the maximum among of variance (the diagonal of a correlation matrix) Other extract methods (e.g. principle axis factoring) focus more on accounting for the correlations between variables (off diagonal correlations) Principle component can be defined as a unique combination of variables but the other factor methods can not Principle component are use for data reduction but more difficult to interpret Factor Analysis-8 Number of factors: Eigenvalues are often used to determine how many factors to take Take as many factors there are eigenvalues greater than 1 Eigenvalue represents the amount of standardized variance in the variable accounted for by a factor The amount of standardized variance in a variable is 1 The sum of eigenvalues is the percentage of variance accounted for Factor Analysis-9 Rotation Objective: to facilitate interpretation Orthogonal rotation: done when data reduction is the objective and factors need to be orthogonal Varimax: attempts to simplify interpretation by maximize the variances of the variable loadings on each factor Quartimax: simplify solution by finding a rotation that produces high and low loadings across factors for each variable Oblique rotation: use when there are reason to allow factors to be correlated Oblimin and Promax (promax runs fast) Factor Analysis-10 Factor scores: if you are satisfy with a factor solution You can request that a new set of variables be created that represents the scores of each observation on the factor (difficult of interpret) You can use the lambda coefficient to judge which variables are highly related to the factor; the compute the sum of the mean of this variables for further analysis (easy to interpret) Factor Analysis-11 Sample size: the sample size should be about 10 to 15 times the number of variables (as other multivariate procedures) Number of methods: there are 8 factoring methods, including principle component Principle axis: account for correlations between the variables Unweighted least-squares: minimize the residual between the observed and the reproduced correlation matrix Factor Analysis-12 Generalize least-squares: similar to Unweighted leastsquares but give more weight to the variables with stronger correlation Maximum Likelihood: generate the solution that is the most likely to produce the correlation matrix Alpha Factoring: Consider variables as a sample; not using factor loadings Image factoring: decompose the variables into a common part and a unique part, then work with the common part Factor Analysis-13 Recommendations: Principle components and principle axis are the most common used methods When there are multicollinearity, use principle components Rotations are often done. Try to use Varimax Reference Factor Analysis from SPSS Much of the wording comes from the SPSS help and tutorial. Factor Analysis Factor Analysis is primarily used for data reduction or structure detection. The purpose of data reduction is to remove redundant (highly correlated) variables from the data file, perhaps replacing the entire data file with a smaller number of uncorrelated variables. The purpose of structure detection is to examine the underlying (or latent) relationships between the variables. Factor Analysis The Factor Analysis procedure has several extraction methods for constructing a solution. For Data Reduction. The principal components method of extraction begins by finding a linear combination of variables (a component) that accounts for as much variation in the original variables as possible. It then finds another component that accounts for as much of the remaining variation as possible and is uncorrelated with the previous component, continuing in this way until there are as many components as original variables. Usually, a few components will account for most of the variation, and these components can be used to replace the original variables. This method is most often used to reduce the number of variables in the data file. For Structure Detection. Other Factor Analysis extraction methods go one step further by adding the assumption that some of the variability in the data cannot be explained by the components (usually called factors in other extraction methods). As a result, the total variance explained by the solution is smaller; however, the addition of this structure to the factor model makes these methods ideal for examining relationships between the variables. With any extraction method, the two questions that a good solution should try to answer are "How many components (factors) are needed to represent the variables?" and "What do these components represent?" Factor Analysis: Data Reduction An industry analyst would like to predict automobile sales from a set of predictors. However, many of the predictors are correlated, and the analyst fears that this might adversely affect her results. This information is contained in the file car_sales.sav . Use Factor Analysis with principal components extraction to focus the analysis on a manageable subset of the predictors. Factor Analysis: Structure Detection A telecommunications provider wants to better understand service usage patterns in its customer database. If services can be clustered by usage, the company can offer more attractive packages to its customers. A random sample from the customer database is contained in telco.sav . Factor Analysis to determine the underlying structure in service usage. Use: Principal Axis Factoring Example of Factor Analysis: Structure Detection Telecommunications provider wants to better understand service usage patterns in its customer database. Selecting service offerings Example of Factor Analysis: Descriptives Click descriptives: Recommend checking Initial Solution (default) In addition, check “Anti-image” and “KMO and …”. Example of Factor Analysis: Extraction Click Extraction: Select Method “Principal axis factoring”. Recommend Keep defaults but also check “Scree plot”. Example of Factor Analysis: Rotation Click Rotation: Select “Varimax” and Loading plot(s)”. Understanding the Output The Kaiser-Meyer-Olkin Measure of Sampling Adequacy is a statistic that indicates the proportion of variance in your variables that might be caused by underlying factors. Perhaps can’t use factor analys if <0.5 KMO and Bartlett's Test Kaiser-Meyer-Olkin Measure of Sampling Adequacy. Bartlett' s Test of Sphericity Approx. Chi-Square df Sig . .888 6230.901 91 .000 Bartlett's test of sphericity tests the hypothesis that your correlation matrix is an identity matrix, which would indicate that your variables are unrelated and therefore unsuitable for structure detection. Sig. <0.05 than factor analysis may be helpful. Understanding the Output Communalities Long distance last month Toll free last month Equipment last month Calling card last month Wireless last month Multiple lines Voice mail Paging service Internet Caller ID Call waiting Call forwarding 3-way calling Electronic billing Initial .297 .510 .579 .266 .660 .276 .471 .527 .455 .552 .545 .532 .506 .416 Extraction .748 .564 .697 .307 .708 .340 .501 .541 .525 .623 .610 .596 .561 .488 Extraction Method: Principal Axis Factoring. Extraction communalities are estimates of the variance in each variable accounted for by the factors in the factor solution. Small values indicate variables that do not fit well with the factor solution, and should possibly be dropped from the analysis. The lower values of Multiple lines and Calling card show that they don't fit as well as the others. Understanding the Output Before rotation Only three factors in the initial solution have eigenvalues greater than 1. Together, they account for almost 65% of the variability in the original variables. This suggests that three latent influences are associated with service usage, but there remains room for a lot of unexplained variation. Understanding the Output After rotation From rotation approximately now 56% of the variation is explained about a 10% loss in explanation of the variation. In general, there are a lot of services that have correlations greater than 0.2 with multiple factors, which muddies the picture. The rotated factor matrix should clear this up. Understanding the Output Before rotation Factor Matrixa 1 Long distance last month Toll free last month Equipment last month Calling card last month Wireless last month Multiple lines Voice mail Paging service Internet Caller ID Call waiting Call forwarding 3-way calling Electronic billing .146 .652 .494 .364 .799 .257 .669 .692 .323 .689 .678 .684 .662 .250 Factor 2 -.254 -.373 .671 -.243 .261 .280 .228 .246 .648 -.345 -.366 -.336 -.338 .652 Extraction Method: Principal Axis Factoring . a. Attempted to extract 3 factors. More than 25 iterations req uired. (Converg ence=.002). Extraction was terminated. 3 .814 .020 .054 .339 .037 .442 -.038 -.050 -.014 -.172 -.126 -.128 -.093 -.035 The relationships in the unrotated factor matrix are somewhat clear. The third factor is associated with Long distance last month. The second corresponds most strongly to Equipment last month, Internet, and Electronic billing. The first factor is associated with Toll free last month, Wireless last month, Voice mail, Paging service, Caller ID, Call waiting, Call forwarding, and 3-way calling. Understanding the Output After rotation Rotated Factor Matrixa 1 Long distance last month Toll free last month Equipment last month Calling card last month Wireless last month Multiple lines Voice mail Paging service Internet Caller ID Call waiting Call forwarding 3-way calling Electronic billing .062 .726 .067 .348 .530 -.025 .455 .468 -.049 .787 .779 .768 .743 -.107 Factor 2 -.121 .018 .831 -.012 .637 .384 .539 .566 .722 .056 .033 .062 .050 .686 Extraction Method: Principal Axis Factoring . Rotation Method: Varimax with Kaiser Normalization. a. Rotation converged in 4 iterations. 3 .854 .191 .049 .431 .146 .438 .054 .044 -.045 .008 .054 .048 .078 -.080 The first rotated factor is most highly correlated with Toll free last month, Caller ID, Call waiting, Call forwarding, and 3way calling. These variables are not particularly correlated with the other two factors. The second factor is most highly correlated with Equipment last month, Internet, and Electronic billing. The third factor is largely unaffected by the rotation. Thus, there are three major groupings of services, as defined by the services that are most highly correlated with the three factors. Given these groupings, you can make the following observations about the remaining services: Understanding the Output Rotated Factor Matrixa 1 Long distance last month Toll free last month Equipment last month Calling card last month Wireless last month Multiple lines Voice mail Paging service Internet Caller ID Call waiting Call forwarding 3-way calling Electronic billing .062 .726 .067 .348 .530 -.025 .455 .468 -.049 .787 .779 .768 .743 -.107 Factor 2 -.121 .018 .831 -.012 .637 .384 .539 .566 .722 .056 .033 .062 .050 .686 Extraction Method: Principal Axis Factoring . Rotation Method: Varimax with Kaiser Normalization. a. Rotation converged in 4 iterations. 3 .854 .191 .049 .431 .146 .438 .054 .044 -.045 .008 .054 .048 .078 -.080 Because of their moderately large correlations with both the first and second factors, Wireless last month, Voice mail, and Paging service bridge the "Extras" and "Tech" groups. Calling card last month is moderately correlated with the first and third factors, thus it bridges the "Extras" and "Long Distance" groups. Multiple lines is moderately correlated with the second and third factors, thus it bridges the "Tech" and "Long Distance" groups. This suggests avenues for cross-selling. For example, customers who subscribe to extra services may be more predisposed to accepting special offers on wireless services than Internet services. Summary: What Was Learned Using a principal axis factors extraction, you have uncovered three latent factors that describe relationships between your variables. These factors suggest various patterns of service usage, which you can use to more efficiently increase crosssales. Using Principal Components Principal Components can aid in clustering. What is principal components? Principal is a statistical technique that creates new variables that are linear functions of the old variables. The main goal of principal components is to to reduce the number of variables needed to analyze. Principal Components Analysis (PCA) What it is and when it should be used. Introduction to PCA What does principal components analysis do? Takes a set of correlated variables and creates a smaller set of uncorrelated variables. These newly created variables are called principal components. There are two main objectives for using PCA 1. Reduce the dimensionality of the data. – – – 2. Identify new meaningful underlying variables. – – In simple English: turn p variables into less than p variables. While reducing the number of variables we attempt to keep as much information of the original variables as possible. Thus we try to reduce the number of variables without loss of information. This is often not possible. The “principal components created are linear combinations of the original variables and often don’t lend to any meaning beyond that. There are several reasons why and situations where PCA is useful. Introduction to PCA There are several reasons why PCA is useful. 1. 2. PCA is helpful in discovering if abnormalities exist in a multivariate dataset. Clustering (which will be covered later): – PCA is helpful when it is desirable to classify units into groups with similar attributes. – 3. It can also be helpful for verifying the clusters created when clustering. Discriminant analysis: – – 4. For example: In marketing you may want to classify your customers into groups (or clusters) with similar attributes for marketing purposes. In some cases there may be more response variables than independent variables. It is not possible to use discriminant analysis in this case. Principal components can help reduce the number of response variables to a number less than that of the independent variables. Regression: – It can help address the issue of multicolinearity in the independent variables. Introduction to PCA Formation of principal components 1. 2. 3. 4. 5. They are uncorrelated The 1st principal component accounts for as much of the variability in the data as possible. The 2nd principal component accounts for as much of the remaining variability as possible. The 3rd … Etc. Principal Components and Least Squares Think of the Least Squares model Y X E Y is a n p matrix of the centered observed variables. X is a n j matrix of the scores on the 1st j principal components. B is a j p matrix of the eigenvectors. E is a n p matrix of the residuals. • Eigenvector <mathematics> A vector which, when acted on by a particular linear transformations, produces a scalar multiple of the original vector. The scalar in question is called the eigenvalue corresponding to this eigenvector. •www.dictionary.com Calculation of the PCA There are two options: 1. 2. Correlation matrix. Covariance matrix. Using the covariance matrix will cause variables with large variances to be more strongly associated with components with large eigenvalues and the opposite is true of variables with small variances. For the above reason you should use the correlation matrix unless the variables are comparable or have been standardized. Limitations to Principal Components PCA converts a set of correlated variables into a smaller set of uncorrelated variables. If the variables are already uncorrelated than PCA has nothing to add. Often it is difficult to impossible to explain a principal component. That is often principal components do not lend themselves to any meaning. SAS Example of PCA We will analyze data on crime. CRIME RATES PER 100,000 POPULATION BY STATE. The variables are: 1. 2. 3. 4. 5. 6. 7. MURDER RAPE ROBBERY ASSAULT BURGLARY LARCENY AUTO SAS command for PCA SAS CODE: PROC PRINCOMP DATA=CRIME OUT=CRIMCOMP; run; The dataset is CRIME and results will be saved to CRIMCOMP SAS Output Of Crime Example Observations 50 7 Variables Simple Statistics MURDER RAPE ROBBERY ASSAULT BURGLARY Mean 7.444000000 25.73400000 124.0920000 211.3000000 StD 3.866768941 10.75962995 88.3485672 100.2530492 LARCENY AUTO 1291.904000 2671.288000 377.5260000 432.455711 725.908707 193.3944175 Correlation Matrix MURDER RAPE ROBBERY ASSAULT BURGLARY LARCENY AUTO MURDER 1.0000 0.6012 0.4837 0.6486 0.3858 0.1019 0.0688 RAPE 0.6012 1.0000 0.5919 0.7403 0.7121 0.6140 0.3489 ROBBERY 0.4837 0.5919 1.0000 0.5571 0.6372 0.4467 0.5907 ASSAULT 0.6486 0.7403 0.5571 1.0000 0.6229 0.4044 0.2758 BURGLARY 0.3858 0.7121 0.6372 0.6229 1.0000 0.7921 0.5580 LARCENY 0.1019 0.6140 0.4467 0.4044 0.7921 1.0000 0.4442 AUTO 0.0688 0.3489 0.5907 0.2758 0.5580 0.4442 1.0000 More SAS Output Of Crime Example 0.09798342=0.22203947 - 0.12045606 Eigenvalues of the Correlation Matrix Eigenvalue Difference Proportion Cumulative 1 4.11495951 2.87623768 0.5879 0.5879 2 1.23872183 0.51290521 0.1770 0.7648 3 0.72581663 0.40938458 0.1037 0.8685 4 0.31643205 0.05845759 0.0452 0.9137 5 0.25797446 0.03593499 0.0369 0.9506 6 0.22203947 0.09798342 0.0317 0.9823 7 0.12405606 0.0177 1.0000 The proportion of variability explained by each principal component individually. This value equals the Eigenvalue/(sum of the Eigenvalues). The first two principal components captures 76.48% of the variation. If you include 6 of the 7 principal components you capture 98.23% of the variability. The 7th component only captures 1.77%. More SAS Output Of Crime Example Eigenvectors Prin1 Prin2 Prin3 Prin4 Prin5 Prin6 Prin7 MURDER 0.300279 -.629174 0.178245 -.232114 0.538123 0.259117 0.267593 RAPE 0.431759 -.169435 -.244198 0.062216 0.188471 -.773271 -.296485 ROBBERY 0.396875 0.042247 0.495861 -.557989 -.519977 -.114385 -.003903 ASSAULT 0.396652 -.343528 -.069510 0.629804 -.506651 0.172363 0.191745 BURGLARY 0.440157 0.203341 -.209895 -.057555 0.101033 0.535987 -.648117 LARCENY 0.357360 0.402319 -.539231 -.234890 0.030099 0.039406 0.601690 AUTO 0.295177 0.502421 0.568384 0.419238 0.369753 -.057298 0.147046 Prin1 has all positive values. This variable can be used as a proxy for overall crime rate. Prin2 has positive and negative values. Murder, Rape, and Assault are all negative (Violent Crimes). Robbery, Burglary, Larceny, and Auto are all positive (Property). This variable can be used for an understanding of Property vs. Violent crime. CRIME RATES PER 100,000 POPULATION BY STATE STATES LISTED IN ORDER OF OVERALL CRIME RATE AS DETERMINED BY THE FIRST PRINCIPAL COMPONENT Lowest 10 States and Then theTop 10 States CRIME RATES PER 100,000 POPULATION BY STATE. STATES LISTED IN ORDER OF PROPERTY VS. VIOLENT CRIME AS DETERMINED BY THE SECOND PRINCIPAL COMPONENT Lowest 10 States and Then theTop 10 States Correlation From SAS: First the Descriptive Statistics (A part of the output from Correlation) Correlation Matrix Correlation Matrix: Just the Variables Note that there is correlation among the crime rates. Correlation Matrix: Just the Principal Components Note that there is no correlation among the principal components. Correlation Matrix: Just the Principal Components Note the higher/very high correlations with the 1st few principal components and it decreases as it goes closer to the last principal component. What If We Told SAS to Produce Only 2 Principal Components? Eigenvalues of the Correlation Matrix Eigenvalue Difference Proportion Cumulative The 2 principal components produced when it is asked to produce only 2 principal components are exactly the same for when it produced all. 1 4.11495951 2 1.23872183 2.87623768 0.5879 0.5879 0.1770 0.7648 Eigenvectors Prin1 Prin2 MURDER 0.300279 -.629174 RAPE 0.431759 -.169435 ROBBERY 0.396875 0.042247 ASSAULT 0.396652 -.343528 BURGLARY 0.440157 0.203341 LARCENY 0.357360 0.402319 AUTO 0.295177 0.502421