advertisement

Dimensionality Reduction Notes Issues to consider: Types of features (numeric vs. symbolic) Objective (MSE reconstruction, ML, …) Type of transform (linear vs. non-linear) Some examples: LSA features symbolic objective MSE transform linear PCA numeric MSE linear PLSA symbolic ML linear Covered in Kevin’s lecture, 6/28 LSA = Latent Semantic Analysis Covered today PCA = Principal Component Analysis PLSA = probabilistic latent semantic analysis References: “Pattern classification” by Duda, Hart and Stork “The elements of statistical learning” by Hastie, Tibshirani and Friedman Principal Component Analysis Interpretations: Orthogonal vectors in directions of greatest variance Basis vectors v_i that give the smallest error in approximating x as: xhat = sum_i a_i v_i + m Solution: 1. Given data {x_1, …, x_n} where x_i \in \Re^d 2. Compute S= sum_{k=1}^n (x_k – m)(x_k – m)^t where m=[sum_k x_k]/n 3. Find the eigendecomposition of S (note that S is real and symmetric) 4. Choose the d’ eigenvectors with the biggest eigenvalues (min error approximation) 5. Reduced dimension feature vector = vector of coefficients for the d’ principal directions: a_i = v_i^t(x-m) Another view: Let X be the dxn data matrix (each column is a sample x_i) where data is zero-mean X = UDV^t is the singular value decomposition (SVD) of X S = X^tX/n = VD^2V^t/n so V are the principle components & D^2 the eigenvalues (variances in the principle directions) Again, choose the directions with the largest singular values. … so LSA is like PCA on word count vectors. Probabilistic Latent Semantic Analysis Interpretation: represent the distribution of x as a mixture and the reduced dimension representation is the vector of mixture component probabilities P(x) = sum_{i=1}^m p(z_i)p(x|z_i) mixture model z_i = latent variables, x = vector of counts of each symbol type (M symbols) p(z_i) and p(x|z_i) learned using the EM algorithms, m<M Reduced dimension representation: y = [y_1 y_2 … y_m]^t y_i= log p(z_i|x) Why is this a linear transform of x??? log p(z_i|x) = log p(x|z_i) + log p(z_i) – log p(x) (assume BOW) = sum_j log p(x_j|z_i) + log p(z_i) – log p(x) (assume p(x_j|z_i)=q_ij^{x_j} … unigram) = sum_j x_j log q_ij + K_i = x^t q_i + K_i