Part 1:
Dimensionality reduction is a technique in data analysis that simplifies the data by reducing the
number of variables or dimensions.
Dimensionality reduction should ensure:
Preserving relevance (retains most crucial information from high-dimensional spaces)
Eliminating redundancy (remove redundant and irrelevant attributes)
Transformation in lower-dimensional space
Motivations for it:
Improve model (performance + reduce overfitting)
Efficiency and resource management (computational efficiency and memory efficiency)
Interpretability and understanding (visualization)
Data quality enhancement (removal of redundant information)
The curse of dimensionality
Sparsity: when points are uniformly generated within a high dimensional hypercube, most
points are much farther from the center than expected.
In high dimensions, traditional distance metrics like Euclidean distance lose their
effectiveness.
Points become more dispersed, posing challenges for distance-based analysis.
Methods:
1- Feature extraction: transform original attributes into new ones. (Linear methods like PCA, or
non-linear methods like t-SNE)
2- Feature selection: selecting a subset of original attributes.
(filter methods: select attributes based on statistical properties, wrapper methods: user the
performance of a data mining model to evaluate the goodness of selected attributes.)
PCA:
1- Why PCA?
Correlation is a common characteristic of real-world datasets, often reflecting complex
relationships.
Examples:
Height and weight in individuals.
Study hours and test score relationships.
Data redundancy:
Correlations can indicate redundancy in the data.
Reducing redundancy can enhance the performance of data mining algorithms.
2- What is PCA?
Goal: transform original data into new variables that follow the variation in the data.
PCA's principle is to rotate the axis to align with the direction of the maximum data
variability.
After rotation, new axes called principal components are formed, the first principal
component PC1 captures most of the variation.
PCA identifies the directions, called principal components, along which the data varies the
most, first PC captures most of the variance within the data, subsequent components are
orthogonal (uncorrelated to the preceding ones).
PCA is widely used in data compression, visualization and noise reduction.
3- Objectives of PCA
Maximizing variance formulation (Hoteling 1933) and Minimizing error formulation (Pearson
1901)
Express the maximum variation within the first component.
Subsequent components express the remaining variation.
4- How to perform PCA?
Step 1: preprocessing: subtract the mean from each attribute (column) to center the data, scale
attributes if their scales differ.
Step 2: Covariance matrix (denoted as ∑)
Step 3: Compute eigenvalues and eigenvectors of the covariance matrix to find principal
components.
Step 4: Principal components selection (choose k < m eigenvector as principal components)
Step 5: Dimensionality reduction (reduce data by projecting it onto the k principal components.)
1. Centering data:
X = X − μ where X is the original data matrix (n ) and μ is the mean of each attribute.
Scaling attributes, if attributes have different scales, standardize them by scaling to unit
variance, this ensures PCA gives each feature equal importance.
2. Covariance matrix Calculation
∑
ov(X, ) =
∑
X X
=
n−1
n
i−1
X
n−1
*symmetric covariance matrix
3. Eigenvalues and Eigenvectors
∑ =
Diagonal matrix D consists of eigen values
Eigenvectors are arranged in the matrix P to enable data projection and dimensionality
reduction.
Spectral theorem ensure eigenvectors orthogonality.
4. Choosing k (number of principal components)
Sort eigenvalues in descending order and select
components.
n
eigenvectors to make them
5. Determining coordinates in principal components
(X the original n data matrix, and P the matrix of all eigen vectors of
covariance matrix)
X pc = X
To calculate coordinates in the new k dimensional space
X pc = X[, 1
] = X
Reconstruction of the data
To recover an approximation of the original data, perform the inverse of the projection.
X pprox = X pc
5- How to choose the number of components in PCA?
Scree plot (tool that aids in deciding how many principal components to retain)
Elbow point in the plot indicates a significant drop in eigenvalues, suggesting an optimal
number of components.
Applications of PCA?
Denoising (reduce noise in data or images)
Face recognition (identify and authenticate individuals from facial images.)
Data visualization (transform high dimensional data into a lower dimensional space for
visualization)
Data compression, anomaly detection, recommendation systems, etc.
Part 2: Feature Selection
1- Features Subset Selection (FSS)
- Definition: select a subset of features from a given features set.
- Goal: optimize an objective function for the chosen features.
- Feature Set X = x i = 1, , n
i
- Objective is to find a subset of features:
- = x , x , , x n that optimizes the objective function ()
i1
i2
i
- Optimization = rx()
Why doing feature selection and not sticking to feature extraction since it does the job?
Feature values may be expensive to collect and obtain, so test many features and choose
only few for the final implementation.
The need to extract meaningful rules from data mining models (since When you transform
the original features into new ones, the measurement units of the features are lost)
Features may not be numeric.
Fewer features means fewer parameters for pattern recognition.
Why it's Challenging:
For n = 100 there is 2
100
− 1
features subset
2- Unsupervised feature selection methods (typically statistical measures)
Variance (features with low variance are likely to be less informative and useless for prediction).
-> Remove features with a high percentage of identical values across all objects (low variance),
constant features lack useful information.
Correlation (features that are highly correlated with each other are likely to contain redundant
information.)
-> Remove highly correlated input numerical features, they indicate redundant information.
(Note: correlation can only detect linear relationships between features.)
Mutual information (Measures the amount of information that two features share, features with
high mutual information are more likely to be predictive of each other.)
-> Measures the amount of information that two categorical features share using entropies.
-> Mutual information can detect any kind of relationship between two features.
-> I (X ) = (X) − (X) = (X) () − (X, )
-> (X), () are individual entropies.
-> (X, ) is joint entropy.
-> I (X ) is mutual information.
3- Supervised Methods
3-1) Filter Methods