Machine Learning Fundamentals
MBZUAI Graduate Admission Preparation Lecture Notes
Kassa Adane Tarekegn
kassaadane16@gmail.com
Abstract
These comprehensive lecture notes cover fundamental machine learning concepts
essential for graduate admission preparation at MBZUAI. Topics include regression
analysis, gradient descent optimization, probability fundamentals, computational
complexity, linear algebra, classification algorithms, neural networks, and evaluation metrics. Each chapter includes practice questions with answer keys to test
understanding of key concepts.
1
MBZUAI Graduate Admission Preparation
Machine Learning Fundamentals
Contents
1 Introduction to Regression
1.1 Simple Linear Regression . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.1.1 Model Formulation . . . . . . . . . . . . . . . . . . . . . . . . . .
1.1.2 Worked Example . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.2 Multiple Linear Regression . . . . . . . . . . . . . . . . . . . . . . . . . .
1.2.1 Model Formulation . . . . . . . . . . . . . . . . . . . . . . . . . .
1.2.2 Worked Example . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.3 Cost Function and Optimization . . . . . . . . . . . . . . . . . . . . . . .
1.3.1 Mean Squared Error (MSE) . . . . . . . . . . . . . . . . . . . . .
1.3.2 Worked Example . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.4 Gradient Descent Optimization . . . . . . . . . . . . . . . . . . . . . . .
1.4.1 Update Rule . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.4.2 Worked Example . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.5 Model Evaluation Metrics . . . . . . . . . . . . . . . . . . . . . . . . . .
1.5.1 Worked Example . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.6 Assumptions of Linear Regression . . . . . . . . . . . . . . . . . . . . . .
1.7 Regularization Techniques . . . . . . . . . . . . . . . . . . . . . . . . . .
1.7.1 Ridge Regression (L2) . . . . . . . . . . . . . . . . . . . . . . . .
1.7.2 Lasso Regression (L1) . . . . . . . . . . . . . . . . . . . . . . . .
1.7.3 Elastic Net . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.7.4 Worked Example . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.8 Screening Exam Questions (25 Questions) . . . . . . . . . . . . . . . . .
5
5
5
5
5
5
6
6
6
6
6
6
7
7
7
7
7
7
8
8
8
8
2 Gradient Descent Optimization
2.1 Introduction to Gradient Descent . . . . . . . . . . . . . . . . . . . . . .
13
13
3 Mathematical Foundations
3.1 Core Objective . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.2 Update Rule . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
13
13
13
4 Convergence Theory
4.1 Smoothness and Convexity . . . . . . . . . . . . . . . . . . . . . . . . . .
4.2 Convergence Rates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
13
13
13
5 Variants of Gradient Descent
5.1 Batch Gradient Descent . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.2 Stochastic Gradient Descent (SGD) . . . . . . . . . . . . . . . . . . . . .
5.3 Mini-batch SGD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
14
14
14
14
6 Advanced Optimization Methods
6.1 Momentum Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
6.2 Adaptive Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
6.2.1 AdaGrad . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
6.2.2 RMSProp . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
6.2.3 Adam . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
14
14
14
14
14
14
7 Learning Rate Scheduling
15
2
MBZUAI Graduate Admission Preparation
Machine Learning Fundamentals
8 Theoretical Analysis
8.1 Optimality Conditions . . . . . . . . . . . . . . . . . . . . . . . . . . . .
8.2 Convergence Proof Sketch . . . . . . . . . . . . . . . . . . . . . . . . . .
15
15
15
9 Practical Considerations
9.1 Gradient Clipping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
9.2 Gradient Noise . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
9.3 Escaping Saddle Points . . . . . . . . . . . . . . . . . . . . . . . . . . . .
15
15
15
15
10 Recent Advances
10.1 Lookahead Optimizer . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
10.2 Sharpness-Aware Minimization (SAM) . . . . . . . . . . . . . . . . . . .
16
16
16
11 PhD Screening Questions
11.1 Theoretical Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
11.2 Practical Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
11.3 Mathematical Foundation . . . . . . . . . . . . . . . . . . . . . . . . . .
11.4 Variants of Gradient Descent . . . . . . . . . . . . . . . . . . . . . . . . .
11.4.1 Batch Gradient Descent . . . . . . . . . . . . . . . . . . . . . . .
11.4.2 Stochastic Gradient Descent (SGD) . . . . . . . . . . . . . . . . .
11.4.3 Mini-batch Gradient Descent . . . . . . . . . . . . . . . . . . . .
11.5 Advanced Optimization Techniques . . . . . . . . . . . . . . . . . . . . .
11.5.1 Momentum . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
11.5.2 Adaptive Learning Rates . . . . . . . . . . . . . . . . . . . . . . .
11.6 Screening Exam Practice Questions . . . . . . . . . . . . . . . . . . . . .
16
16
16
16
17
17
17
17
17
17
18
18
12 Probability Fundamentals for Machine Learning
12.1 Introduction to Probability in ML . . . . . . . . . . . . . . . . . . . . . .
12.2 Basic Probability Concepts . . . . . . . . . . . . . . . . . . . . . . . . . .
12.2.1 Probability Axioms . . . . . . . . . . . . . . . . . . . . . . . . . .
12.2.2 Conditional Probability . . . . . . . . . . . . . . . . . . . . . . .
12.2.3 Bayes’ Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . .
12.3 Random Variables and Distributions . . . . . . . . . . . . . . . . . . . .
12.4 Expectation and Variance . . . . . . . . . . . . . . . . . . . . . . . . . .
12.5 Key Probability Rules . . . . . . . . . . . . . . . . . . . . . . . . . . . .
12.5.1 Law of Total Probability . . . . . . . . . . . . . . . . . . . . . . .
12.5.2 Independence . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
12.6 Bayesian Inference in Machine Learning . . . . . . . . . . . . . . . . . . .
12.6.1 Bayesian Parameter Estimation . . . . . . . . . . . . . . . . . . .
12.6.2 Conjugate Priors . . . . . . . . . . . . . . . . . . . . . . . . . . .
12.7 Probabilistic Models in ML . . . . . . . . . . . . . . . . . . . . . . . . .
12.7.1 Naive Bayes Classifier . . . . . . . . . . . . . . . . . . . . . . . .
12.7.2 Gaussian Mixture Models (GMM) . . . . . . . . . . . . . . . . . .
12.8 Information Theory Basics . . . . . . . . . . . . . . . . . . . . . . . . . .
12.8.1 Entropy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
12.8.2 Cross-Entropy and KL Divergence . . . . . . . . . . . . . . . . . .
12.9 Screening Exam Practice Questions . . . . . . . . . . . . . . . . . . . . .
18
18
19
19
19
19
19
20
20
20
20
20
20
21
21
21
21
21
21
21
22
3
MBZUAI Graduate Admission Preparation
Machine Learning Fundamentals
13 Computational Complexity & Data Structures
13.1 Introduction to Computational Complexity . . . . . . . . . . . . . . . . .
13.2 Asymptotic Notation . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
13.2.1 Big-O Notation (Upper Bound) . . . . . . . . . . . . . . . . . . .
13.2.2 Big-Ω Notation (Lower Bound) . . . . . . . . . . . . . . . . . . .
13.2.3 Big-Θ Notation (Tight Bound) . . . . . . . . . . . . . . . . . . .
13.3 Fundamental Data Structures . . . . . . . . . . . . . . . . . . . . . . . .
13.3.1 Arrays . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
13.3.2 Linked Lists . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
13.3.3 Hash Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
13.3.4 Trees . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
13.3.5 Heaps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
13.4 Graph Data Structures . . . . . . . . . . . . . . . . . . . . . . . . . . . .
13.5 Machine Learning Complexity . . . . . . . . . . . . . . . . . . . . . . . .
13.6 Screening Exam Practice Questions . . . . . . . . . . . . . . . . . . . . .
22
22
22
22
22
23
23
23
23
23
24
24
24
24
25
14 Linear Algebra for Machine Learning
14.1 Introduction to Linear Algebra in ML . . . . . . . . . . . . . . . . . . . .
14.2 Vectors and Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
14.2.1 Basic Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . .
14.3 Eigenvalues and Eigenvectors . . . . . . . . . . . . . . . . . . . . . . . .
14.4 Matrix Decompositions . . . . . . . . . . . . . . . . . . . . . . . . . . . .
14.4.1 Eigen Decomposition . . . . . . . . . . . . . . . . . . . . . . . . .
14.4.2 Singular Value Decomposition (SVD) . . . . . . . . . . . . . . . .
14.5 Principal Component Analysis (PCA) . . . . . . . . . . . . . . . . . . . .
14.6 Linear Algebra in ML Applications . . . . . . . . . . . . . . . . . . . . .
14.6.1 Linear Regression . . . . . . . . . . . . . . . . . . . . . . . . . . .
14.6.2 Neural Networks . . . . . . . . . . . . . . . . . . . . . . . . . . .
14.7 Screening Exam Practice Questions . . . . . . . . . . . . . . . . . . . . .
25
25
25
25
26
26
26
26
26
27
27
27
27
15 Classification Algorithms
15.1 Introduction to Classification . . . . . . . . . . . . . . . . . . . . . . . .
15.2 Logistic Regression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
15.3 Support Vector Machines (SVM) . . . . . . . . . . . . . . . . . . . . . .
15.4 Decision Trees . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
15.5 Ensemble Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
15.5.1 Random Forests . . . . . . . . . . . . . . . . . . . . . . . . . . . .
15.5.2 Gradient Boosting . . . . . . . . . . . . . . . . . . . . . . . . . .
15.6 k-Nearest Neighbors (k-NN) . . . . . . . . . . . . . . . . . . . . . . . . .
15.7 Naive Bayes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
15.8 Screening Exam Practice Questions . . . . . . . . . . . . . . . . . . . . .
27
27
28
28
28
28
28
28
28
29
29
16 Neural Networks & Deep Learning
16.1 Introduction to Neural Networks . . . . . . . . . . . . . . . . . . . . . .
16.2 Perceptron and MLP . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
16.2.1 Single Perceptron . . . . . . . . . . . . . . . . . . . . . . . . . . .
16.2.2 Multilayer Perceptron (MLP) . . . . . . . . . . . . . . . . . . . .
16.3 Activation Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
16.4 Backpropagation Algorithm . . . . . . . . . . . . . . . . . . . . . . . . .
29
29
29
29
30
30
30
4
MBZUAI Graduate Admission Preparation
Machine Learning Fundamentals
16.5 Convolutional Neural Networks (CNN) . . . . . . . . . . . . . . . . . . .
16.6 Recurrent Neural Networks (RNN) . . . . . . . . . . . . . . . . . . . . .
16.7 Regularization Techniques . . . . . . . . . . . . . . . . . . . . . . . . . .
16.8 Screening Exam Practice Questions . . . . . . . . . . . . . . . . . . . . .
30
30
31
31
17 Evaluation Metrics & Model Selection
17.1 Introduction to Model Evaluation . . . . . . . . . . . . . . . . . . . . . .
17.2 Cross-Validation Methods . . . . . . . . . . . . . . . . . . . . . . . . . .
17.3 Classification Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
17.3.1 Confusion Matrix . . . . . . . . . . . . . . . . . . . . . . . . . . .
17.3.2 Key Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
17.4 ROC Curves and AUC . . . . . . . . . . . . . . . . . . . . . . . . . . . .
17.5 Regression Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
17.6 Bias-Variance Tradeoff . . . . . . . . . . . . . . . . . . . . . . . . . . . .
17.7 Hyperparameter Tuning . . . . . . . . . . . . . . . . . . . . . . . . . . .
17.8 Loss Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
17.9 Screening Exam Practice Questions . . . . . . . . . . . . . . . . . . . . .
31
31
32
32
32
32
32
32
33
33
33
33
5
MBZUAI Graduate Admission Preparation
1
Machine Learning Fundamentals
Introduction to Regression
Regression analysis is a fundamental statistical method used to model the relationship
between a dependent variable and one or more independent variables.
Key Characteristics:
• Predicts continuous numerical outcomes
• Establishes functional relationships between variables
• Used for forecasting, trend analysis, and causal inference
Applications:
• House price prediction
• Stock market forecasting
• Sales prediction
• Risk analysis
1.1
Simple Linear Regression
1.1.1
Model Formulation
y = β0 + β1 x + ε
Components:
• y: Dependent variable (target)
• x: Independent variable (feature)
• β0 : Intercept (bias term)
• β1 : Slope (coefficient)
• ε: Error term (residual)
1.1.2
Worked Example
Predicting house prices based on square footage:
Price = 50, 000 + 200 × SquareFootage
Predictions:
• 1500 sq ft: 50, 000 + 200 × 1500 = $350, 000
• 2000 sq ft: 50, 000 + 200 × 2000 = $450, 000
1.2
Multiple Linear Regression
1.2.1
Model Formulation
y = β 0 + β 1 x1 + β 2 x2 + · · · + β n xn + ε
6
MBZUAI Graduate Admission Preparation
1.2.2
Machine Learning Fundamentals
Worked Example
House price prediction with multiple features:
Price = 20, 000 + 150 × Size + 10, 000 × Bedrooms − 5, 000 × Age
Prediction for:
• Size: 1800 sq ft, Bedrooms: 3, Age: 10 years
• Price = 20, 000 + 150 × 1800 + 10, 000 × 3 − 5, 000 × 10
• Price = 20, 000 + 270, 000 + 30, 000 − 50, 000 = $270, 000
1.3
Cost Function and Optimization
1.3.1
Mean Squared Error (MSE)
m
1 X
(hβ (x(i) ) − y (i) )2
J(β) =
2m i=1
Where:
• m: Number of training examples
• hβ (x(i) ): Predicted value for i-th example
• y (i) : Actual value for i-th example
1.3.2
Worked Example
Given 3 data points: (1,2), (2,3), (3,4) and model y = 1 + x
MSE Calculation:
1 (2 − 2)2 + (3 − 3)2 + (4 − 4)2
2×3
1
= × [0 + 0 + 0] = 0
6
J(β) =
1.4
Gradient Descent Optimization
1.4.1
Update Rule
m
1 X
(i)
βj := βj − α
(hβ (x(i) ) − y (i) )xj
m i=1
Parameters:
• α: Learning rate
• m: Number of training examples
7
MBZUAI Graduate Admission Preparation
1.4.2
Machine Learning Fundamentals
Worked Example
Initial: β0 = 0, β1 = 0, α = 0.1, Data: (1,1), (2,2)
First iteration:
1
β0 := 0 − 0.1 × × [(0 − 1) + (0 − 2)] = 0 − 0.1 × 0.5 × (−3) = 0.15
2
1
β1 := 0 − 0.1 × × [(0 − 1) × 1 + (0 − 2) × 2] = 0 − 0.1 × 0.5 × (−5) = 0.25
2
1.5
Model Evaluation Metrics
Table 1: Regression Evaluation Metrics
Metric
Formula and Interpretation
Pm
1
Mean Squared Error (MSE)
(y − ŷi )2
n Pi=1 i
m
1
Mean Absolute Error (MAE)
i=1 |yi − ŷi |
m
P
m
(yi −ŷi )2
2
R-squared (R )
1 − Pi=1
m
2
i=1 (yi −ȳ)
√
Root Mean Squared Error (RMSE)
MSE
1.5.1
Worked Example
Actual values: [2, 3, 5, 7], Predicted: [1.8, 3.2, 4.9, 7.1]
Calculations:
1
MSE = [(2 − 1.8)2 + (3 − 3.2)2 + (5 − 4.9)2 + (7 − 7.1)2 ] = 0.0225
4
1
MAE = [0.2 + 0.2 + 0.1 + 0.1] = 0.15
4
0.09
R2 = 1 −
≈ 0.9965
26
1.6
Assumptions of Linear Regression
1. Linearity: Relationship between features and target is linear
2. Independence: Observations are independent of each other
3. Homoscedasticity: Constant variance of errors
4. Normality: Errors are normally distributed
5. No Multicollinearity: Features are not highly correlated
6. No Endogeneity: No correlation between features and errors
1.7
Regularization Techniques
1.7.1
Ridge Regression (L2)
J(β) = MSE + λ
n
X
j=1
8
βj2
MBZUAI Graduate Admission Preparation
1.7.2
Machine Learning Fundamentals
Lasso Regression (L1)
J(β) = MSE + λ
n
X
|βj |
j=1
1.7.3
Elastic Net
J(β) = MSE + λ1
n
X
|βj | + λ2
j=1
Method
βj2
j=1
Table 2: Comparison of Regularization Methods
Advantages
Disadvantages
Ridge
Lasso
Elastic Net
1.7.4
n
X
Handles multicollinearity,
stable solutions
Performs feature selection,
sparse solutions
Combines benefits of both
No feature selection
Unstable with correlated
features
More hyperparameters to
tune
Worked Example
For model: y = 2 + 3x1 + 4x2 , with λ = 0.1
Ridge Penalty: 0.1 × (32 + 42 ) = 0.1 × 25 = 2.5
Lasso Penalty: 0.1 × (|3| + |4|) = 0.1 × 7 = 0.7
1.8
Screening Exam Questions (25 Questions)
1. Which of the following is NOT a characteristic of regression analysis?
(A) Predicts continuous outcomes
(B) Models relationships between variables
(C) Classifies data into categories
(D) Used for forecasting
2. In the linear regression equation y = β0 + β1 x + ε, what does ε represent?
(A) Independent variable
(B) Dependent variable
(C) Coefficient
(D) Error term
3. For a house price prediction model: Price = 50,000 + 200 SquareFootage, what is
the predicted price for a 1800 sq ft house?
(A) $50,000
9
MBZUAI Graduate Admission Preparation
Machine Learning Fundamentals
(B) $200,000
(C) $410,000
(D) $360,000
4. Multiple linear regression differs from simple linear regression in that it:
(A) Uses multiple dependent variables
(B) Uses multiple independent variables
(C) Has multiple error terms
(D) Is non-linear
5. The Mean Squared Error (MSE) cost function is defined as:
P
|yi − ŷi |
(A) m1
P
(B) m1 (yi − ŷi )2
P
(C) m1 (yi − ŷi )
q P
1
(D)
(yi − ŷi )2
m
6. In gradient descent, the learning rate parameter:
(A) Determines the number of iterations
(B) Controls the step size during updates
(C) Initializes the weights
(D) Computes the gradient
7. Given actual values [3, 5, 7] and predictions [2.8, 5.2, 6.9], the MAE is:
(A) 0.1
(B) 0.2
(C) 0.3
(D) 0.4
8. A high R-squared value close to 1 indicates:
(A) Poor model fit
(B) Good model fit
(C) Overfitting
(D) Underfitting
9. Which assumption of linear regression requires that errors have constant variance?
(A) Linearity
(B) Independence
(C) Homoscedasticity
(D) Normality
10
MBZUAI Graduate Admission Preparation
Machine Learning Fundamentals
10. Ridge regression adds a penalty proportional to:
(A) Square of coefficients
(B) Absolute value of coefficients
(C) Cube of coefficients
(D) Logarithm of coefficients
11. Lasso regression is particularly useful for:
(A) Handling non-linearity
(B) Feature selection
(C) Increasing model complexity
(D) Reducing training time
12. The gradient descent update rule for βj is:
∂J
(A) βj := βj + α ∂β
j
∂J
(B) βj := βj − α ∂β
j
∂J
(C) βj := α − βj ∂β
j
∂J
(D) βj := αβj − ∂β
j
13. Multicollinearity in regression refers to:
(A) High correlation between features and target
(B) High correlation between features
(C) No correlation between features
(D) Non-linear relationships
14. For a regression model with 5 features, how many coefficients (excluding intercept)
are learned?
(A) 4
(B) 5
(C) 6
(D) Depends on sample size
15. Which metric is most sensitive to outliers?
(A) MAE
(B) MSE
(C) R-squared
(D) All equally sensitive
16. The normal equation for linear regression is:
(A) X T y
11
MBZUAI Graduate Admission Preparation
Machine Learning Fundamentals
(B) (X T X)−1 y
(C) (X T X)−1 X T y
(D) Xy T (XX T )−1
17. Regularization helps to:
(A) Increase training speed
(B) Prevent overfitting
(C) Handle categorical variables
(D) Improve computational efficiency
P
18. In the cost function J(β) = MSE + λ |βj |, the λ parameter:
(A) Learning rate
(B) Regularization strength
(C) Number of iterations
(D) Error tolerance
19. Polynomial regression is used when:
(A) There are multiple features
(B) Data has categorical variables
(C) Relationship is non-linear
(D) Dataset is very large
20. The bias-variance tradeoff suggests that:
(A) Models should minimize bias
(B) Increasing complexity reduces bias but increases variance
(C) Simple models always perform better
(D) Bias and variance can be minimized simultaneously
21. Feature scaling is important for gradient descent because it:
(A) Increases dimensionality
(B) Makes cost function convex
(C) Ensures parameters update at similar rates
(D) Reduces need for learning rate
22. A learning rate that is too large in gradient descent may cause:
(A) Slow convergence
(B) Getting stuck in local minima
(C) Oscillations or divergence
(D) Perfect convergence
12
MBZUAI Graduate Admission Preparation
Machine Learning Fundamentals
23. The main purpose of using a validation set is to:
(A) Increase training data
(B) Tune hyperparameters
(C) Reduce model complexity
(D) Handle missing values
24. In k-fold cross-validation, the data is divided into:
(A) Training and test sets
(B) k equal-sized folds
(C) Random subsets
(D) Feature groups
25. Elastic Net regularization combines:
(A) L1 and L2 regularization
(B) Ridge and Linear regression
(C) Lasso and Ridge regularization
(D) Polynomial and Linear regression
Answer Key
1. C 2. D 3. C 4. B 5. B
11. B 12. B 13. B 14. B
21. C 22. C 23. B 24. B
6. B 7. B 8. B 9. C 10. A
15. B 16. C 17. B 18. B 19. C
25. C
13
20. B
MBZUAI Graduate Admission Preparation
Machine Learning Fundamentals
2
Gradient Descent Optimization
2.1
Introduction to Gradient Descent
Gradient Descent is a first-order iterative optimization algorithm for finding the local
minimum of a differentiable function. It is the backbone of most machine learning optimization problems.
Key Characteristics:
• Iterative parameter update method
• Uses gradient information to navigate parameter space
• Guaranteed to converge to local minimum for convex functions
• Foundation for training neural networks and other ML models
3
Mathematical Foundations
3.1
Core Objective
Given a differentiable function f : Rd → R, find:
θ∗ = arg min f (θ)
θ∈Rd
3.2
Update Rule
θt+1 = θt − ηt ∇f (θt )
where ηt > 0 is the learning rate at iteration t.
4
Convergence Theory
4.1
Smoothness and Convexity
• L-smooth: k∇f (x) − ∇f (y)k ≤ Lkx − yk
• µ-strongly convex: f (y) ≥ f (x) + ∇f (x)T (y − x) + µ2 ky − xk2
4.2
Convergence Rates
√
• Non-convex, L-smooth: O(1/ T ) for gradient norm
• Convex, L-smooth: O(1/T ) for function value
• Strongly convex, L-smooth: O(exp(−T µ/L)) linear rate
14
MBZUAI Graduate Admission Preparation
Machine Learning Fundamentals
5
Variants of Gradient Descent
5.1
Batch Gradient Descent
Uses full dataset:
n
θt+1 = θt − ηt ∇f (θt ) = θt − ηt
5.2
1X
∇fi (θt )
n i=1
Stochastic Gradient Descent (SGD)
Single random sample:
θt+1 = θt − ηt ∇fit (θt )
5.3
Mini-batch SGD
Small random subset:
θt+1 = θt − ηt
1X
∇fi (θt )
b i∈B
t
6
Advanced Optimization Methods
6.1
Momentum Methods
vt = γvt−1 + η∇f (θt )
θt+1 = θt − vt
6.2
Adaptive Methods
6.2.1
AdaGrad
θt+1 = θt − √
where Gt =
6.2.2
Pt
i=1 gi
η
Gt + gi
RMSProp
vt = βvt−1 + (1 − β)gt
η
θt+1 = θt − √
gt
vt + 6.2.3
gt
gt
Adam
mt = β1 mt−1 + (1 − β1 )gt
vt = β2 vt−1 + (1 − β2 )gt gt
mt
vt
m̂t =
, v̂t =
t
1 − β1
1 − β2t
η
θt+1 = θt − √
m̂t
v̂t + 15
MBZUAI Graduate Admission Preparation
7
Machine Learning Fundamentals
Learning Rate Scheduling
• Constant: Fixed η
√
• Decaying: ηt = η0 / t or ηt = η0 /t
• Cosine: ηt = 21 η0 (1 + cos(πt/T ))
• Cyclical: Varies between bounds periodically
• Warmup: Linear increase then decay
8
Theoretical Analysis
8.1
Optimality Conditions
• First-order: ∇f (θ∗ ) = 0
• Second-order: ∇2 f (θ∗ ) 0
8.2
Convergence Proof Sketch
For L-smooth functions:
f (θt+1 ) ≤ f (θt ) − η(1 −
ηL
)k∇f (θt )k2
2
Choosing η < 2/L ensures descent.
9
Practical Considerations
9.1
Gradient Clipping
Prevent exploding gradients:
τ
gt ← gt · min 1,
kgt k
9.2
Gradient Noise
SGD introduces noise:
E[∇fi (θ)] = ∇f (θ),
9.3
Var[∇fi (θ)] = σ 2
Escaping Saddle Points
• Noise in SGD helps escape saddle points
• Adaptive methods can get stuck
• Perturbed gradient descent guarantees escape
16
MBZUAI Graduate Admission Preparation
10
Recent Advances
10.1
Lookahead Optimizer
Machine Learning Fundamentals
Maintains fast and slow weights:
θt,k = θt,k−1 − η∇f (θt,k−1 )
θt+1 = θt+1 + α(θt,K − θt )
10.2
Sharpness-Aware Minimization (SAM)
min max f (θ + )
θ
kk≤ρ
11
PhD Screening Questions
11.1
Theoretical Questions
1. Prove convergence rate of GD for strongly convex functions
2. Compare SGD vs batch GD convergence rates
3. Analyze Adam’s bias correction terms
4. Discuss conditions for escaping saddle points
11.2
Practical Questions
1. When to use adaptive vs momentum methods?
2. How to choose learning rate schedules?
3. Handle vanishing/exploding gradients
4. Optimizer selection for different architectures
11.3
Mathematical Foundation
The core update rule for gradient descent:
θ := θ − α∇θ J(θ)
Where:
• θ: Parameters (weights) to be optimized
• α: Learning rate (step size)
• ∇θ J(θ): Gradient of cost function w.r.t parameters
• J(θ): Cost function to be minimized
17
MBZUAI Graduate Admission Preparation
11.4
Variants of Gradient Descent
11.4.1
Batch Gradient Descent
Machine Learning Fundamentals
Uses entire dataset to compute gradient:
m
1 X
∇θ J(θ; x(i) , y (i) )
θ := θ − α
m i=1
Advantages: Stable convergence, deterministic
Disadvantages: Computationally expensive for large datasets
11.4.2
Stochastic Gradient Descent (SGD)
Uses single training example per iteration:
θ := θ − α∇θ J(θ; x(i) , y (i) )
Advantages: Faster updates, escapes local minima
Disadvantages: Noisy convergence, may oscillate
11.4.3
Mini-batch Gradient Descent
Compromise between Batch and SGD:
b
1X
∇θ J(θ; x(i) , y (i) )
θ := θ − α
b i=1
Where b is the batch size (typically 32, 64, 128, 256).
Learning Rate
Table 3: Learning Rate Effects
Too Small
Appropriate
Convergence
Very slow
Behavior
Result
Too Large
Smooth conver- May diverge
gence
Gets stuck in flat Stable descent
Oscillations
regions
Underfitting
Optimal
solu- Overfitting/Divergence
tion
11.5
Advanced Optimization Techniques
11.5.1
Momentum
Accumulates past gradients to smooth updates:
vt = γvt−1 + α∇θ J(θ)
θ = θ − vt
Where γ is the momentum term (typically 0.9).
18
MBZUAI Graduate Admission Preparation
11.5.2
Machine Learning Fundamentals
Adaptive Learning Rates
AdaGrad: Adapts learning rate per parameter:
α
θt+1 = θt − √
∇θ J(θt )
Gt + RMSProp: Improves AdaGrad with exponential decay:
E[g 2 ]t = γE[g 2 ]t−1 + (1 − γ)gt2
Adam: Combines Momentum and RMSProp:
mt = β1 mt−1 + (1 − β1 )gt
vt = β2 vt−1 + (1 − β2 )gt2
m̂t
θt+1 = θt − α √
v̂t + 11.6
Screening Exam Practice Questions
1. What is the primary purpose of the learning rate in gradient descent?
(A) To initialize the weights randomly
(B) To determine the number of training iterations
(C) To control the step size during parameter updates
(D) To compute the gradient of the cost function
2. In gradient descent, the negative gradient points in the direction of:
(A) Steepest ascent of the cost function
(B) Steepest descent of the cost function
(C) Maximum curvature of the cost function
(D) The global minimum regardless of starting point
Answer Key: 1. C, 2. B, 3. C, 4. C, 5. B, 6. B, 7. C, 8. D, 9. C, 10. C, 11. B, 12.
C, 13. B, 14. B, 15. C
12
Probability Fundamentals for Machine Learning
12.1
Introduction to Probability in ML
Probability theory provides the mathematical foundation for uncertainty quantification
in machine learning. It is essential for Bayesian methods, probabilistic modeling, and
evaluation of ML systems.
Key Applications in ML:
• Bayesian learning and inference
• Probabilistic graphical models
• Uncertainty quantification in predictions
• Model evaluation and comparison
• Naive Bayes classifiers
19
MBZUAI Graduate Admission Preparation
12.2
Basic Probability Concepts
12.2.1
Probability Axioms
Machine Learning Fundamentals
For any event A in sample space S:
1. P (A) ≥ 0
2. P (S) = 1
S
P
3. If A1 , A2 , . . . are mutually exclusive, then P ( i Ai ) = i P (Ai )
12.2.2
Conditional Probability
The probability of event A given event B has occurred:
P (A|B) =
12.2.3
P (A ∩ B)
,
P (B)
where P (B) > 0
Bayes’ Theorem
The foundation of Bayesian inference:
P (A|B) =
P (B|A) · P (A)
P (B)
Where:
• P (A|B): Posterior probability
• P (B|A): Likelihood
• P (A): Prior probability
• P (B): Evidence
12.3
Random Variables and Distributions
Distribution
Table 4: Common Discrete Distributions
PMF
Parameters ML Application
Bernoulli
Binomial
Poisson
P (X = k) = pk(1 − p)1−k
P (X = k) = nk pk (1 − p)n−k
k e−λ
P (X = k) = λ k!
20
p
n, p
λ
Binary classification
Count successes in trials
Event counts in time
MBZUAI Graduate Admission Preparation
Machine Learning Fundamentals
Table 5: Common Continuous Distributions
Distribution PDF
Parameters ML Application
Normal
Exponential
Beta
12.4
(x−µ)2
f (x) = σ√12π e− 2σ2
f (x) = λe−λx
α−1 (1−x)β−1
f (x) = x B(α,β)
µ, σ
λ
α, β
Error distributions
Time between events
Bayesian priors
Expectation and Variance
Expected Value:
• For discrete X: E[X] =
P
• For continuous X: E[X] =
x x · P (X = x)
R∞
−∞
xf (x)dx
• Linearity: E[aX + bY ] = aE[X] + bE[Y ]
Variance and Covariance:
• Variance: Var(X) = E[(X − E[X])2 ] = E[X 2 ] − (E[X])2
• Covariance: Cov(X, Y ) = E[(X − E[X])(Y − E[Y ])]
12.5
Key Probability Rules
12.5.1
Law of Total Probability
For partition B1 , B2 , . . . , Bn of sample space:
P (A) =
n
X
P (A|Bi )P (Bi )
i=1
Example: From a bag of 6 red and 2 blue balls, two balls drawn without replacement:
P (2nd red) = P (1st red) · P (2nd red|1st red)
+ P (1st blue) · P (2nd red|1st blue)
6 5 2 6
42
3
= · + · =
=
8 7 8 7
56
4
12.5.2
Independence
Events A and B are independent if:
P (A ∩ B) = P (A)P (B) or P (A|B) = P (A)
12.6
Bayesian Inference in Machine Learning
12.6.1
Bayesian Parameter Estimation
Given data D and parameters θ:
P (θ|D) =
P (D|θ)P (θ)
∝ P (D|θ)P (θ)
P (D)
21
MBZUAI Graduate Admission Preparation
12.6.2
Machine Learning Fundamentals
Conjugate Priors
Table 6: Common Conjugate Prior Distributions
Likelihood
Conjugate Prior Posterior
Bernoulli
Beta
Binomial
Beta
Poisson
Gamma
Normal (known variance) Normal
Multinomial
Dirichlet
12.7
Probabilistic Models in ML
12.7.1
Naive Bayes Classifier
Beta
Beta
Gamma
Normal
Dirichlet
Assumes feature independence given class:
P (Y |X1 , . . . , Xn ) ∝ P (Y )
n
Y
P (Xi |Y )
i=1
Application: Text classification, spam detection
12.7.2
Gaussian Mixture Models (GMM)
Probability density as weighted sum of Gaussians:
p(x) =
K
X
πk N (x|µk , Σk )
k=1
12.8
Information Theory Basics
12.8.1
Entropy
Measure of uncertainty in a random variable:
X
H(X) = −
P (x) log P (x)
x
12.8.2
Cross-Entropy and KL Divergence
P
• Cross-entropy: H(P, Q) = − x P (x) log Q(x)
P
P (x)
• KL Divergence: DKL (P kQ) = x P (x) log Q(x)
ML Application: Loss functions in classification.
22
MBZUAI Graduate Admission Preparation
12.9
Machine Learning Fundamentals
Screening Exam Practice Questions
1. From a bag of 6 red and 4 blue balls, two balls are drawn without replacement.
What is the probability that both balls are red?
(A) 31
(B) 12
(C) 14
(D) 13
2. If events A and B are independent, which of the following is TRUE?
(A) P (A ∩ B) = 0
(B) P (A|B) = P (B)
(C) P (A ∩ B) = P (A)P (B)
(D) P (A ∪ B) = P (A) + P (B)
Answer Key: 1. A, 2. C, 3. D, 4. B, 5. C, 6. B, 7. B, 8. C, 9. B, 10. C, 11. C, 12.
C, 13. D, 14. B, 15. B
13
Computational Complexity & Data Structures
13.1
Introduction to Computational Complexity
Computational complexity theory provides a framework for understanding the efficiency
of algorithms and the inherent difficulty of computational problems. It is crucial for
designing scalable machine learning systems.
Key Concepts:
• Time complexity: How runtime grows with input size
• Space complexity: How memory usage grows with input size
• Asymptotic analysis: Behavior as input size approaches infinity
• Trade-offs between time and space efficiency
13.2
Asymptotic Notation
13.2.1
Big-O Notation (Upper Bound)
O(g(n)) represents functions that grow no faster than g(n):
O(g(n)) = {f (n) : ∃c > 0, n0 > 0 such that 0 ≤ f (n) ≤ c · g(n) ∀n ≥ n0 }
13.2.2
Big-Ω Notation (Lower Bound)
Ω(g(n)) represents functions that grow at least as fast as g(n):
Ω(g(n)) = {f (n) : ∃c > 0, n0 > 0 such that 0 ≤ c · g(n) ≤ f (n) ∀n ≥ n0 }
23
MBZUAI Graduate Admission Preparation
13.2.3
Machine Learning Fundamentals
Big-Θ Notation (Tight Bound)
Θ(g(n)) represents functions that grow at the same rate as g(n):
Θ(g(n)) = O(g(n)) ∩ Ω(g(n))
Table 7: Common Time Complexities
Complexity Growth Rate ML Application
O(1)
O(log n)
O(n)
O(n log n)
O(n2 )
O(2n )
O(n!)
Constant
Logarithmic
Linear
Linearithmic
Quadratic
Exponential
Factorial
Parameter lookup
Decision trees
Iterating through dataset
Training random forests
Nearest neighbors
Exact inference in Bayes nets
Brute force optimization
13.3
Fundamental Data Structures
13.3.1
Arrays
• Structure: Contiguous memory locations
• Access: O(1) random access
• Search: O(n) linear search, O(log n) if sorted
• Insert/Delete: O(n) at beginning/middle
13.3.2
Linked Lists
• Structure: Nodes with data and pointer to next node
• Access: O(n) sequential access
• Search: O(n)
• Insert/Delete: O(1) at known position
13.3.3
Hash Tables
• Structure: Array with hash function mapping keys to indices
• Average Case: O(1) insert, delete, search
• Worst Case: O(n) due to collisions
of entries
• Load Factor: α = number
table size
24
MBZUAI Graduate Admission Preparation
13.3.4
Machine Learning Fundamentals
Trees
• Binary Search Tree:
– Search: O(h) where h is height
– Balanced: O(log n), Degenerate: O(n)
• AVL/Red-Black Trees: Self-balancing, O(log n) operations
• B-Trees: Designed for disk-based systems, used in databases
13.3.5
Heaps
• Structure: Complete binary tree with heap property
• Insert: O(log n)
• Extract Min/Max: O(log n)
• Peek: O(1)
• Application: Priority queues, heap sort
13.4
Graph Data Structures
Table 8: Graph Representation Complexities
Method
Space
Check Edge List Neighbors
Adjacency Matrix
Adjacency List
Edge List
13.5
O(V 2 )
O(1)
O(V + E) O(deg(v))
O(E)
O(E)
O(V )
O(deg(v))
O(E)
Machine Learning Complexity
Table 9: ML Algorithm Time Complexities
Algorithm
Training
Prediction
Linear Regression
Logistic Regression
Decision Tree
k-NN
Neural Network
O(n2 d + d3 )
O(nd · iterations)
O(nd log n)
O(1)
O(nd · layers · iterations)
25
O(d)
O(d)
O(depth)
O(nd)
O(layers · units)
MBZUAI Graduate Admission Preparation
13.6
Machine Learning Fundamentals
Screening Exam Practice Questions
1. In the worst case, the number of comparisons needed to search a singly linked list
of length n for a given element is:
(A) O(log n)
(B) O(n/2)
(C) O(log n − 1)
(D) O(n)
2. Which data structure provides O(1) average case time complexity for insert, delete,
and search operations?
(A) Binary search tree
(B) Linked list
(C) Hash table
(D) Array
Answer Key: 1. D, 2. C, 3. C, 4. C, 5. C, 6. B, 7. B, 8. B, 9. B, 10. B, 11. C, 12.
C, 13. B, 14. C, 15. C, 16. C, 17. C, 18. C
14
Linear Algebra for Machine Learning
14.1
Introduction to Linear Algebra in ML
Linear algebra forms the mathematical foundation for most machine learning algorithms.
Understanding vectors, matrices, and their operations is crucial for implementing and
understanding ML models.
Key Applications:
• Data representation as vectors and matrices
• Model parameter optimization
• Dimensionality reduction
• Neural network operations
14.2
Vectors and Matrices
14.2.1
Basic Definitions
v1
v2
• Vector: Ordered list of numbers, e.g., v = ..
.
vn
a11 a12
• Matrix: 2D array of numbers, e.g., A =
a21 a22
• Tensor: Multi-dimensional array (generalization of vectors and matrices)
26
MBZUAI Graduate Admission Preparation
Machine Learning Fundamentals
Table 10: Essential Matrix Operations
Operation
Definition and Properties
P
Matrix Multiplication C = AB where cij = k aik bkj , Not commutative: AB 6= BA in general
Transpose
AT where aTij = aji , (AB)T = B T AT
Inverse
AA−1 = A−1 A = I, Only defined for square,
non-singular matrices
Determinant
Scalar value, denoted |A| or det(A), Zero determinant indicates singular matrix
14.3
Eigenvalues and Eigenvectors
For a square matrix A, an eigenvector v and eigenvalue λ satisfy:
Av = λv
Properties:
• Characteristic equation: det(A − λI) = 0
P
• Trace:
λi = tr(A)
Q
• Determinant:
λi = det(A)
14.4
Matrix Decompositions
14.4.1
Eigen Decomposition
For diagonalizable matrix A:
A = QΛQ−1
Where Q contains eigenvectors and Λ is diagonal with eigenvalues.
14.4.2
Singular Value Decomposition (SVD)
For any matrix Am×n :
A = U ΣV T
Where U and V are orthogonal, Σ contains singular values.
14.5
Principal Component Analysis (PCA)
Dimensionality reduction technique using eigen decomposition of covariance matrix:
1. Center the data: x0i = xi − x̄
2. Compute covariance matrix: C = n1 X T X
3. Find eigenvectors (principal components) of C
4. Project data onto top k eigenvectors
27
MBZUAI Graduate Admission Preparation
Machine Learning Fundamentals
14.6
Linear Algebra in ML Applications
14.6.1
Linear Regression
Normal equation: θ = (X T X)−1 X T y
14.6.2
Neural Networks
Layer operations: a(l) = σ(W (l) a(l−1) + b(l) )
14.7
Screening Exam Practice Questions
1. For matrices A (2 × 3) and B (3 × 4), the dimensions of AB are:
(A) 2 × 4
(B) 3 × 3
(C) 4 × 2
(D) 3 × 4
2. The determinant of a singular matrix is:
(A) 1
(B) Positive
(C) Negative
(D) 0
Answer Key: 1. A, 2. D, 3. C, 4. B, 5. C, 6. C, 7. C, 8. B
15
Classification Algorithms
15.1
Introduction to Classification
Classification is a supervised learning task where the goal is to predict discrete class
labels. It’s fundamental to many AI applications.
Key Algorithms:
• Logistic Regression
• Support Vector Machines (SVM)
• Decision Trees
• Random Forests
• k-Nearest Neighbors
• Naive Bayes
28
MBZUAI Graduate Admission Preparation
15.2
Machine Learning Fundamentals
Logistic Regression
Despite its name, logistic regression is a classification algorithm that models the probability of class membership using the logistic function:
1
P (y = 1|x) =
1 + e−wT x
Cost Function: Binary cross-entropy
m
1 X (i)
J(w) = −
[y log(h(x(i) )) + (1 − y (i) ) log(1 − h(x(i) ))]
m i=1
15.3
Support Vector Machines (SVM)
Finds the optimal hyperplane that maximizes the margin between classes.
Optimization Problem:
m
X
1
ξi
min kwk2 + C
w,b 2
i=1
Subject to: y (i) (wT x(i) + b) ≥ 1 − ξi
Kernel Trick: Maps data to higher dimensions using kernel functions (linear, polynomial, RBF).
15.4
Decision Trees
Tree-based model that makes decisions based on feature values.
Splitting Criteria:
P
• Gini Impurity: G = 1 − ci=1 p2i
P Ni
• Information Gain: IG = H(parent) −
H(childi )
N
P
• Entropy: H = − ci=1 pi log2 pi
15.5
Ensemble Methods
15.5.1
Random Forests
Combines multiple decision trees using bagging (bootstrap aggregating).
15.5.2
Gradient Boosting
Builds trees sequentially, each correcting errors of previous ones.
15.6
k-Nearest Neighbors (k-NN)
Instance-based learning: classifies points based on majority vote of k-nearest neighbors.
Distance Metrics:
pP
• Euclidean: d(x, y) =
(xi − yi )2
P
• Manhattan: d(x, y) = |xi − yi |
x·y
• Cosine: d(x, y) = 1 − kxkkyk
29
MBZUAI Graduate Admission Preparation
15.7
Machine Learning Fundamentals
Naive Bayes
Probabilistic classifier based on Bayes’ theorem with feature independence assumption:
P (y|x) ∝ P (y)
n
Y
P (xi |y)
i=1
15.8
Screening Exam Practice Questions
1. Logistic regression is used for:
(A) Predicting continuous values
(B) Binary classification
(C) Clustering
(D) Dimensionality reduction
2. The kernel trick in SVM allows:
(A) Faster computation
(B) Handling non-linearly separable data
(C) Automatic feature selection
(D) Reducing model complexity
Answer Key: 1. B, 2. B, 3. B, 4. B, 5. B, 6. B, 7. C, 8. B
16
Neural Networks & Deep Learning
16.1
Introduction to Neural Networks
Neural networks are computational models inspired by biological neural networks, capable
of learning complex patterns from data.
Key Components:
• Neurons and layers
• Activation functions
• Forward and backward propagation
• Optimization algorithms
16.2
Perceptron and MLP
16.2.1
Single Perceptron
y = φ(wT x + b)
Where φ is the activation function.
30
MBZUAI Graduate Admission Preparation
16.2.2
Machine Learning Fundamentals
Multilayer Perceptron (MLP)
Stacked layers of perceptrons:
a(l) = φ(l) (W (l) a(l−1) + b(l) )
16.3
Activation Functions
Function
Sigmoid
Tanh
ReLU
Softmax
16.4
Table 11: Common Activation Functions
Formula
Range Properties
σ(x) = 1+e1−x
x
−x
tanh(x) = eex −e
+e−x
max(0, x)
x
Pe ixj
je
(0,1)
(-1,1)
[0, ∞)
(0,1)
Smooth, saturating
Zero-centered
Simple, avoids saturation
Probability distribution
Backpropagation Algorithm
The core learning algorithm for neural networks that computes gradients using chain rule:
Steps:
1. Forward pass: Compute predictions and loss
∂L
2. Backward pass: Compute gradients ∂W
(l)
∂L
3. Update weights: W (l) := W (l) − α ∂W
(l)
16.5
Convolutional Neural Networks (CNN)
Specialized for grid-like data (images, time series).
Key Layers:
• Convolutional: Learns feature detectors
• Pooling: Reduces spatial dimensions
• Fully Connected: Final classification
16.6
Recurrent Neural Networks (RNN)
Designed for sequential data with temporal dependencies.
Variants:
• LSTM: Long Short-Term Memory with gating mechanisms
• GRU: Gated Recurrent Unit (simplified LSTM)
31
MBZUAI Graduate Admission Preparation
16.7
Machine Learning Fundamentals
Regularization Techniques
• Dropout: Randomly disable neurons during training
• Batch Normalization: Normalize layer inputs
• Weight Decay: L2 regularization on weights
• Early Stopping: Stop training when validation performance degrades
16.8
Screening Exam Practice Questions
1. The backpropagation algorithm uses:
(A) Genetic algorithms
(B) Chain rule of calculus
(C) Linear programming
(D) Dynamic programming
2. ReLU activation function is defined as:
(A) 1+e1−x
(B) max(0, x)
x
−x
(C) eex −e
+e−x
(D) x2
Answer Key: 1. B, 2. B, 3. B, 4. A, 5. B, 6. B, 7. B, 8. A
17
Evaluation Metrics & Model Selection
17.1
Introduction to Model Evaluation
Proper evaluation is crucial for assessing model performance and selecting the best model
for deployment.
Key Concepts:
• Training, validation, and test sets
• Cross-validation techniques
• Performance metrics for different tasks
• Bias-variance tradeoff
32
MBZUAI Graduate Admission Preparation
Machine Learning Fundamentals
Table 12: Cross-Validation Techniques
Description
Method
k-Fold
Data divided into k folds, each fold used as validation
once
Stratified k-Fold Preserves class distribution in each fold
Leave-One-Out
Each sample used as validation once (k=n)
Time Series Split Preserves temporal order for time series data
Table 13: Confusion Matrix Components
Predicted Positive Predicted Negative
Actual Positive True Positive (TP)
Actual Negative False Positive (FP)
17.2
Cross-Validation Methods
17.3
Classification Metrics
17.3.1
Confusion Matrix
17.3.2
Key Metrics
False Negative (FN)
True Negative (TN)
N
• Accuracy: T P +TTNP +T
+F P +F N
P
• Precision: T PT+F
P
P
• Recall/Sensitivity: T PT+F
N
• F1-Score: 2 × Precision×Recall
Precision+Recall
N
• Specificity: T NT+F
P
17.4
ROC Curves and AUC
Receiver Operating Characteristic (ROC) curve plots:
• True Positive Rate (Recall) vs False Positive Rate (1 - Specificity)
• AUC (Area Under Curve): Measures overall performance (1.0 = perfect, 0.5 =
random)
17.5
Regression Metrics
P
• Mean Absolute Error (MAE): n1
|yi − ŷi |
P
• Mean Squared Error (MSE): n1 (yi − ŷi )2
√
• Root MSE (RMSE): MSE
P
2
i −ŷi )
• R-squared: 1 − P(y
(yi −ȳ)2
33
MBZUAI Graduate Admission Preparation
17.6
Machine Learning Fundamentals
Bias-Variance Tradeoff
• Bias: Error from erroneous assumptions (underfitting)
• Variance: Error from sensitivity to small fluctuations (overfitting)
• Total Error = Bias2 + Variance + Irreducible Error
17.7
Hyperparameter Tuning
• Grid Search: Exhaustive search over parameter grid
• Random Search: Random sampling of parameter space
• Bayesian Optimization: Model-based parameter selection
17.8
Loss Functions
Table 14: Common Loss Functions
Loss Function
Formula
Application
P
1
Mean Squared Error
(yi − ŷi )2
Regression
n
P
1
log(ŷ
)
+
(1
−
y
)
log(1
−
ŷ
)]
Binary classification
Cross-Entropy
− n P[y
i
i
i
Pi
1
yij log(ŷij )
Multi-class classification
Categorical Cross-Entropy − n
Hinge Loss
max(0, 1 − yi ŷi )
SVM
17.9
Screening Exam Practice Questions
1. In k-fold cross-validation, the data is divided into:
(A) Training and test sets
(B) k equal-sized folds
(C) Random subsets
(D) Feature groups
2. High precision means:
(A) Most positive predictions are correct
(B) Most actual positives are correctly predicted
(C) The model is very accurate
(D) The model never makes false positives
Answer Key: 1. B, 2. A, 3. C, 4. B, 5. B, 6. A, 7. B, 8. B
34
MBZUAI Graduate Admission Preparation
Machine Learning Fundamentals
Conclusion
These comprehensive lecture notes cover the fundamental concepts of machine learning
essential for graduate admission preparation at MBZUAI. The material spans from basic regression analysis to advanced deep learning concepts, with practice questions to
test understanding. Mastery of these topics provides a solid foundation for success in
graduate-level machine learning courses and research.
Author: Kassa Adane Tarekegn
Email: kassaadane16@gmail.com
35
0
You can add this document to your study collection(s)
Sign in Available only to authorized usersYou can add this document to your saved list
Sign in Available only to authorized users(For complaints, use another form )