NEW
EDITION
MACHINE
LEARNING
ATO Z -i
MASTERY FROM A TO Z WITH CODE
PRACTICAL EXAMPLES,
ALGORITHMS, AND APPLICATION.
VAN SHI KA
INDEX
Introduction to machine learning
1.
•
•
•
2.
Data in Machine Learning: The
backbone of insights
•
•
•
•
•
3.
In a nutshell, machine learning
Key point
features of machine learning
Type of data
Data splitting
Data preprocessing
Data advantages
Data disadvantages
Machine Learning: Pioneering
Innovations in Diverse Fields
4.
Work?
•
How Does Machine Learning
How machine learning engineers typically work
5.
Selecting a machine learning
algorithm
6.
•
•
•
7.
•
•
•
8.
•
Three sets
Training set
Validation set
Test set
Types of machine learning
Supervised learning
Unsupervised learning
Reinforcement learning
Supervised learning
Classification
•
Regression
9.
•
•
•
•
•
•
•
•
•
•
•
Classification
Introduction to Classification
Type of classification
Common classification algorithms
Type of learners in classification algorithm
Evaluating classification models
How classification works
Applications of classification algorithm
Implementation
Understanding classification in data mining
Steps in building a classification model
Importance of classification in data mining
10.
Explanation
11.
Classification & Regression
12.
Regression
•
•
•
•
•
Linear regression
Cost function for linear regression(squared error)
Gradient descent or stochastic gradient descent
Batch gradient descent
Batch SGD vs Minibatch SGD vs SGD
• Explain Briefly Batch Gradient Descent,
Stochastic Gradient Descent, and Mini-batch
Gradient Descent?
• List the Pros and Cons of Each.
• Polynomial Regression
• Overfitting and Underfitting
13.
Type of Regression
• Linear regression
14.
Cost function for Linear
regression
•
Type of Cost function in linear regression
15.
Naive Bayes
16.
Prior probabilities
17.
Evaluating your Naive Bayes
Classifiers
• Type of naive bayes classifiers
• Example
18.
•
•
Gradient Descent
Implementation
How does the gradient descent algorithm work
19.
Stochastic Gradient Descent
20.
Batch Gradient Descent
21.
Mini-Batch Gradient Descent
•
Key differences
22.
•
Polynomial Regression
How does it work
23.
Overfitting
• Why does overfitting occur?
•
How can you detect overfitting?
24.
Underfitting
• Why does underfitting occur?
• How can you detect underfitting?
25.
•
Unsupervised learning
How unsupervised learning works
26.
27.
Clustering algorithms
Anomaly detection algorithm
28.
Dimensionality Reduction
29.
What is Predictive Modeling?
30.
Why Dimensionality Reduction
important in machine learning &
Predictive Modeling?
31.
Two components of
Dimensionality Reduction
•
•
Feature selection
Feature extraction
32.
•
•
Decision tree
Gini Index
Information Gain
33.
Gini Index
34.
Information Gain
35.
Logistic regression
•
•
•
Type of logistic regression
How does logistic regression work?
Logistic regression equation
36.
Neural Networks
37.
Neural Networks: How They
Mimic The Brain
Introduction
to Machine Learning
~ It gives computers the ability to learn from data and
become more humans like in their behavior.
~ ML is based on statistical techniques and algorithms
to find patterns in data and make predictions.
~ Unlike traditional programming, ML models learn
from input data and generate their own programs.
~ Good quality data is essential for training ML models.
~ Different algorithms are used based on the type of
data and the task at hand.
In a nutshell, Machine Learning is
•
•
AI Subset: It's a part of Artificial Intelligence.
Automated Learning: Machines learn and
improve with minimum human intervention.
• Data~Driven: Models are trained on data to
find patterns and make decisions.
• No Explicit Programming: Unlike
traditional programming, ML models generate
their own programs.
Key Points
~ ML allows computers to learn from data and make
decisions like humans.
~ It's a subset of AI and relies on statistical techniques
and algorithms.
~ Input data and outputs are used to train ML models
during the learning phase.
~ ML requires good quality data for effective training.
~ Different algorithms are used depending on the data
and the task.
Remember: Machine Learning enables computers to
learn and adapt, making it an exciting and powerful
technology!
Features of machine learning
• Machine learning and data have a closely intertwined
relationship. Machine learning algorithms are designed
to learn and make predictions based on patterns
present in the data. The basic idea is that machine
learning models are trained on historical data to extract
patterns and relationships. Once trained, these models
can be used to make predictions or decisions on new,
unseen data.
• . By analyzing large volumes of data, organizations
can gain a deeper understanding of trends, customer
behaviors, market dynamics, and more..
• They can handle complex and unstructured data
types, such as images, text, and sensor data, which
may be challenging for traditional programming
approaches.
For example, in finance, machine learning models can
predict stock prices or detect fraud based on historical
transaction data. In healthcare, machine learning models
can predict disease outbreaks or assist in diagnosing
medical conditions based on patient data and medical
records.
• . By analyzing customer interactions, purchase history,
and browsing patterns, machine learning models can
create personalized experiences for customers.
Data in Machine Learning:
The
Backbone of
Insights
~ Crucial Component: Data forms the foundation of Machine
Learning, providing observations or measurements used to
train ML models.
~ Quality and Quantity: The performance of ML models
heavily relies on the quality and quantity of available data
for training and testing.
~ Various Forms: Data can be numerical, categorical, or
time~series, sourced from databases, spreadsheets, or APIs.
~ Learning Patterns: ML algorithms use data to learn
patterns and relationships between input variables and
target outputs for prediction and classification tasks.
Types of Data
1. Labeled Data: Includes a target variable for
prediction.
2. Unlabeled Data: Lacks a target variable.
3. Numeric Data: Represented by measurable values
(e.g., age,
income).
4. Categorical Data: Represents categories (e.g.,
gender, fruit type).
5. Ordinal Data: Categorical data with ordered
categories (e.g., clothing sizes, customer satisfaction scale).
Data Splitting
~ Training Data: Used to train the model on input~output
pairs.
~ Validation Data: Used to optimize model
hyperparameters during training.
~ Testing Data: Evaluates the model's performance on
unseen data after training.
Data Preprocessing
~ Cleaning and Normalizing: Preparing data for analysis
by handling missing values and scaling features.
~ Feature Selection/Engineering: Selecting relevant
features or creating new ones to improve model
performance.
Data Advantages
~ Improved Accuracy: More data allows ML models to
learn complex relationships, leading to better predictions.
~ Automation: ML automates decision~making and
repetitive tasks efficiently.
~ Personalization: ML enables personalized experiences
for users, increasing satisfaction.
~ Cost Savings: Automation reduces manual labor and
increases efficiency, leading to cost savings.
Data Disadvantages
~ Bias: Biased data can result in biased predictions and
classifications.
~ Privacy Concerns: Data collection raises privacy and
security risks.
~ Quality Impact: Poor data quality leads to inaccurate
model predictions.
~ Lack of Interpretability: Some ML models are complex
and hard to interpret, making decision understanding
difficult.
Machine Learning: Pioneering Innovations
in Diverse Fields
APPLICATIONS :
1. Image Recognition:
- Evolving from basic cat-dog classification to
sophisticated face recognition.
- Revolutionizing healthcare with disease recognition and
accurate diagnosis.
2. Speech Recognition:
- Empowering smart systems like Alexa and Siri for
seamless interactions.
- Enabling convenient voice-based Google searches and
virtual assistants.
3. Recommender Systems:
- Personalizing services based on user preferences and
search history.
- Examples: YouTube video recommendations,
personalized Netflix movie suggestions.
4. Fraud Detection:
- Efficiently identifying and preventing fraudulent
transactions and activities.
- Providing real-time notifications for suspicious user
behavior.
5. Self-Driving Cars:
- Enabling cars to navigate autonomously without human
intervention.
- Tesla cars as prominent examples of successful
autonomous driving technology.
6. Medical Diagnosis:
- Achieving high accuracy in disease classification and
diagnosis.
- Utilizing machine learning models for detecting human
and plant diseases.
7. Stock Market Trading:
- Predicting future price trends and market values through
time series forecasting.
- Assisting traders with intelligent systems for data-driven
decision-making.
8. Virtual Try On:
- Providing virtual simulation for trying on products like
glasses or accessories.
- Utilizing facial recognition to accurately place virtual
items on users' faces.
How Machine Learning
Engineers Typically Work:
1. Use of Libraries: Machine learning engineers often
leverage existing libraries and frameworks rather than
implementing algorithms from scratch.
2. Open Source Libraries: Many machine
learning libraries are open source, making them
accessible to a wide range of developers and
researchers.
3. Efficiency and Stability: Libraries are
developed and maintained to ensure stability and
efficiency in implementing complex algorithms.
4. Algorithm Selection: Engineers choose
libraries based on the specific problem they are trying
to solve. For example, they might use scikit-learn for
traditional machine learning tasks.
5. Training Models: To train a machine learning
model, engineers typically follow a systematic process:
def train(x, y):
from sklearn.linear_model import LinearRegression
model = LinearRegression().fit(x,y)
return model
model = train(x,y)
x_new = 23.0
y_new = model.predict(x_new)
print(y_new)
•
Import the necessary library or module (e.g.,
from sklearn.linear_model import
LinearRegression ).
•
Instantiate a model object (e.g., model =
LinearRegression() ).
•
Train the model using labeled data (e.g.,
model.fit(x, y) ).
•
Return the trained model (e.g., return model ).
6. Making Predictions: Once the model is
trained, it can be used to make predictions:
•
Provide new input data (e.g., x_new = 23.0 ).
•
Use the trained model to predict the output
(e.g., y_new = model.predict(x_new) ).
7. Output: Engineers often print or use the
predicted values for further analysis or decision­
making.
8. Iterative Process: Machine learning is often
an iterative process. Engineers may adjust
hyperparameters, try different algorithms, or fine-tune
the model based on performance evaluation.
9. Data Handling: Data preparation, cleaning,
and feature engineering are crucial steps before
training a machine learning model.
10. Evaluation: Engineers evaluate model
performance using various metrics and techniques to
ensure it meets the desired criteria.
11. Deployment: In real-world applications,
models are deployed to production environments, often
using platforms like cloud services or APIs.
12. Monitoring: Engineers monitor the deployed
models for performance, drift, and potential issues,
ensuring they remain effective over time.
In the example you provided, the engineer is using scikit-
learn, a popular machine learning library in Python, to train
a linear regression model. They follow a systematic
approach to load the library, create and train the model, and
make predictions. This is a common workflow for machine
learning engineers when working with established libraries
to solve real-world problems.
Selecting a machine learning
algorithm
1. Explainability:
•
Consider whether your model needs to be
explainable to a non-technical audience.
•
Some accurate algorithms, like neural networks,
can be "black boxes," making it challenging to
understand and explain their predictions.
•
Simpler algorithms such as kNN, linear
regression, or decision trees offer more
transparency in how predictions are made.
2. In-memory vs. Out-of-memory:
•
Determine if your dataset can fit into the
memory (RAM) of your server or computer.
•
If it fits in memory, you have a broader range of
algorithms to choose from.
•
If not, consider incremental learning algorithms
that can handle data in smaller chunks.
3. Number of Features and Examples:
•
Assess the number of training examples and
features in your dataset.
•
Some algorithms, like neural networks and
gradient boosting, can handle large datasets with
millions of features.
•
Others, like SVM, may perform well with more
modest capacity.
4. Categorical vs. Numerical Features:
•
Identify if your data consists of categorical
features, numerical features, or a mix.
•
Certain algorithms require numeric input,
necessitating techniques like one-hot encoding for
categorical data.
5. Nonlinearity of the Data:
•
Determine whether your data exhibits linear
separability or can be effectively modeled with
linear techniques.
•
Linear models like SVM with linear kernels,
logistic regression, or linear regression are suitable
for linear data.
•
Complex, nonlinear data may require deep
neural networks or ensemble algorithms.
6. Training Speed:
•
Consider the time allowance for training your
model.
•
Some algorithms, like neural networks, are
slower to train, while simpler ones like logistic
regression or decision trees are faster.
•
Parallel processing can significantly speed up
certain algorithms like random forests.
7. Prediction Speed:
•
Evaluate the speed requirements for generating
predictions, especially if the model will be used in
production.
•
Algorithms like SVMs, linear regression, or
logistic regression are fast for prediction.
•
Others, like kNN or deep neural networks, can
be slower.
8. Validation Set Testing:
•
If unsure about the best algorithm, it's common
to test multiple algorithms on a validation set to
assess their performance.
•
The choice of algorithm can be guided by
empirical testing and validation results.
These considerations help machine learning engineers make
informed decisions when selecting the most suitable
algorithm for a specific problem and dataset. The choice of
algorithm should align with the problem's requirements,
data characteristics, and computational constraints.
Three sets (Training set, Validation set,
and Test set)
Models learn the task
Which model
is the best?
How good
is this
model truly?
Certainly, here are the important key points about the use
of three sets (training set, validation set, and test set) in
machine learning:
1. Three Sets of Labeled Examples:
•
In practical machine learning, data analysts
typically work with three subsets of labeled
examples: training set, validation set, and test set.
•
These sets are used for different purposes in
the model development process.
2. Data Splitting:
•
After obtaining an annotated dataset, the first step is
to shuffle the examples randomly.
•
The dataset is then divided into three subsets:
training, validation, and test.
3. Set Sizes:
•
The training set is usually the largest, used for
building the machine learning model.
•
The validation and test sets are smaller and roughly
of similar size.
•
The model is not allowed to use examples from the
validation and test sets during training, which is why
they are often called "hold-out sets."
4. Proportion of Split:
There is no fixed or optimal proportion for splitting
•
the dataset into these three subsets.
In the past, a common rule of thumb was 70% for
•
training, 15% for validation, and 15% for testing.
In the age of big data, it may be reasonable to
•
allocate 95% for training and 2.5% each for validation
and testing.
5. Purpose of Validation Set:
•
The validation set serves two main purposes:
5.1.
It helps choose the appropriate learning algorithm.
5.2.
It assists in finding the best values for
hyperparameters.
6. Purpose of Test Set:
•
The test set is used to assess the model's
performance objectively.
•
It ensures that the model performs well on data
it hasn't seen during training.
7. Preventing Overfitting:
•
The use of validation and test sets helps prevent
overfitting, where a model becomes too specialized in
predicting the training data but fails on new, unseen
data.
8. Model Generalization:
•
The ultimate goal is to build a model that generalizes
well to new, unseen examples.
•
Performance on the validation and test sets provides
insights into the model's ability to generalize.
Using these three sets helps ensure that the machine
learning model is robust, performs well on unseen data, and
is suitable for practical use, rather than just memorizing
training examples.
Types of Machine Learning
1. Supervised Learning:
- Introduction: Supervised learning involves training a
model on labeled data, where the target variable is known.
- Learning Process: The model learns from input-output
pairs to make predictions on new, unseen data.
- Common Algorithms: Linear Regression, Decision Trees,
Support Vector Machines (SVM), Neural Networks.
2. Unsupervised Learning:
- Introduction: Unsupervised learning deals with unlabeled
data, where the model explores patterns and relationships
within the data on its own.
- Learning Process: The model identifies hidden structures
or clusters in the data without any explicit guidance.
Common Algorithms: K-Means Clustering, Hierarchical
Clustering,PrincipalComponentAnalysis(PCA).
Unsupervised Learning
Input
Output
3. Reinforcement Learning:
- Introduction: Reinforcement learning involves an agent
interacting with an environment to achieve a goal.
- Learning Process: The agent takes actions, receives
feedback (rewards/punishments), and adjusts its strategy to
maximize rewards.
- Applications: Game playing, Robotics, Autonomous
vehicles.
Supervised Learning
Here's an example of
supervised learning using
Python code with the scikitlearn library and a simple
linear regression algorithm:
#
Import the necessary libraries
from sklearn import linear.model
import numpy as np
#
Sample input data (features)
X = np.array([[l], [2], [3], [4], [5]])
#
Corresponding output data (labels)
y = np.array([2, 4, 5, 4, 5])
#
Create a linear regression model
model = linear_model.LinearRegression()
#
Fit the model to the data (training)
model.fit(X, y)
#
Make predictions
X.new = np.array([[6]])
prediction = model.predict(X.new)
#
Print the prediction
Introduction to Classification
- Classification is the process of categorizing data or objects
into predefined classes or categories based on their features
or attributes.
- It falls under supervised machine learning, where an
algorithm is trained on labeled data to predict the class or
category of new, unseen data.
Types of Classification
1. Binary Classification:
- Involves classifying data into two distinct classes or
categories.
- Example: Determining whether a person has a certain
disease or not.
2. Multiclass Classification:
- Involves classifying data into multiple classes or
categories.
- Example: Identifying the species of a flower based on its
characteristics.
Common Classification Algorithms
- Linear Classifiers: Create a straight decision boundary
between classes.
Examples Logistic Regression and Support Vector Machines (SVM).
- Non-linear Classifiers: Create complex decision
boundaries between classes. Examples K-Nearest Neighbors and Decision Trees.
Types of Learners in Classification Algorithms
- Slow Learners: Make predictions based on stored training
data. Examples include k-nearest neighbors.
- Fast Learners: Create models during training and use
them for predictions. Examples include decision trees and
support vector machines.
Evaluating Classification Models
- Classification Accuracy: Measures how many instances
are correctly classified out of the total instances.
- Confusion Matrix: Shows true positives, true negatives,
false positives, and false negatives for each class.
- Precision and Recall: Useful for imbalanced datasets,
measuring true positive rate and true negative rate.
- F1-Score: Harmonic mean of precision and recall for
imbalanced datasets.
- ROC Curve and AUC: Analyze classifier performance at
different thresholds.
- Cross-validation: Obtaining a reliable estimate of model
performance.
How Classification Works
1. Understanding the Problem: Define class labels and
their relationship with input data.
2. Data Preparation: Clean and preprocess data, split into
training and test sets.
3. Feature Extraction: Select relevant features from the
data.
4. Model Selection: Choose an appropriate classification
model.
5. Model Training: Train the model on the labeled training
data.
6. Model Evaluation: Assess the model's performance on
a validation set.
7. Fine-tuning the Model: Adjust model parameters for
better performance.
8. Model Deployment: Apply the trained model to make
predictions on new data.
Applications of Classification Algorithms
- Email spam filtering
- Credit risk assessment
- Medical diagnosis
- Image classification
- Sentiment analysis
- Fraud detection
- Quality control
- Recommendation systems
IMPLEMENTATION - Don’t worry the code explained
line by line .
# Importing the required libraries
import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
from sklearn import datasets
from sklearn import svm
from sklearn.tree import DecisionTreeClassifier
from sklearn.naive_bayes import GaussianNB
# import the iris dataset
iris = datasets.load_iris()
X = iris.data
y = iris.target
# splitting X and y into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.3, random_state=1)
1. # GAUSSIAN NAIVE BAYES
gnb = GaussianNB()
# train the model
gnb.fit(X_train, y_train)
# make predictions
gnb_pred = gnb.predict(X_test)
# print the accuracy
print("Accuracy of Gaussian Naive Bayes: ",
accuracy_score(y_test, gnb_pred))
2. # DECISION TREE CLASSIFIER
dt = DecisionTreeClassifier(random_state=0)
# train the model
dt.fit(X_train, y_train)
# make predictions
dt_pred = dt.predict(X_test)
# print the accuracy
print("Accuracy of Decision Tree Classifier: ",
accuracy_score(y_test, dt_pred))
3.# SUPPORT VECTOR MACHINE
svm_clf = svm.SVC(kernel = 'linear') -- # Linear Kernel
# train the model
svm_clf.fit(X_train, y_train)
# make predictions using svm
svm_clf_pred = svm_clf.predict(X_test)
# print the accuracy
print("Accuracy of Support Vector Machine: ",
accuracy_score(y_test, svm_clf_pred))
Output:
Accuracy of Gaussian Naive Bayes:
0.9333333333333333
Accuracy of Decision Tree Classifier:
Accuracy of Support Vector Machine:
0.9555555555555556
1.0
Understanding Classification in Data Mining
Definition of Data Mining: Data mining is the process of
analyzing and exploring large datasets to discover patterns
and gain insights from the data. It involves sorting and
identifying relationships in the data to solve problems and
perform data analysis.
Introduction to Classification: Classification is a data
mining task that involves assigning class labels to instances
in a dataset based on their features. The goal of
classification is to build a model that can accurately predict
the class labels of new instances based on their features.
Steps in Building a Classification Model:
1. Data Collection: Collect relevant data for the problem
at hand from various sources like surveys, questionnaires,
and databases.
2. Data Preprocessing: Handle missing values, deal with
outliers, and transform the data into a suitable format for
analysis.
3. Feature Selection: Identify the most relevant attributes
in the dataset for classification.
4. Model Selection: Choose the appropriate classification
algorithm, such as decision trees, support vector machines,
or neural networks.
5. Model Training: Use the selected algorithm to learn
patterns in the data from the training set.
6. Model Evaluation: Assess the performance of the
trained model on a validation set and a test set to ensure it
generalises well.
Importance of Classification in Data Mining:
- Classification is widely used in various domains, including
email filtering, sentiment analysis, and medical diagnosis.
- It helps in making informed decisions by categorizing data
into meaningful classes.
- By predicting outcomes, it assists in identifying risks and
opportunities.
Independent
m* Variables
Classification
Model
rataonrirai
outpX^ble
Figure 2: Classification of vegetables and groceries
#Everything explained in detail 1-14
Supervised Learning Types:
Supervised learning, a cornerstone of machine learning,
finds widespread utility across diverse domains such as
finance, healthcare, and house-price. It represents a
category of machine learning wherein algorithms are trained
using labelled data to foresee outcomes based on input
data.
In supervised learning, the algorithm forges a link between
input and output data. This relationship is discerned from
labelled datasets, containing pairs of input-output examples.
The algorithm endeavors to comprehend the connections
between input and output, equipping it to make precise
forecasts on fresh, unencountered data.
Labeled datasets in supervised learning encompass input
features coupled with corresponding output labels. Input
features encapsulate data attributes that inform predictions,
while output labels signify the desired results the algorithm
aims to predict.
Supervised learning diverges into two principal classes:
regression and classification. In regression, the algorithm
grapples with predicting continuous outcomes, be it
estimating the value of a house or gauging the temperature
of a city. In contrast, classification tackles categorical
predictions, like discerning whether a customer is likely to
embrace a product or abstain.
A cardinal benefit of supervised learning is its capacity to
craft intricate models proficient in making accurate
predictions on novel data. Nonetheless, this proficiency
demands an abundant supply of accurately labelled training
data. Additionally, the caliber and inclusiveness of the
training data wield a substantial influence on the model's
precision.
Supervised learning further fragments into two
branches:
1. Regression: Inferring Continuity
- Linear Regression:
- Polynomial Regression:
- Decision Trees for Regression:
2. Classification:
- Logistic Regression:
- Decision Trees for Classification:
- Support Vector Machines (SVM) for Classification:
■
■
Aspect
Classification
Regression
Purpose
Predicts categorical outcomes
Predicts continuous numeric
outcomes
Output
Discrete classes or labels
Continuous values
Example
Email spam detection
House price prediction
Target
Class labels (e.g., spam or not spam)
Numeric values (e.g., temperature)
Algorithms
Logistic Regression, Decision Trees,
Linear Regression, Polynomial
Support Vector Machines, Neural
Regression, Decision Trees,
Networks, etc.
Support Vector Machines, etc.
Accuracy, Precision, Recall, F1-score
Mean Squared Error, R-squared, etc.
Visualization
Confusion Matrix, ROC Curve, etc.
Scatter Plots, Residual Plots, etc.
Use Cases
Image recognition, Sentiment
Stock market prediction,
Evaluation
Metrics
Analysis
Medical diagnosis, Customer churn,
Population growth estimation,
Fraud detection, etc.
Sales forecasting, etc.
Task
Algorithm
Explanation
Example
Classification
Decision Tree
Hierarchical structure to
Predicting whether an
classify data based on
email is spam or not
features
Classification
Classification
Classification
Regression
Regression
Regression
Random Forest
Ensemble of decision trees
Identifying handwritten
Classifier
for improved classification
digits (0-9)
K - Nearest
Classifies based on majority
Identifying customer
Neighbors
class among k-nearest
segments for targeted
neighbors
marketing
Support Vector
Creates optimal decision
Detecting fraudulent credit
Machine
boundary to separate classes
card transactions
Linear
Predicts continuous values
Estimating house prices
Regression
using linear relationship
based on features
Polynomial
Fits curves to data for more
Modeling stock market
Regression
complex relationships
price trends
Decision Trees
Uses tree structure to predict
Forecasting future
for Regression
continuous values
temperature based on
weather data
Regression
Random Forest
Ensemble of decision trees
Predicting energy
Regressor
for improved regression
consumption in a building
#explanation
1. Machine Learning:
Machine learning is the process of enabling machines to
learn from data and improve their performance over time
without being explicitly programmed.
- Example: An email spam filter that learns to identify
spam messages based on patterns in the text content.
2. Supervised Learning:
Supervised learning is a type of machine learning where
the algorithm learns from labeled data, which includes input
features and corresponding output labels.
- Example: Training a model to predict housing prices
using historical data where each data point includes
features like square footage and location along with the
actual sale price.
3. Labeled Datasets:
Labeled datasets consist of input data points along with
the correct corresponding output labels, which serve as the
ground truth for training the model.
- Example: A dataset containing images of cats and dogs
along with labels indicating whether each image contains a
cat or a dog.
4. Regression:
Regression is a type of supervised learning where the goal
is to predict continuous numerical values based on input
features.
- Example: Predicting a person's annual income based on
factors such as education level, work experience, and
location.
5. Classification:
Classification is a type of supervised learning where the
goal is to categorize input data into predefined classes or
categories.
- Example: Classifying emails as either spam or not spam
based on the words and phrases contained in the email
content.
6. Linear Regression:
Linear regression is a regression algorithm that aims to
find a linear relationship between input features and the
predicted output.
- Example: Predicting the price of a used car based on its
mileage and age using a straight-line equation.
7. Logistic Regression:
Logistic regression is a classification algorithm that
predicts the probability of a binary outcome (e.g., true/false
or 0/1).
- Example: Determining whether a customer will churn
(cancel their subscription) from a service based on factors
like usage history and customer satisfaction.
8. Decision Trees:
Decision trees are hierarchical structures that make
decisions based on the values of input features, leading to
different outcomes.
- Example: Predicting whether a flight will be delayed
based on factors like departure time, airline, and weather
conditions.
9. Random Forests:
Random forests are an ensemble of multiple decision
trees that work together to make more accurate predictions.
- Example: Classifying customer preferences for a product
based on multiple factors like age, income, and purchase
history using a collection of decision trees.
10. Support Vector Machines (SVM):
SVM is a powerful algorithm used for both classification
and regression tasks that finds the optimal hyperplane to
separate data points into different classes.
- Example: Classifying whether a given email is a
phishing attempt or not based on features like sender
address, subject, and content using a support vector
machine.
11. Enabling Machines to Learn:
This refers to the process of allowing machines to
improve their performance by analyzing and understanding
patterns in data.
- Example: Teaching a self-driving car to navigate through
traffic by learning from the behavior of human drivers.
12. Input Features:
Input features are the attributes or characteristics of the
data that are used to make predictions.
- Example: In a weather prediction model, input features
could include temperature, humidity, and wind speed.
13. Output Labels:
Output labels are the desired outcomes or targets that
the algorithm aims to predict.
- Example: For a medical diagnosis model, the output
label could indicate whether a patient has a specific disease
or not.
14. Training Data and Testing Data:
Training data is used to teach the model, while testing
data is used to evaluate the model's performance.
- Example: Training a language translation model using
pairs of sentences in two languages, and then testing its
accuracy on new, unseen sentences.
(Note: The examples provided are for illustrative purposes
and may not reflect real-world accuracy or complexity.)
Classification and Regression
1.classification:
• Objective: Classification is a supervised learning task
where the goal is to assign predefined labels or
categories to input data based on its features. It's used
for tasks like spam detection, image recognition,
sentiment analysis, and more.
• output : The output of a classification model is a
discrete class label or category. It's typically
represented as a single value, such as "spam" or "not
spam," "cat," or "dog."
• Algorithms: Common classification algorithms include
logistic regression, decision trees, random forests,
support vector machines (SVM), k-nearest neighbors
(KNN), and deep neural networks.
• Evaluation : Classification models are evaluated using
metrics like accuracy, precision, recall, F1-score, and
ROC-AUC, depending on the nature of the problem and
the balance between classes.
• Loss Function : Cross-entropy is a commonly used loss
function for training classification models. It measures
the dissimilarity between predicted class probabilities
and actual labels.
• Example Application : Spam email detection, image
classification (e.g., identifying objects in images),
sentiment analysis (classifying text as positive,
negative, or neutral).
Here's an example of classification using
Python code with the scikit-learn library and a
simple decision tree classifier:
#
Import the necessary libraries
from sklearn import tree
import numpy as np
#
Sample input data (features)
X = np.array([[i, 2], [2, 3], [3, 4], [4, 5], [5, 6]])
#
Corresponding class labels
y = np.array([0, 1, 0, 1, 0])
#
# Example binary classification labels
Create a decision tree classifier model
model = tree.DecisionlreeClassifierO
#
Fit the model to the data (training)
model.fit(X, y)
#
Make predictions for new data
X.new = np.array([[3, 3]])
prediction = model.predict(X.new)
# Print the prediction
if prediction == 0:
print("The input belongs to Class 0")
else:
print("The input belongs to Class 1")
2.Regression:
• Objective : Regression is also a supervised
learning task, but its goal is to predict a
continuous numeric output or target variable. It's
used for tasks like predicting stock prices, house
prices, temperature, and more.
• Output : The output of a regression model is a
continuous value. For example, in predicting house
prices, the output might be a price in dollars.
• Algorithms : Common regression algorithms
include linear regression, polynomial regression,
Salary
decision trees, support vector regression (SVR),
and various neural network architectures.
• Evaluation : Regression models are evaluated
using metrics like mean squared error (MSE),
mean absolute error (MAE), root mean squared
error (RMSE), and R-squared (coefficient of
determination).
• Loss Function : Mean squared error (MSE) is a
widely used loss function for training regression
models. It measures the average squared
difference between predicted and actual values.
• Example Application : Predicting house prices
based on features like square footage, number of
bedrooms, and location; forecasting stock prices;
estimating temperature based on historical data.
Here's an example of regression using Python code
with the scikit-learn library and a simple linear
regressionalgorithm:______________________
# Import the necessary libraries
from sklearn import linear_model
import numpy as np
# Sample input data (features)
X = np.array([[l], [2], [3], L4], [5]])
# Corresponding output data (numeric values)
y = np.array([2, 4, 5, 4, 5])
# Create a linear regression model
model = linear_model.LinearRegression()
# Fit the model to the data (training)
model.fit(X, y)
# Make predictions for new data points
X_new = np.arrayf[[6]])
prediction = model.predict(X_new)
# Print the prediction
print("Prediction for X_new:", prediction)
Regression
• Linear Regression
• Cost Function for Linear Regression (Squared Error)
• Gradient Descent or Stochastic Gradient Descent
• Batch Gradient Descent
• Batch SGD Vs Minibatch SGD Vs SGD
• Explain Briefly Batch Gradient Descent, Stochastic
Gradient Descent, and Mini-batch Gradient Descent?
List the Pros and Cons of Each.
• Polynomial Regression
• Overfitting and Underfitting
# In supervised learning, the model is trained on a labeled
dataset, where each data point is associated with a label.
The goal of supervised learning is to learn a mapping from
input data to output labels.
There are several types of classifiers that can be used in
supervised learning. Some of the most common classifiers
include:
1. Naive Bayes classifier
2. Linear Regression
3. Neural Networks
Type of regression
Linear
Regression
?
Polynomial Regression
Sup port Vector
Regression
Types
of
Regression
Decision tree
Regression
Random Forest
Regression
Ridge Regression
7
Lasso Regression
Logistic
Regression
❖ LINEAR REGRESSION:
• Cost Function for linear regression(squared
error)
• Gradient descent or stochastic gradient
descent
• Adam algorithm (adaptive moment
estimation)
• Feature scaling
• Batch gradient descent
1.linear regression:
linear regression is a way to
understand and predict how one variable is influenced by
another by finding the best-fitting straight line through the
data points. It's a fundamental tool in statistics and data
analysis.
•
Linear regression is a statistical method used to find a
relationship between two variables: one is the
"independent variable" (often called the predictor), and
the other is the "dependent variable" (often called the
outcome).
Some key point are:
1. Basic Idea: Linear regression helps us understand
how changes in one variable are related to changes
in another. It's like trying to find a line that best fits
the data points on a graph.
2. Line of Best Fit: The goal is to find a straight line
that comes as close as possible to all the data
points. This line is the "line of best fit."
3. Equation: The equation of this line is in the form: Y
= mx + b, where:
• Y is the outcome you want to predict.
• x is the predictor variable.
• m is the slope of the line (how steep it is).
• b is the intercept (where the line hits the Y-axis).
Slope (m): The slope tells you how much Y is expected to
change for a one-unit change in x. If it's positive, as x goes
up, Y goes up; if it's negative, as x goes up, Y goes down.
4. Prediction: Linear regression can be used for prediction.
If you have a new value of x, you can plug it into the
equation to predict what Y is likely to be.
5. Errors: Linear regression accounts for errors or the
differences between the predicted values and the actual
values. The goal is to minimize these errors, making the line
a good fit for the data.
6. R-Squared: This is a number between 0 and 1 that tells
you how well the line fits the data. A higher R-squared
means a better fit.
7. Assumptions: Linear regression assumes that the
relationship between the variables is linear (forming a
straight line), and it assumes that the errors are normally
distributed.
# The equation for a simple linear regression model is:
X+E
*
Y=bo+b1
Let's break down this equation in detail:
• Y: This represents the dependent variable
• X: This represents the independent variable
• b0: This is the y-intercept, also known as the constant
term. It represents the value of Y when X is equal to 0.
In other words, it's the predicted value of Y when there
is no effect of X.
• b1: This is the slope of the line. It represents the
change in Y for a one-unit change in X.
• e (epsilon): This represents the error term or residual.
Linear Regression
linear regression represented by the equation h0(x)
= 00 + 01x1 + 02x2:
1. Hypothesis
Function (h0(x)): The
hypothesis function
represents the linear
relationship between the
input features (x1 and x2)
and the predicted output.
2.
00, 01, and 02:
These are the parameters
(coefficients) of the linear
regression model. 00 is the
intercept (bias term), 01
represents the weight for
the first feature (x1), and
02 represents the weight
for the second feature (x2).
These parameters are
learned during the training
process to minimize the
prediction error.
3. x1 and x2: These are
the input features or
independent variables. In a
linear regression model,
you have multiple features,
but in this equation, we
focus on x1 and x2.
4. Prediction: The
equation allows you to
make predictions for a
given set of input features
(x1 and x2). You plug these
features into the equation,
and the result h0(x)
represents the predicted
output.
5. Linear Relationship:
Linear regression assumes
a linear relationship
between the features and
the output. This means that
the predicted output is a
linear combination of the
features, and the model
tries to find the best linear
fit to the data.
6. Training: During the
training phase, the model
adjusts the parameters (00,
01, and 02) to minimize the
difference between the
predicted values (h0(x))
and the actual target
values in the training
dataset. This process is
typically done using a cost
function and optimization
techniques like gradient
descent.
7. Bias Term
(Intercept): 00 represents
the bias term or intercept.
It accounts for the constant
offset in the prediction,
even when the input
features are zero.
8. Gradient Descent:
Gradient descent is often
used to find the optimal
values of 90, 91, and 92 by
iteratively updating them in
the direction that reduces
the cost (prediction error).
9. Cost Function: The
cost function quantifies
how well the model's
predictions match the
actual target values. The
goal is to minimize this cost
function during training.
10. Least Squares: In
the context of linear
regression, the method of
least squares is commonly
used to find the optimal
parameters (90, 91, 92) by
minimizing the sum of
squared differences
between predicted and
actual values.
In summary, linear regression is a simple but powerful
algorithm for modeling and predicting continuous numeric
values. It relies on finding the best-fit linear relationship
between input features and the output, represented by 90,
91, and 92.
Cost Function for Linear Regression
(Squared Error)
Cost Function
In order to implement linear regression, we need to
first define the cost function.
• The cost function tells us how well the model is
doing, so we can improve it further.
•
Just a bit of context and recap before we dive into what
the cost function is:
Model: ( f ) function created by our learning
algorithm represented as f(x)=wx+b for linear
regression.
• w,b here are called parameters, or coefficients,
or weights. These terms are used interchangeably.
• Depending on what the values of w,b are our
function changes.
•
Lets break down what each of these terms mean here in the
formula above.
It takes the prediction y^(y cap) and compares it to
the target y by taking y^-y ,This difference is called
error, aka how far off our prediction is from the target.
• Next, we will compute the square of this error. We will
do this because we will want to compute this value
from different training examples in the training set.
• Finally, we want to measure the error across the entire
training set. Thus, we will sum up the squared error.
• To build a cost function that doesn’t automatically get
bigger as the training set size gets larger by
convention, we will compute the average squared error
instead of the total squared error, and we do that by
dividing by m like this.
•
The last part remaining here is that by convention, the
cost function that is used in ML divides by 2 times m .
This extra division by 2 is to make sure our later
calculations look neater, but the cost function is still
effective if this step is disregarded.
• J(w,b) is the cost function and is also called the
squared error cost function since we are taking the
squared error of these terms.
• The squared error cost function is by far the most
commonly used cost function for linear regression, and
for all regression problems at large.
• Goal: Find the parameters w or w,b that result in the
smallest possible value for the cost function J
•
* We will use batch gradient descent here; gradient descent
and its variations are used to train, not just linear
regression, but other more common models in AI.
Types of a cost function in linear
regression
In linear regression, different cost functions are used to
measure how well a model's predictions match the actual
data. Here are some common types of cost functions in
linear regression:
• Mean Error (ME): This cost function calculates the
average of all errors by simply finding the difference
between the predicted values (Y') and the actual values
(Y). Since errors can be positive or negative, they may
cancel each other out, resulting in an average error of
zero. It's not often recommended but forms the basis
for other cost functions.
• Mean Squared Error (MSE): MSE is a widely used cost
function. It measures the average of the squared
differences between predicted and actual values. By
squaring the errors, negative errors don't cancel out
positive ones. The formula for MSE is:
Where:
Yi: Actual value
Yi: Predicted value from the regression model
N: Number of data points
Salary of Employee
• Mean Absolute Error (MAE): MAE measures the
average absolute difference between predicted and
actual values. It considers all individual variances
equally and is useful for understanding the magnitude
of errors without regard to their direction. The formula
for MAE is:
Where:
Yi: Actual value
Yi: Predicted value from the regression model
N: Number of data points
• Root Mean Squared Error (RMSE): RMSE is the square
root of the mean of the squared errors. It's a measure
of the error between observed (actual) values and
predicted values. The formula for RMSE is:
RMSE =
(Si - Oi)2
Where:
Yi: Actual value
Yi: Predicted value from the regression model
N: Number of data points
In simpler terms, these cost functions help quantify how well
a regression model is performing. MSE and RMSE give more
weight to larger errors, while MAE treats all errors equally.
Choose the one that best suits your problem and the
importance of different errors in your analysis.
Here's a Python code example of how to
implement the cost function for linear
regression using squared error:
import numpy as np
# Define the actual target values (ground truth)
actual.values = np.array([2, 4, 5, 4, 5])
# Define the predicted values from your linear regression model
predicted_values = np.array([1.8, 3.9, 4.8, 4.2, 5.1])
# Calculate the squared error for each data point
squared.errors = (actual.values - predicted.values) ** 2
# Calculate the mean squared error (MSE)
mse = np.mean(squared.errors)
print("Mean Squared Error (MSE):", mse)
In this code:
actual_values represents the actual target values or
ground truth for your dataset.
• predicted_values represents the predicted values
generated by your linear regression model.
• squared_errors calculates the squared difference
between each actual and predicted value.
• mse computes the mean squared error by taking the
average of all the squared errors.
•
The mean squared error is a common metric used to assess
the performance of a linear regression model. Lower values
of MSE indicate a better fit of the model to the data,
meaning that the model's predictions are closer to the
actual target values.
Naive Bayes
Naive Bayes is a simple yet effective classification algorithm
that leverages Bayes' theorem and the assumption of
conditional independence among features. It is widely used
in text-related tasks and serves as a valuable tool for many
classification problems, especially when data is high­
dimensional and computational efficiency is crucial.
1. Probabilistic Classification: Naive Bayes is a
probabilistic classification algorithm that
assigns class labels to instances based on the
calculated probabilities. It estimates the
probability that a given instance belongs to
each class and selects the class with the
highest probability.
2. Bayes' Theorem: Naive Bayes is based on
Bayes' theorem, a fundamental concept in
probability theory. Bayes' theorem allows us to
update our beliefs about the probability of a
particular event based on new evidence or
information.
3. Independence Assumption:
• The "Naive" in Naive Bayes comes from the
assumption of independence among features or
attributes.
• It assumes that all features are conditionally
independent given the class label.
• This simplifying assumption makes calculations
tractable and efficient, though it may not hold true in
all real-world scenarios.
4. Formula:
• The Naive Bayes classification formula can be written
as:
P(y | X) = (P(X | y) * P(y)) / P(X)
Where:
• P(y | X) is the posterior probability of class y given
features X.
• P(X | y) is the likelihood of observing features X given
class y.
• P(y) is the prior probability of class y.
• P(X) is the evidence, the probability of observing
features X across all classes.
5. Classification Process:
• To classify a new data point:
Calculate the prior probabilities P(y) for each
class in the training data.
2. Estimate the likelihoods P(X | y) for each
feature in each class.
3. Apply Bayes' Theorem to compute the
posterior probabilities P(y | X) for each class.
4. Select the class with the highest posterior
probability as the predicted class.
1.
6. Conditional Independence: The "naive"
assumption in Naive Bayes is that the features used for
classification are conditionally independent, meaning
that the presence or absence of a particular feature is
assumed to be independent of the presence or absence
of other features, given the class label. This simplifies
the calculation of probabilities.
7. Feature Vector: Input data is represented as a
feature vector, where each feature corresponds to
some attribute or characteristic of the data. These
features are used to make predictions.
8. Types of Naive Bayes:
• Multinomial Naive Bayes: Commonly used for text
classification tasks, where features represent the
frequency of words or tokens in documents.
• Gaussian Naive Bayes: Suitable for continuous data
where features are assumed to follow a Gaussian
(normal) distribution.
• Bernoulli Naive Bayes: Appropriate for binary data,
where features are either present (1) or absent (0),
often used in spam detection.
9. Parameter Estimation: Naive Bayes uses statistics
from the training data to estimate probabilities. These
statistics include prior probabilities (the probability of
each class) and conditional probabilities (the
probability of each feature given the class).
10. Text Classification: Naive Bayes is particularly
popular in text classification tasks, such as spam email
detection, sentiment analysis, and document
categorization, due to its simplicity and effectiveness
with high-dimensional data.
11. Training and Prediction: Naive Bayes is
computationally efficient and has a relatively fast
training and prediction process, making it suitable for
large datasets.
12. Independence Violation: While Naive Bayes
assumes conditional independence among features,
this assumption is not always true in real-world data. In
practice, it may still perform well, but it's essential to
be aware of this limitation.
13. Applications: Besides text classification, Naive
Bayes is also used in applications like spam filtering,
document categorization, recommendation systems,
and medical diagnosis.
Prior probabilities
• Prior probabilities are like starting points in figuring
out if an email is "spam" or "not spam." They are like
initial guesses about the chances of an email being
spam before we look at the email's content. These
initial guesses are important because they affect our
final decision.
• To make this decision, we use a formula that combines
our initial guess (prior probability) with the evidence
from the email's content (class-conditional probability).
It's like giving more importance to what's in the email
while also considering what we originally thought.
• This formula helps us calculate the most likely label,
either "spam" or "not spam." The final equation for this
process can look like this:
Final Decision = Prior Probability x Class­
Conditional Probability
Or in a more commonly used way:
Final Decision=log(Prior Probability)+log(Class-Conditional
Probability)
In simple terms, it's about combining our initial hunch with
the email's content to decide if it's spam or not. This is a
basic idea behind a technique called Naive Bayesian
classification.
Evaluating your Naive Bayes
classifier
A confusion matrix is like a chart that helps you see how
well your classifier is doing. It compares the actual values
with what your classifier predicted. The rows in this chart
usually show the actual values, while the columns show the
predicted values.
Confusion matrix
Predicted classes
For example, if you were trying to predict numbers from 0 to
9 in images, you'd have a big chart with 10 rows and 10
columns. Each cell in the chart tells you how many times
your classifier got it right or got it wrong. So, if you want to
know how often your classifier mixed up the number 4 with
the number 9, you'd just look at the cell where the 4th row
and 9th column meet.
Confusion matrix example
Predictions
Types of Naive Bayes
classifiers
1. Gaussian Naive Bayes (GaussianNB): This one
is great when you have data that follows a normal,
bell-shaped curve, like continuous numbers. It
figures things out by calculating the average and
spread of each category.
2. Multinomial Naive Bayes (MultinomialNB):
When you're dealing with data that's counted and
falls into discrete categories, like word frequencies
in text (common in things like spam detection or
text classification), this type of Naive Bayes is your
friend.
3. Bernoulli Naive Bayes (BernoulliNB): If your
data is really simple, like just having two options,
such as "yes" or "no," or "1" or "0," then
BernoulliNB is handy. It's often used in problems
with binary outcomes.
Example
Certainly! Let's simplify the explanation of Bayes' formula
and its application with an example:
• Imagine we have an object, let's call it "Object X," and
we want to classify it into different classes, like "Class
1," "Class 2," and so on (up to "Class K").
• Now, we have some information about Object X, which
we'll represent as "n." This information could be
anything relevant to our classification task.
Bayes' Formula helps us calculate the probability that Object
X belongs to a specific class, let's say "Class i," given this
information "n." In simple terms, we want to know how likely
it is that Object X is in Class i based on what we know.
Here's how we do it step by step:
1. Prior Probability (P(ci)): First, we estimate how
likely each class is without considering any
information ("n"). If we don't have any specific
information, we might assume that all classes are
equally likely (so each class has a 1/K chance,
where K is the number of classes).
2. Likelihood (P(n|ci)): Next, we determine how
likely we would see the information "n" if Object X
truly belongs to Class i. This part depends on the
specific problem and data we have.
3. Evidence (P(n)): To complete the equation, we
calculate the overall probability of observing the
information "n" regardless of the class. This
involves considering the likelihood of "n" for each
class and the prior probability of each class.
Once we have these probabilities, we can use Bayes'
Formula to find the probability of Object X being in Class i
given the information "n":
P(ci|n) = P(n|ci) * P(ci) / P(n)
So, in practical terms, we're assessing how well the
information "n" matches with each class and combining it
with our prior belief about the classes to make a more
informed decision.
Keep in mind that the actual calculations will depend on the
specific problem and data, but this formula provides a
structured way to estimate the probability of class
membership based on available information.
Gradient Descent
• Objective: Gradient descent is a method used to find
the optimal values of parameters (in this case, 'w' and
'b') that minimize a cost function. It's commonly
applied in machine learning to train models like linear
regression.
• Cost Function: In linear regression, we aim to
minimize the mean squared error (MSE) between
predicted and actual values.
• Initialization: We start with initial values for 'w' and
'b,' often set to 0.
• Training Data: We use a dataset with features (in
this case, spending on radio ads) and corresponding
target values (sales units).
• Partial Derivatives: Gradient descent involves
calculating the partial derivatives of the cost function
with respect to 'w' and 'b.'
• Chain Rule: The chain rule is applied to find these
derivatives efficiently.
• Learning Rate (a): A hyperparameter that controls
the step size in each iteration. It influences the
convergence speed and stability.
• Update Rule: We iteratively update 'w' and 'b' using
the computed partial derivatives and the learning rate.
• Epochs: One pass through the entire training dataset
is called an epoch. We repeat this process for multiple
epochs until 'w' and 'b' converge to stable values.
• Convergence: We stop when 'w' and 'b' stop
changing significantly or when a predefined number of
epochs is reached
A convex function can be visualized as a gracefully
rolling landscape, featuring a solitary serene valley at
its heart, where a global minimum resides in splendid
isolation. On the flip side, a non-convex function
resembles a rugged terrain, with numerous hidden
valleys and peaks, making it unwise to apply the
gradient descent algorithm. Doing so might lead to an
entrapment at the first enticing valley encountered,
missing the possibility of reaching a more optimal,
remote low point.
In this context, gradient descent helps us find the best 'w'
and 'b' values to create a linear regression model that
predicts sales units based on radio advertising spending. It
iteratively adjusts these parameters to minimize prediction
errors, as measured by the MSE, and it continues this
process until the model is sufficiently accurate.
# Gradient Descent (GD) is an optimization algorithm used
to minimize a cost or loss function by iteratively updating
model parameters based on the gradients of the cost
function.
★ w: Represents the weight or slope of the linear
regression model.
★ b: Represents the bias or intercept of the linear
regression model.
★ J(w, b): Denotes the cost function, which measures
how well the model fits the data.
★ a (alpha): Represents the learning rate, a
hyperparameter controlling the step size in the
optimization process.
★ Objective: We aim to find the values of w and b that
minimize the cost function J(w, b). For linear regression,
a common cost function is the Mean Squared Error
(MSE):
J(w, b) = (1/2m) * Z(yi - (wxi + b))2
Where:
★ m: The number of training examples.
# Here are some important key points about Gradient
Descent along with Python code to illustrate the
concept:
• Objective: GD aims to find the values of model
parameters that minimize a cost function (often
denoted as J or L).
• Gradient Calculation: It computes the gradient of
the cost function with respect to each parameter. This
gradient represents the direction of the steepest
increase in the cost function.
• Parameter Update: GD updates the model
parameters in the opposite direction of the gradient to
reduce the cost.
Code: parameter = parameter - learning_rate *
gradient
• Iteration: GD repeats the gradient calculation and
parameter update process for a fixed number of
iterations or until convergence (when the change in the
cost function becomes very small).
Code: import numpy as np
# Generate synthetic data
np.random.seed(0)
X = 2 * np.random.rand(100, 1)
y = 4 + 3 * X + np.random.randn(100, 1)
# Initialize model parameters
theta = np.random.randn(2, 1) # Random initialization
learning_rate = 0.1
iterations = 1000
# Perform Gradient Descent
for iteration in range(iterations):
gradients = -2 * X.T.dot(y - X.dot(theta)) # Calculate
gradient
theta -= learning_rate * gradients # Update parameters
# The 'theta' values after convergence represent the
optimized model parameters
print("Optimized theta:", theta)
* * HOW DOES THE GRADIENT DESCENT ALGOTHM
WORK:
Hypothesis:
he(x) — 90 + 9xx
Parameters:
0o, #i
m
Cost Function:
2
(M3^) - J/w)
J(#o,0i) =
i=l
Goal:
minimize J(0o,0i)
00,01
Cost Function (J): A mathematical function that
quantifies how far off the model's predictions are
from the actual target values. The goal is to minimize
this function.
Model Parameters (0): These are the variables that
the algorithm adjusts to minimize the cost function.
In the context of linear regression, for example, 0
represents the weights and biases of the model.
Here's a simple Python code example of
gradient descent for a univariate linear
regression problem:
import numpy as np
# Sample data
X = np.array([l, 2, 3, 4, 5])
y = np.array([2, 4, 5, 4, 5])
# Initial guess for parameters
thetaO = 0.0
thetal = 0.0
# Learning rate
alpha = 0.01
# Number of iterations
num.iterations = 1OOO
# Gradient Descent
for _ in range(num_iterations):
# Calculate predictions
predictions = thetaO + thetal * X
#
Calculate predictions
predictions = thetaO + thetal * X
#
Calculate the gradient of the cost function
gradient.thetaO = np.mean(predictions - y)
gradient.thetal = np.mean((predictions - y) * X)
#
Update parameters using the learning rate and gradients
thetaO -= alpha * gradient.thetaO
thetal -= alpha * gradient.thetal
#
Final parameter values
print("Optimal thetaO:", thetaO)
print("Optimal thetal:", thetal)
Stochastic Gradient Descent
Stochastic Gradient Descent introduces randomness
and processes data in smaller batches, leading to
faster convergence and improved generalization.
However, it requires careful tuning of
hyperparameters and may exhibit more erratic
behavior compared to standard Gradient Descent.
1. Batch Size:
• SGD: It processes a single randomly selected
training example or a small random subset (mini­
batch) of training examples in each iteration.
• Batch Gradient Descent: It processes the entire
training dataset in each iteration.
2. Learning Rate:
• SGD: Often uses a smaller learning rate
Batch Gradient Descent
Uses the entire training dataset for gradient calculations.
Pros:
Stable convergence to the minimum.
Suitable for small to moderately sized datasets.
Cons:
Slow convergence, especially with large datasets.
Stochastic Gradient Descent (SGD):
Selects a random training instance for each iteration to
compute the gradient.
Pros:
• Faster training, especially with large datasets.
• Potential to escape local minima due to its
randomness.
Cons:
• Less stable convergence, as it oscillates around the
minimum.
• Final parameters may not be optimal due to
randomness.
• Requires tuning of the learning rate and a welldesigned training schedule.
Mini-batch Gradient Descent:
Computes gradients on randomly selected subsets (mini­
batches) of the training data.
Pros:
• Balances stability and efficiency.
• Utilizes hardware optimizations, especially with GPUs.
• Helps escape local minima better than Batch GD.
Cons:
• May still struggle to escape local minima compared to
SGD.
• The choice of mini-batch size can impact convergence
and may require tuning.
Key Differences:
• Batch GD uses the entire dataset, SGD uses single
instances, and Mini-batch GD uses small random
subsets.
• Batch GD provides stable convergence but is slow,
while SGD is fast but less stable. Mini-batch GD
balances speed and stability.
Batch GD can't handle large datasets, while SGD and Mini­
batch GD are suitable for such cases.
SGD requires tuning of learning rate and schedules for
effective convergence.
Mini-batch GD can benefit from hardware optimizations and
is widely used in deep learning.
Polynomial Regression
1. Polynomial Regression is a type of regression
analysis used in machine learning and statistics to
model the relationship between a dependent
variable (target) and one or more independent
variables (predictors) when the relationship is not
linear but follows a polynomial form. Here are the
key points to understand about Polynomial
Regression:
2. Polynomial Equation: In Polynomial Regression, the
relationship between the variables is represented
by a polynomial equation of a certain degree. The
equation takes the form:
Y=p0+p1X+p2X + p3X +... + PnX^n
Where:
•
•
•
Y: dependent variable.
X: independent variable.
p0, pl, p2, ..., pn are the coefficients of the
polynomial terms.
•
n is the degree of the polynomial, which determines
the number of terms.
3. Degree of the Polynomial: The degree of the
polynomial determines the complexity of the
model. A higher degree allows the model to fit the
data more closely but can lead to overfitting, where
the model captures noise in the data rather than
the underlying pattern. Choosing the right degree is
a crucial step in Polynomial Regression.
4. Linear vs. Non-linear Relationships: Polynomial
Regression is used when the relationship between
the variables is not linear, meaning that a straight
line (as in simple linear regression) does not
adequately capture the pattern in the data.
5. Overfitting: As mentioned earlier, high-degree
polynomials can lead to overfitting. Overfitting
occurs when the model fits the training data
extremely well but fails to generalize to new,
unseen data.
6. Model Evaluation: Just like linear regression,
Polynomial Regression models are evaluated using
metrics like Mean Squared Error (MSE), R-squared,
or others to assess how well they fit the data.
7. Visualization: Plotting the data points along with the
polynomial curve can be a useful way to
understand how well the model fits the data and
whether it captures the underlying trend accurately.
8. Applications: Polynomial Regression can be applied
in various fields, including finance, economics,
physics, biology, and engineering, where
relationships between variables are often nonlinear.
How Does it Work?
In Python, you can easily find the relationship between data
points and create a regression line without delving into
complex math. Let's break it down with a practical example:
Imagine you've recorded data from 18 cars passing a
tollbooth. You've noted the speed of each car and the time
of day (in hours) when they passed.
On a chart, the x-axis (horizontal) represents the
hours of the day.
• The y-axis (vertical) shows the speed of the cars.
•
This setup allows you to visually analyze and find patterns in
the data, like how speed changes throughout the day. With
Python, you can use specific methods to do this analysis and
draw a regression line without needing to handle the
mathematical formulas yourself.
Example
Start by drawing a scatter plot:
import matplotlib.pyplot as pit
x = [1,2,3,5,6,7,8,9,10,12,13,14,15,16,18,19,21,22]
y = [100,90,80,60,60,55,60,65,70,70,75,76,78,79,90,99,99,100]
pit.scatter(x, y)
pit.show()
Overfitting
Overfitting
A modeling error in
statistics that occurs when
a function is too closely
aligned to a limited set of
data points.
Overfitting occurs when a machine learning model
learns the training data too well, capturing noise and
irrelevant details.
• It often results in poor generalization to new, unseen
data, as the model is too tailored to the training set.
• Overfit models have excessively complex
representations, with too many parameters or features.
•
Overfitting happens when a machine learning model gets
too focused on the training data and doesn't work well with
new data. It's like a dog detector that learned to spot dogs
in parks but can't recognize them inside a house.
Why does overfitting occur?
It occurs because:
•
•
•
•
The training data is too small or not representative.
The data has too much irrelevant stuff (noise).
The model trains for too long on the same data.
The model is too complex and learns the training
data's quirks.
For instance, if a student prediction model only sees data
from one gender or ethnicity, it won't do well with others.
That's overfitting in action!
How can you detect overfitting?
Detecting overfitting is like making sure your learning model
isn't too obsessed with its training data. One way to do this
is by using a technique called K-fold cross-validation.
Here's how it works:
You take your training data and split it into K equal
parts (let's say K=5).
• Then, you train your model using 4 of these parts and
keep one part as a test.
• You see how well your model does on the test part.
• Repeat this process, each time using a different part
as the test.
• Finally, you average the results to get a good idea of
how well your model performs overall.
•
If your model does great in training but poorly in this
testing, it might be overfitting.
Underfitting
Underfitting happens when a model is too simple to
capture the underlying patterns in the data.
• It leads to poor performance on both the training and
test data.
• Underfit models may lack the capacity or complexity
needed to represent the data adequately.
•
Why does underfitting occur?
Underfitting occurs when a machine learning model is too
simple to capture the underlying patterns in the data. It
happens for several reasons:
1. Model Complexity: The model used is too basic,
lacking the capacity to understand complex
relationships in the data.
2. Insufficient Training: The model hasn't been trained
enough. It didn't have a chance to learn the data
patterns and is like trying to learn a subject in a few
minutes.
3. Inadequate Features: The features or input
variables provided to the model may not contain
enough information to make accurate predictions.
4. Over-Generalization: The model is too generalized
and doesn't adapt well to the training data. It's like
trying to fit all shoes into a one-size-fits-all mold.
How can you detect underfitting?
Detecting underfitting is essential to ensure your machine
learning model can effectively capture the patterns in the
data. Here are some common methods to detect
underfitting:
1. Training Performance: Check the model's
performance on the training data. If it's performing
poorly even on the training data, it's a sign of
underfitting. You might notice that the model
struggles to fit the training examples.
2. Validation Performance: Use a validation dataset
that the model hasn't seen during training. If the
performance on the validation data is also weak, it's
an indicator of underfitting. The model isn't
generalizing well beyond the training set.
3. Visual Inspection: Create visualizations of your
model's predictions. If you see that the predictions
are far from the actual data points, especially in a
consistent pattern, it suggests underfitting.
4. Model Complexity: Evaluate the complexity of your
model. If it's too simple and lacks the capacity to
capture the underlying patterns in the data, it may
be underfitting.
5. Learning Curves: Plot learning curves showing how
the model's performance changes with the amount
of training data. In underfitting, increasing the data
usually won't help, and the performance remains
consistently poor.
6. Feature Analysis: Review the features used for
training. If you believe that crucial information is
missing, adding relevant features might help
mitigate underfitting.
7. Model Evaluation Metrics: Use appropriate
evaluation metrics for your problem, such as
accuracy, mean squared error, or F1 score. If these
metrics show consistently poor results, it's a sign of
underfitting.
Unsupervised Learning
• Clustering Algorithm
• Anomaly Detection Algorithm
• Dimensionality Reduction
1. Definition: Unsupervised learning is a type of
machine learning where the algorithm is trained on
unlabeled data, meaning it doesn't have predefined
target outputs to learn from.
2. Clustering: where the algorithm groups similar
data points together based on patterns or
similarities in the data.
3. Dimensionality Reduction: Unsupervised
learning also involves dimensionality reduction
techniques like Principal Component Analysis (PCA)
and t-SNE, which help in visualizing and simplifying
high-dimensional data.
4. Anomaly Detection: Another important
application is anomaly detection, where the
algorithm identifies unusual or abnormal data
points that deviate from the majority of the data.
5. Exploratory Data Analysis: Unsupervised
learning is valuable for exploratory data analysis
(EDA), helping to uncover hidden patterns,
relationships, and insights within data.
6. Variety of Algorithms: Unsupervised learning
encompasses various algorithms, including K-means
clustering, hierarchical clustering, Gaussian Mixture
Models (GMM), and Autoencoders, among others.
7. Anonymization: Unsupervised learning can be
used to anonymize data by generating synthetic
data that retains statistical properties of the original
data while protecting privacy.
8. Challenges: Challenges in unsupervised learning
include selecting the appropriate number of
clusters, dealing with high-dimensional data, and
ensuring the quality of clustering results.
9. Real-world Applications: Unsupervised learning
is widely used in various fields, including image and
speech recognition, customer segmentation, fraud
detection, and recommendation systems.
10.
Semi-supervised Learning:
Sometimes, unsupervised learning is combined with
a small amount of labeled data to improve the
performance of models in semi-supervised learning
scenarios.
11.
Data Preprocessing: It often plays a
crucial role in data preprocessing, helping to
identify outliers and anomalies before further
analysis.
12.
Unstructured Data: Unsupervised
learning is also applicable to unstructured data
types, such as text and audio, where clustering and
topic modeling are common tasks.
These key points highlight the significance and versatility of
unsupervised learning in extracting valuable information
and patterns from data without the need for labeled
examples.
How unsupervised learning
works
1. Lack of Labels: Unsupervised learning starts with
data sets that do not have predefined labels or
categories. Each data point is an unlabeled input
object or sample, making it different from
supervised learning, where data is labeled.
2. Objective: The primary goal of unsupervised
learning is to discover underlying patterns or
structure within the data. It aims to group or
categorize data points based on inherent
similarities without any predefined criteria.
3. Pattern Identification: Unsupervised learning
algorithms analyze the data and extract useful
information or features. They do this by identifying
patterns and relationships between data points.
These patterns are not explicitly defined but
emerge from the data itself.
4. Clustering: One common task in unsupervised
learning is clustering. Algorithms attempt to group
similar data points together into clusters or
categories. This can be likened to organizing a
library without knowing the book titles, where
books with similar content are placed on the same
shelf.
5. Dimensionality Reduction: Unsupervised
learning can also involve dimensionality reduction,
where algorithms simplify complex data by
reducing the number of features. Principal
Component Analysis (PCA) is an example of a
technique used for dimensionality reduction.
6. Anomaly Detection: Another application is
identifying anomalies or outliers within the data.
This can be crucial for detecting fraud, errors, or
unusual behavior in various domains.
7. No Human-Defined Labels: Unsupervised
learning doesn't rely on human-defined labels or
categories. It allows for the discovery of
unexpected or subtle patterns that might not be
apparent through manual categorization.
8. Examples: If given a set of images of animals, an
unsupervised learning algorithm might group them
into categories like "fur-covered," "scales," and
"feathers" without being explicitly told to look for
these features.
9. Flexibility: Unsupervised learning is more flexible
than supervised learning. It can adapt to
unforeseen data patterns, which can be both an
advantage and a challenge. The system may
discover new categories or groupings that were not
initially anticipated.
10.
Semi-Supervised Learning: To strike a
balance between supervised and unsupervised
learning, semi-supervised learning combines
labeled and unlabeled data. This approach allows
for human guidance while still leveraging the data's
inherent structure.
Clustering algorithms
1. Definition: Clustering algorithms are unsupervised
machine learning techniques used to group similar
data points together based on certain features or
characteristics.
2. Objective: The primary goal of clustering is to
identify natural groupings or patterns in data,
making it easier to understand and analyze
complex datasets.
3. Types of Clustering:
• Hard Clustering: Assigns each data point to a single
cluster exclusively.
• Soft Clustering: Allows data points to belong to
multiple clusters with associated probabilities or
membership scores.
4. Common Clustering Algorithms:
• K-Means: Divides data into K clusters by minimizing
the sum of squared distances from data points to
cluster centroids.
• Hierarchical Clustering: Creates a hierarchical tree­
like structure of clusters, which can be visualized as a
dendrogram.
• DBSCAN: Density-based algorithm that forms clusters
based on data point density, suitable for irregularly
shaped clusters.
• Agglomerative Clustering: Hierarchical clustering
approach that starts with individual data points as
clusters and iteratively merges them.
• Gaussian Mixture Models (GMM): Represents
clusters as Gaussian distributions, accommodating
complex cluster shapes.
• Spectral Clustering: Utilizes the eigenvectors of a
similarity matrix to group data points into clusters.
• Mean-Shift: Finds mode locations in the data
distribution, serving as cluster centers.
5. Distance Metrics: Clustering often relies on
distance measures like Euclidean distance,
Manhattan distance, or cosine similarity to
determine the similarity or dissimilarity between
data points.
6. Number of Clusters: Determining the optimal
number of clusters, K, is a critical challenge in
clustering, and various methods like the elbow
method or silhouette score are used for this
purpose.
7. Initialization: Many clustering algorithms, such as
K-Means, require careful initialization of cluster
centroids, which can impact the final results.
8. Scalability: The scalability of clustering algorithms
is an important consideration, as some methods
may not be suitable for large datasets.
9. Applications: Clustering algorithms find
applications in various domains, including customer
segmentation, image segmentation, anomaly
detection, and natural language processing.
from sklearn.cluster import KMeans
I
I
J
import numpy as np
# Generate some example data
data = np.array([[1, 2], [5, 8], [1.5, 1.8], [8, 8], [1, 0.6], [9, 11
# Create a K-Means clustering model with 2 clusters
kmeans - KMeans(n_clusters=2)
# Fit the model to the data
kmeans.fit(data)
# Get cluster labels for each data point
labels - kmeans.labels.
# Get cluster centers
centers = kmeans.cluster.centers.
print("Cluster Labels:", labels)
print!"Cluster Centers:", centers)
# K-means clustering is like sorting marbles into groups.
Imagine you have a bunch of marbles, and you want to
group them into clusters based on their colors. K-means
helps by creating clusters so that marbles with similar colors
end up in the same group. It works like this:
1. You decide how many groups (clusters) you want,
let's say K clusters.
2. Then, you start with K random marbles to represent
the centers of those clusters.
3. Now, you assign each marble to the cluster whose
center (mean) is closest to it. This is like saying
each marble joins the group with the most similar
colors.
4. After that, you recalculate the center (mean) of
each cluster based on the marbles inside it. So, the
center marble becomes the average color of all the
marbles in its group.
5. You repeat steps 3 and 4 until the clusters don't
change much, and each marble is in the group with
the closest mean color.
In the end, you've sorted your marbles into K groups based
on their colors, and that's what K-means clustering does for
data points - it groups them based on their similarities.
Anomaly detection algorithms
1. Definition: Anomaly detection algorithms are
machine learning or statistical techniques designed
to identify unusual or rare data points in a dataset
that deviate significantly from the expected
behavior.
2. Unsupervised Learning: Anomaly detection deals
with unlabeled data, where the algorithm learns to
distinguish anomalies from normal data patterns.
3. Types of Anomalies:
• Point Anomalies: Isolation of individual data points as
anomalies.
• Contextual Anomalies: Anomalies that depend on
context or other data points.
• Collective Anomalies: Groups of data points together
form an anomaly.
4. Common Anomaly Detection Techniques:
• Statistical Methods: Employ statistical measures such
as Z-scores, percentiles, or histograms to identify
anomalies based on data distribution.
• Machine Learning Algorithms: Algorithms like Isolation
Forest, One-Class SVM, and Autoencoders are used for
anomaly detection.
• Density-Based Approaches: Methods like DBSCAN and
LOF (Local Outlier Factor) identify anomalies by
analyzing data point density.
• Time Series Analysis: Techniques like ARIMA and
Prophet are applied to detect anomalies in time-series
data.
5. Threshold Setting: Setting an appropriate anomaly
detection threshold is a critical step, as it determines the
sensitivity and specificity of the algorithm.
6. Scalability: Some algorithms may be more scalable and
suitable for large datasets, which is an important
consideration in real-world applications.
7. Evaluation Metrics: Metrics like precision, recall, F1score, and area under the receiver operating characteristic
(ROC-AUC) curve are used to evaluate the performance of
anomaly detection algorithms.
8. Applications: Anomaly detection is used in various
domains, including fraud detection in finance, network
intrusion detection in cybersecurity, equipment failure
prediction in manufacturing, and disease outbreak detection
in healthcare.
from sklearn.decomposition import PCA
import numpy as np
# Generate some example data
data = np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]])
# Create a PCA model with 2 components
pea = PCA(n_components=2)
# Fit the model to the data and transform it
transformed_data = pca.fit.transform(data)
print("Original Data:")
print(data)
print("Transformed Data (2 Components):")
print(transformed.data)
Dimensionality Reduction
1. Definition: Dimensionality reduction is a technique
used in machine learning and data analysis to
reduce the number of features or variables in a
dataset while retaining its essential information.
2. Feature Selection vs. Feature Extraction:
• Feature Selection: Involves choosing a subset of the
most relevant features while discarding others based
on their importance to the task.
• Feature Extraction: Creates new, lower-dimensional
features by combining or transforming the original
features.
3. Principal Component Analysis (PCA):
• One of the most common dimensionality reduction
techniques.
• Identifies orthogonal linear combinations of features
(principal components) that capture the most variance
in the data.
• Useful for data compression and visualization.
4. t-Distributed Stochastic Neighbor Embedding
(t-SNE):
• Non-linear dimensionality reduction technique.
• Focuses on preserving the pairwise similarities
between data points in a lower-dimensional space.
• Valuable for visualizing high-dimensional data.
5. Autoencoders:
• Neural network-based approach for unsupervised
feature learning.
• Consists of an encoder and a decoder that learn a
compact representation of the data.
• Widely used for tasks like image denoising and
anomaly detection.
6. Reduction Techniques for Interpretability:
• Some dimensionality reduction methods aim to create
interpretable features, making it easier to understand
the underlying structure of the data.
7. Loss of Information: Dimensionality reduction
inevitably involves some loss of information, so it's
crucial to strike a balance between reducing
dimensionality and preserving critical data
characteristics.
8. Data Preprocessing: Dimensionality reduction
can be used as a preprocessing step to improve the
performance of machine learning algorithms by
reducing noise or multicollinearity.
9. Curse of Linearity: Linear dimensionality
reduction techniques like PCA may not capture
complex, non-linear relationships in the data. Non­
linear methods like manifold learning can address
this limitation.
10.
Applications: Dimensionality reduction
is applied in various domains, including image and
speech recognition, natural language processing,
genomics, and recommendation systems.
code
from sklearn.ensemble import IsolationForest
import numpy as np
# Generate some example data
data = np.random.randn(100, 2)
# Introduce some anomalies
data[95:100] = np.random.uniform(5, 10, (5, 2))
# Create an Isolation Forest model
iso_forest = IsolationForest(contamination=0.05)
# Fit the model to the data and predict anomalies
iso_forest.fit(data)
anomaly_scores = iso_forest.decision_function(data)
print("Anomaly Scores:")
print(anomaly_scores)
What is Predictive Modeling?
Predictive Modeling: Predictive modeling is like fortune­
telling for data. It's a way to make educated guesses about
future outcomes based on certain clues (predictors). These
predictors are like pieces of a puzzle that help us figure out
what the final picture (outcome) will look like. It's all about
using data to make informed predictions.
Dimensionality Reduction: Think of dimensionality
reduction as simplifying a complex problem. It's like turning
a 3D movie into a 2D one, keeping the most important
scenes. We do this to make things easier to handle, like
simplifying a recipe. Techniques like PCA and SVD help us
focus on the main ingredients of our data while removing
the extras, so we can understand it better or make our
models run faster.
Why is Dimensionality Reduction
important in Machine Learning and
Predictive Modeling?
Simplifying with Dimensionality Reduction: Imagine
you're dealing with a pile of emails, trying to figure out if
they're spam or not. You have lots of clues to consider, like
the email's title, content, and more. But some of these clues
are kind of similar, like two people telling you the same
thing in different ways.
Point 1: So, in dimensionality reduction, we're like detectives
trying to spot these similarities and group them together.
We might find that two clues are saying pretty much the
same thing, and we don't need both.
Point 2: For example, if we're dealing with a weather
forecast, we might have data on humidity and rainfall. But
guess what? They usually go hand in hand. So, we can
simplify things by just looking at one of them because
they're strongly connected. It's like saying, "I'll just focus on
the rain, and it'll give me a good idea of the humidity too."
Point 3: Think of it like turning a 3-D puzzle into a 2-D one,
or even simpler, into a straight line. It's way easier to wrap
your head around a flat puzzle than one that sticks out in all
directions. In data, this makes our job easier and our models
faster. So, dimensionality reduction helps us cut through the
complexity and get to the heart of the matter.
There are two components of dimensionality
reduction:
1. Feature Selection: Imagine you have a lot of
ingredients to make a meal, but you don't need all
of them. So, you pick out only the essential ones.
Feature selection is like that; it's about choosing
just the right ingredients (features) to cook up your
problem-solving recipe. There are three ways to do
this:
•
Filter: It's like using a filter to sift out the best
ingredients based on some criteria. You keep the ones
that make the dish taste the best.
•
Wrapper: Imagine you're trying different
combinations of ingredients to see which one makes
the most delicious dish. This method is like trial and
error to find the perfect mix.
•
Embedded: Here, you build a special recipe that
automatically selects the right ingredients as it cooks.
It's like a chef that knows exactly what works while
preparing the dish.
2. Feature Extraction: Picture you have a big,
cluttered kitchen, and you want to tidy it up. You
decide to put away some kitchen gadgets you
rarely use. Feature extraction is like tidying up your
kitchen by storing away things you don't need daily.
You keep only the most important tools for cooking.
•
Methods of Dimensionality Reduction: When you
want to simplify things, you use methods like these:
•
Principal Component Analysis (PCA): Think of
this as a magical way to take a messy kitchen and
neatly organize your most-used cooking tools into a
compact drawer. It helps you focus on what's essential.
•
Linear Discriminant Analysis (LDA): This is like
having a chef who knows how to arrange the kitchen
tools so that they work together perfectly for a specific
recipe. It optimizes your kitchen for a particular type of
cooking.
•
Generalized Discriminant Analysis (GDA): Think
of this as a super chef who not only organizes the
kitchen but also fine-tunes the tools to create the
tastiest meals. It goes beyond LDA and makes
everything work harmoniously for multiple dishes.
Decision tree learning
Decision tree learning is a machine learning technique used
for both classification and regression tasks. It is a powerful
and interpretable model that makes decisions by recursively
partitioning the input space into regions or intervals based
on the values of input features. Let's highlighting important
key points:
1. Tree Structure:
• A decision tree is a tree-like structure comprising
nodes and edges.
• Nodes represent decision points or tests on specific
features.
• Edges connect nodes and represent the outcome of a
feature test.
• Leaves (terminal nodes) contain the final prediction or
value.
2. Root Node:
• The top node of the tree is called the root node.
• It represents the entire dataset or the starting point of
the decision-making process.
3. Internal Nodes:
• Nodes other than the root and leaves are internal
nodes.
• Internal nodes perform feature tests and decide how
to partition the data.
4. Leaves:
• Leaves are the endpoints of the decision tree.
• They provide the final prediction or output for a given
input.
5. Feature Tests:
• At each internal node, a feature from the dataset is
chosen for testing.
• The feature is tested against a certain threshold or
condition.
• The outcome of the test determines the path to follow
down the tree.
6. Splitting Criteria:
• Decision trees use various criteria to determine the
best feature and condition for splitting the data at each
internal node.
• Common criteria include Gini impurity (for
classification) and mean squared error (for regression).
7. Recursive Partitioning:
• Decision trees recursively split the data into subsets
based on feature tests.
• This process continues until a stopping condition is
met, such as a maximum depth, a minimum number of
samples per leaf, or a purity threshold.
8. Pruning:
• Pruning is a technique used to prevent overfitting by
simplifying the tree.
• It involves removing branches that do not contribute
significantly to improving the model's generalization.
9. Interpretability:
• Decision trees are highly interpretable models.
• It's easy to understand and visualize the decisions
made at each node, making them valuable for
explaining the model's predictions.
10.
Ensemble Methods:
• Decision trees can be combined into ensemble
methods like Random Forests and Gradient Boosting,
which often result in more robust and accurate models.
11.
Handling Categorical Data:
• Decision trees can handle both categorical and
numerical features.
• For categorical features, they can perform multi-way
splits based on categories.
12.
Feature Importance:
• Decision trees can provide information about feature
importance, which helps in feature selection and
understanding the most influential variables in the
model.
13. Bias-Variance Trade-off:
• Decision trees tend to have a high variance, which can
lead to overfitting.
• Tuning hyperparameters and employing ensemble
methods can mitigate this issue.
In summary, decision tree learning is a versatile and
interpretable machine learning technique that builds a tree
structure to make decisions based on input features. Its
simplicity and transparency make it a valuable tool for
various applications, from classification to regression and
beyond. However, careful parameter tuning and
consideration of overfitting are essential to maximize its
effectiveness
# Decision tree algorithm is a type of supervised learning
that can be used to solve both regression and classification
problems. It uses the tree representation to solve the
problem in which each leaf node corresponds to a class
label and attributes are represented on the internal node of
the tree . Decision trees are used for both categorical and
continuous data, but continuous data is discretized prior to
building the model . The records are distributed recursively
on the basis of attribute values . Statistical methods are
used for ordering attributes as root or the internal node
The decision tree works on the Sum of Product form, also
known as Disjunctive Normal Form . The major challenge in
decision tree is identifying the attribute for the root node at
each level, which is known as attribute selection . There are
two popular attribute selection methods:
1. Gini Index
2. Information Gain
Gini Index
The Gini Index is a splitting measure used in decision
trees to determine the degree of impurity of a
particular variable when it is randomly chosen . The
degree of Gini Index varies between 0 and 1, where 0
denotes that all elements belong to a certain class or
there exists only one class (pure), and 1 denotes that
the elements are randomly distributed across various
classes (impure) . A Gini Index of 0.5 denotes equally
distributed elements into some classes
To calculate the Gini Index, we use the following
formula:
Ginilndex — 1 —
i = 1
where
• n is the number of classes, and
• pi is the probability of an element belonging to
class i
Here’s an example of how to calculate the Gini Index:
Suppose we have a dataset with 10 elements, where 6
belong to class A and 4 belong to class B. The
probability of an element belonging to class A is 6/10
,and the probability of an element belonging to class B
is 4/10. Therefore, the Gini Index can be calculated as
follows:
ANSWER:
67
4 7
Ginilndex = 1 — (—) — (—) = 0.48
v107
v107
Information Gain
It's a measure of the reduction in uncertainty (or entropy)
achieved by partitioning a dataset based on a particular
attribute or feature. Here's a detailed explanation along with
some key points:
1. Entropy: Entropy is a measure of impurity or disorder in
a dataset. In the context of decision trees, it quantifies the
randomness or unpredictability of the class labels in a
dataset. High entropy means the data is highly disordered,
and low entropy means it's very well-structured.
The formula for calculating entropy in the context of
information theory is as follows:
Entropy (H(X)) = -1 [p(x) * log2(p(x))]
Where:
• H(X) represents the entropy of a random variable X.
• 1 denotes the summation over all possible values of X.
• p(x) is the probability of the random variable X taking
on a particular value x.
• log2 represents the base-2 logarithm.
This formula quantifies the uncertainty or randomness
associated with the random variable X. The higher the
entropy, the more uncertain and random the variable is,
while lower entropy indicates a more predictable and less
uncertain variable.
2. Information Gain (IG): Information Gain is a metric that
measures how much the knowledge of a particular attribute
or feature reduces the uncertainty in predicting the class
labels of data points. In simple terms, it quantifies how
much knowing a feature helps us make more accurate
predictions.
3. Calculation: Information Gain is typically calculated as
the difference between the entropy of the original dataset
(before splitting) and the weighted average of entropies of
the sub-datasets (after splitting), using a specific attribute.
Mathematically, it's represented as:
IG(Attribute) = Entropy(Original Dataset) - 1 [(|Sv| / |S|) *
Entropy(Sv)], for all values of the attribute.
Where:
• IG(Attribute) is the Information Gain for the attribute.
• |Sv| is the number of data points in the sub-dataset
created by using a specific value of the attribute.
• |S| is the total number of data points in the original
dataset.
• Entropy(Sv) is the entropy of the sub-dataset after
splitting.
4. Attribute Selection: In a decision tree, Information Gain
is used to select the best attribute for splitting the dataset
at each node. The attribute with the highest Information
Gain is chosen as the splitting attribute, as it provides the
most valuable information for classifying the data.
5. Key Points:
a. High Information Gain: An attribute with high
Information Gain is considered more informative, as it
reduces uncertainty in classifying data points.
b. Low Information Gain: An attribute with low
Information Gain doesn't help much in classification and
might not be a good choice for splitting.
c. Decision Trees: Information Gain is a crucial criterion in
decision tree algorithms like ID3, C4.5, and CART for
selecting the best attribute to split the data.
d. Feature Selection: Information Gain can also be used
for feature selection in feature engineering. Features with
higher Information Gain are more likely to be relevant for
prediction tasks.
e. Limitations: Information Gain tends to favor attributes
with many distinct values, so it may not work well for
continuous attributes. In such cases, other criteria like Gain
Ratio or Gini Impurity are used.
In summary, Information Gain is an important concept in
machine learning, particularly in decision tree-based
algorithms, as it helps to choose the most informative
features for decision making and classification, ultimately
leading to more accurate and efficient models.
Logistic Regression
Logistic Regression is a statistical method used for binary
classification, which means it's employed to predict the
probability that an input belongs to one of two possible
classes or categories.
1.Binary Classification: Logistic Regression is
primarily used for binary classification problems. In this
context, you have two possible outcomes or classes,
often denoted as 0 and 1, where 0 represents the
negative class (e.g., "not spam") and 1 represents the
positive class (e.g., "spam").
2.Linear Combination: Logistic Regression starts with
a linear combination of the independent variables.
Each independent variable is assigned a weight, and
these weights are multiplied by the corresponding
values of the independent variables. This is
represented as:
Z = &0 +
* #1 + &2 * X2
+ ... + bn * Xn
where:
• z is the linear combination of weights and variables.
• b0 is the intercept or bias term.
• b1, b2, ..., bn are the coefficients (weights) associated
with the independent variables x1, x2, ..., xn.
3.Logistic Function (Sigmoid Function): The linear
combination z is passed through a logistic function,
also known as the sigmoid function. The sigmoid
function transforms the linear combination into a value
between 0 and 1, which can be interpreted as a
probability. The sigmoid function is defined as follows:
•
P(Y=1) represents the probability of the dependent
variable being in class 1
• e is the base of the natural logarithm.
• This function ensures that the output probability is
bounded between 0 and 1.
4 .Decision Boundary: Logistic Regression uses a
decision boundary to classify the input data. By default,
the decision boundary is set at 0.5. If the predicted
probability is greater than or equal to 0.5, the instance
is classified as class 1; otherwise, it's classified as class
0.
5 .Training: The model's parameters (the coefficients
$b_0, b_1, b_2, ..., b_n$) are learned from a labeled
training dataset. The model aims to find the best set of
parameters that minimize the difference between the
predicted probabilities and the actual class labels in the
training data.
6 .Evaluation: Once trained, the Logistic Regression
model can be evaluated on a separate validation or
test dataset to assess its performance using metrics
like accuracy, precision, recall, F1-score, and the ROC
curve.
7 .Assumptions: Logistic Regression makes certain
assumptions about the data, including:
Linearity: The relationship between the independent
variables and the log-odds of the dependent variable is
assumed to be linear.
• Independence of Errors: Errors in the prediction should
be independent of each other.
• No Multicollinearity: Independent variables should not
be highly correlated with each other.
•
8.Regularization: To prevent overfitting,
regularization techniques like L1 (Lasso) and L2 (Ridge)
regularization can be applied to the Logistic Regression
model.
9.Applications: Logistic Regression is widely used in
various fields, including medicine (disease prediction),
marketing (customer churn prediction), finance (credit
scoring), and natural language processing (sentiment
analysis).
Type of Logistic Regression
On the basis of the categories, Logistic Regression can be
classified into three types:
•
Binomial: In binomial Logistic regression, there
can be only two possible types of the dependent
variables, such as 0 or 1, Pass or Fail, etc.
•
Multinomial: In multinomial Logistic regression,
there can be 3 or more possible unordered types
of the dependent variable, such as “cat”, “dogs”,
or “sheep”
•
Ordinal: In ordinal Logistic regression, there can be 3
or more possible ordered types of dependent variables,
such as “low”, “Medium”, or “High”.
code
In Python, you can use libraries like scikit-learn to perform
Logistic Regression. Below is an example of implementing a
binomial Logistic Regression using scikit-learn:
1. Binomial
Import the necessary libraries
#
from sklearn.linear.model import LogisticRegression
from sklearn.model.selection import train_test_split
from sklearn.metrics import accuracy.score
Sample data
#
X = [[2.5], [3.5], [5.5], [6.7], [8.9], [10.1]]
y = [0, 0, 1, 1, 1, 1]
#0 represents "Fail", and 1 represents "Pass"
Split the data into training and testing sets
#
X.train, X.test, y.train, y.test = train_test_split(X, y, test_size=0.2, ran
Create a Logistic Regression model
#
model = LogisticRegression()
Fit the model to the training data
#
model.fit(X_train, y.train)___________________________________________
#
Fit the model to the training data
model.fit(X.train, y.train)
#
Make predictions on the test data
y_pred = model.predict(X.test)
#
Calculate the accuracy of the model
accuracy = accuracy_score(y_test, y.pred)
print(f"Accuracy: {accuracy * 100:.2f}%")
2. Multinomial
from sklearn.linear.model import LogisticRegression as M
from sklearn.model.selection import train.test.split as T
from sklearn.metrics import accuracy.score as A
X = [[2.5, 1], [3.5, 2], [5.5, 2.5], [6.7, 2.8], [8.9, 3.2], [10.1, 3.5]]
y = [0, 1, 2, 1, 2, 0]
x,y,p,q=T(X,y,t,.2)
m=M(multinomial=1 multinomial1,solver='Ibfgs').fit(x,p)
print(f"Accuracy: {A(q,m.predict(y)) * 100:.2f}%")
3.Ordinal
from mord import LogisticIT
from sklearn.model.selection import train.test.split
from sklearn.metrics import accuracy.score
X = [[2.5], [3.5], [5.5], [6.7], [8.9], [10.1]]
y = [0, 1, 2, 1, 2, 0]
x,y,p,q = train_test.split(X, y, test_size=0.2, random.state=42)
m = LogisticIT().fit(x, y)
print(f"Accuracy: {accuracy.score(q, m.predict(p)) * 100:.2f}%")
How does Logistic Regression work?
Logistic Regression works like a detective trying to solve a
mystery. It's used when we want to figure out the chances of
something happening or not happening.
Here's how it works in simple terms:
1 .Collect Data: First, we gather information about what
we're studying. For example, if we're looking at whether a
student will pass an exam, we collect data like study hours,
previous grades, and so on.
2 .Calculate Odds: Imagine we're detectives and we want
to know the odds of a suspect being guilty. Logistic
Regression calculates these odds. In our student example, it
calculates the odds of passing the exam based on the
collected data.
3 .Logistic Function: We use a special formula called the
logistic function. It takes the odds and squishes them into a
range between 0 and 1. This range helps us say, "very
unlikely" (close to 0) or "very likely" (close to 1).
4 .Threshold: We set a threshold, like 0.5. If the result from
the logistic function is greater than 0.5, we say it's likely to
happen. If it's less than 0.5, we say it's unlikely.
So, Logistic Regression helps us make predictions based on
data by calculating the odds of something happening and
then deciding if it's likely or not. It's a bit like a detective
using evidence to solve a case, but with numbers!
Logistic Regression Equation
• Logistic Regression helps us predict the chances of
something happening (like passing an exam) using a
special formula. It calculates the odds and turns them
into probabilities between 0 and 1.
• The formula looks like this: p(X) = 1 / (1 + *
X
e^-(w
+
b)).
Likelihood Function for Logistic Regression:
• To find the best predictions, we use a likelihood
function. It measures how well our predictions match
the actual outcomes.
• It's like a score that tells us how good our predictions
are:
L(b, w) = n(yi * p(xi) + (1-yi) * (1-p(xi)) for all
data points.
Gradient of the Log-Likelihood Function:
• To improve our predictions, we need to adjust the
formula. This involves finding the maximum likelihood
estimates. We do this by finding the gradient of the
likelihood function.
• The gradient helps us figure out how to change the
formula to make better predictions. It's like fine-tuning
a recipe to make it taste just right.
Assumptions for Logistic Regression:
• When using Logistic Regression, we make a few
assumptions:
Each data point is independent; they don't affect each other.
• The thing we're predicting has only two possible
outcomes, like yes or no.
• The relationship between the data we collect and the
prediction is a straight line.
• There are no weird, extreme data points that could
mess up our predictions.
• We have enough data to work with, so our predictions
are reliable.
Neural Networks
1. The original motivation behind the development of
neural networks was to create software that could
emulate the functioning of the human brain.
2. To gain a deeper understanding of how neural
networks work, let's consider an example: demand
prediction for a shirt. In this context, we'll examine
the representation of data using a sigmoid function,
which is commonly applied in logistic regression.
3. We'll introduce the concept of an "activation
function," denoted as "a," which plays a crucial role
in neural networks. This activation function takes an
input "x," which, in our example, represents the
price of the shirt. The activation function processes
this input using a specific formula and produces an
output that represents the probability that the shirt
will become a top seller.
# Key points to remember
• Neural networks were inspired by the human brain
and were designed to mimic its functioning in software.
• Neural networks have had a profound impact on
various application areas, with early successes in fields
like speech recognition.
• In the context of demand prediction for a shirt, a
sigmoid function is used to represent the data, typically
associated with logistic regression.
Neural Networks: How They Mimic
the Brain
Neural networks, a fundamental concept in artificial
intelligence and machine learning, indeed draw inspiration
from the structure and functioning of the human brain. The
connection between artificial neural networks and the brain
lies in the following simplified manner:
l.Neurons: In artificial neural networks, artificial neurons,
often referred to as nodes or units, are the building blocks.
Just like biological neurons, these artificial neurons process
and transmit information.
2. Connections: Neurons in the brain are connected to
each other through dendrites and axons, allowing them to
pass signals. In artificial neural networks, connections are
represented as weighted links between artificial neurons.
These weights determine the strength of the connection,
much like the efficiency of synapses in the brain.
3. Information Processing: Just as biological neurons
receive and transmit information through electrical
impulses, artificial neurons perform calculations on the data
they receive from their connected neurons. This process
involves a weighted sum of inputs, an activation function,
and an output.
4. Learning: The brain adapts and learns from experience
through a process called synaptic plasticity. Artificial neural
networks learn by adjusting the weights of connections,
known as training. During training, these networks fine-tune
their weights to improve their performance on specific
tasks.
5. Functionality: The human brain carries out a multitude
of functions, from sensory perception to motor control and
memory storage. Similarly, artificial neural networks can be
designed for various tasks, such as image recognition,
natural language processing, and more, by configuring their
architecture and training them on specific data.
While artificial neural networks mimic certain aspects of the
brain's structure and function, it's important to note that
they are highly simplified models. The brain is vastly more
complex, with intricate biological mechanisms and
processes that are far from fully understood. Artificial neural
networks are a powerful tool in machine learning, but they
are not a perfect replication of the human brain; rather, they
are inspired by its basic principles.
Neural Network Layer
A neural network layer is a fundamental building block of a
neural network, a type of machine learning model inspired
by the human brain. Neural network layers are responsible
for processing and transforming data as it flows through the
network. Let's delve deeper into neural network layers and
highlight important key points:
1. Input Layer:
• The input layer is the first layer of a neural network.
• Its nodes (neurons) represent the features or input
variables of the problem.
• Data is fed into the input layer, and each node
typically corresponds to a specific input feature.
2. Hidden Layers:
• Hidden layers are the intermediate layers between the
input and output layers.
• They perform complex transformations on the input
data.
• The number of hidden layers and the number of
neurons in each layer are design choices and can vary
depending on the problem.
3. Neurons or Nodes:
• Neurons in a layer process information. Each neuron
receives inputs, performs a computation, and produces
an output.
• Neurons in the hidden layers use activation functions
to introduce non-linearity into the network, enabling it
to learn complex patterns.
4. Weights and Bias:
• Each connection between neurons has an associated
weight that determines the strength of the connection.
5. Activation Functions:
• Activation functions introduce non-linearity to the
network, enabling it to learn complex functions.
• Common activation functions include ReLU (Rectified
Linear Unit), sigmoid, and tanh.
• The choice of activation function can impact how well
the network learns and converges.
6. Output Layer:
• The output layer produces the final result or prediction
of the neural network.
• The number of neurons in the output layer depends on
the type of problem. For example, in binary
classification, there may be one neuron, while in multi­
class classification, there could be multiple neurons.
• The activation function in the output layer depends on
the nature of the problem (e.g., sigmoid for binary
classification, softmax for multi-class classification).
7. Feedforward Process:
• During training and inference, data flows through the
neural network in a forward direction, from the input
layer to the output layer.
• Each layer computes its output based on the input,
weights, bias, and activation function.
8. Backpropagation:
• Backpropagation is the process of updating the
weights of the neural network to minimize the
difference between the predicted output and the actual
target (training data).
• It uses techniques like gradient descent to adjust
weights and biases.
9. Deep Learning:
• Deep neural networks consist of multiple hidden
layers, enabling them to model highly complex
relationships in data.
• Deep learning has shown remarkable success in tasks
such as image recognition, natural language
processing, and reinforcement learning.
In summary, neural network layers are crucial components
that process and transform data, making them capable of
learning complex patterns and solving a wide range of
machine learning problems. Understanding the architecture
and functioning of these layers is essential for effectively
designing and training neural networks.