AI, ML & dAtA AnALysIs CoMpILed by: AMMAr ArAIn dAte: 3/31/2026 1. The Big Picture: How They Relate People often use these terms interchangeably, but they actually fit inside one another like Russian nesting dolls: • Artificial Intelligence (AI) is the broadest concept. It's the goal of making machines mimic human intelligence and behavior. • Machine Learning (ML) is a specific subset of AI. It's the method we use to get there—by training algorithms on data so they can learn patterns without being explicitly programmed. • Data Analytics is the practice of examining raw data to find trends and answer questions. It overlaps with both AI and ML to help make sense of the past and predict the future. 2. Breaking Down the Trio Artificial Intelligence (AI) AI is any technique that enables computers to mimic human intelligence. This includes everything from the chess-playing computers of the 1990s to the voice assistants and generative chatbots you use today. • How it works: It uses rules, logic, or learning algorithms to perceive its environment, make decisions, and solve problems. • Real-world examples: Self-driving cars, facial recognition on your phone, and AI art generators. Machine Learning (ML) Instead of writing a strict list of rules for a computer to follow (like "if X happens, do Y"), Machine Learning feeds the computer thousands of examples and lets the computer figure out the rules on its own. • How it works: You feed an algorithm data (e.g., thousands of pictures of cats and dogs). The algorithm learns the features that distinguish them and can eventually identify a cat in a brand-new picture it has never seen before. • Real-world examples: Netflix recommendations, email spam filters, and predictive text on your keyboard. Data Analytics Data Analytics is all about looking at the data we already have to uncover insights, draw conclusions, and support decision-making. • How it works: It generally falls into four stages: 1. Descriptive: What happened? (e.g., "Our sales went up 10% last month.") 2. Diagnostic: Why did it happen? (e.g., "Because we ran a successful ad campaign.") 3. Predictive: What is likely to happen next? (This is where ML steps in!) 4. Prescriptive: What should we do about it? • Real-world examples: A retail store analyzing purchasing habits to place popular items next to each other, or sports teams using player stats to draft the best roster. Summary of Differences To keep it super simple, here is a quick comparison: Feature Data Analytics Machine Learning Artificial Intelligence Primary Goal To find patterns and answer questions from data. To create algorithms that learn from data and improve automatically. To create smart machines capable of mimicking human intelligence. Focus Past and present trends. Future predictions and pattern recognition. Action, automation, and decision-making. Human Involvement High (humans interpret the charts and reports). Medium (humans train the model, but it learns on its own). Low to High (depending on how autonomous the system is). Mathematical Foundations 1. Linear Algebra (The Language of Data) In AI, data isn't just a list; it’s stored in Matrices and Vectors. Whether it's a grayscale image (a 2D matrix of pixel intensities) or a database of user profiles, Linear Algebra allows us to perform operations on huge chunks of data simultaneously. • Key Concepts: Matrix multiplication, Eigenvalues, and Eigenvectors. • Why it matters: Principal Component Analysis (PCA) uses linear algebra to reduce the "noise" in data, keeping only the most important features. 2. Calculus (The Optimizer) If Machine Learning is about "learning," Calculus is the tool that tells the computer how to improve. Specifically, we use Differential Calculus. • Key Concept: Gradient Descent. This is an optimization algorithm used to minimize the "Loss Function" (the error rate of a model). • How it works: Imagine standing on a foggy mountain (high error) and wanting to get to the valley (low error). Calculus helps you calculate the "slope" beneath your feet so you can take a step in the steepest downward direction. 3. Probability & Statistics (The Decision Maker) Since the real world is messy and uncertain, AI uses probability to make "best guesses." Data Analytics relies almost entirely on statistics to validate if a trend is real or just a random fluke. • Descriptive Statistics: Mean, Median, Variance, and Standard Deviation (summarizing what the data looks like). • Inferential Statistics: Using a small sample of data to make predictions about a large population. • Bayes' Theorem: A way to update the probability of a hypothesis as more evidence or information becomes available. This is the foundation of "Naive Bayes" classifiers used in spam detection. 4. Discrete Mathematics This is the backbone of the Logic and Graph Theory used in AI. • Graph Theory: Essential for understanding Neural Networks (which are essentially weighted graphs) and Social Network Analysis. • Boolean Logic: The foundation of decision trees and rule-based expert systems. Measures of Central Tendency These tell you where the "middle" of your data lies. The Mean (Average) The sum of all values divided by the total number of values. Best for: Data that is spread out evenly. • Weakness: It is very sensitive to outliers (extreme high or low values). For example, if 9 people earn $30k and 1 person earns $1M, the "mean" income will look much higher than what most people actually make. The Median (Middle Value) The middle number in a list that is sorted from smallest to largest. • How to find it: If you have an odd number of values, it's the center one. If you have an even number, it's the average of the two center values. • Best for: Skewed data (like salaries or home prices) because it isn't affected by extreme outliers. The Mode (Most Frequent) The value that appears most often in a dataset. • Best for: Categorical data (e.g., "What is the most popular programming language in this survey?"). A dataset can have one mode, multiple modes, or no mode at all. Python for AI & Data Analytics Python is the primary language for AI and Data Analytics because it is easy to read and has a massive ecosystem of pre-built libraries. Instead of writing complex mathematical formulas from scratch, you use these libraries to do the heavy lifting. Think of Python as the "glue" that connects your data to the mathematical foundations we discussed. 1. The Core Library Stack To work in this field, you generally need to master these four "pillars": NumPy (Numerical Python) This is the foundation for all scientific computing in Python. It allows you to handle Arrays and Matrices efficiently. • Why use it: Standard Python lists are slow. NumPy arrays are optimized for speed and memory, making them perfect for the "Linear Algebra" part of AI. • Key feature: Vectorization (performing an operation on an entire array at once). Pandas (Data Manipulation) If NumPy is for matrices, Pandas is for Tables. It introduces the DataFrame, which is essentially a super-powered Excel spreadsheet inside your code. • Why use it: It’s the go-to tool for "Data Cleaning." You use it to handle missing values, filter rows, and calculate the Mean, Median, and Mode we just covered. Matplotlib & Seaborn (Visualization) These libraries turn your numbers into graphs. • Matplotlib: The flexible, "low-level" library for creating basic plots (line, bar, scatter). • Seaborn: Built on top of Matplotlib, it makes high-level statistical graphics (like Heatmaps and Violin plots) look beautiful with very little code. Scikit-Learn (Machine Learning) This is the most popular library for traditional Machine Learning. It contains tools for: • Regression: Predicting a number (like house prices). • Classification: Predicting a category (like "Spam" or "Not Spam"). • Clustering: Grouping similar data points together. 2. Deep Learning Frameworks When you move past basic Machine Learning into Neural Networks (AI that mimics the brain), you use specialized frameworks: • TensorFlow / Keras: Developed by Google. It is very powerful for production-scale AI models. • PyTorch: Developed by Meta (Facebook). It is widely loved by researchers for being "Pythonic" and flexible. 3. A Simple Workflow Example Here is how these libraries work together in a real-world scenario: 1. Pandas: Load a CSV file of student grades and fill in any missing scores. 2. Matplotlib: Plot a histogram to see the "Normal Distribution" of the grades. 3. NumPy: Convert the grades into a matrix for the computer to process. 4. Scikit-Learn: Train a model to predict which students might need extra help based on their past performance. Basic Code Structure Here are the most common functions you will use to inspect and clean your data. 1. Inspecting the Data Before you fix anything, you need to see what's inside. • df.head(): Shows the first 5 rows of your data. • df.info(): Gives a technical summary (column names, data types, and how many non-null values exist). • df.describe(): Automatically calculates the Mean, Standard Deviation, Min, and Max for all numerical columns. 2. Handling Missing Values (.isnull()) Missing data (NaN) is a huge problem for AI models. These functions help you find and fix them. Finding Nulls • df.isnull(): Returns True for every empty cell and False for filled ones. • df.isnull().sum(): This is the most common way to use it. It tells you exactly how many missing values are in each column. Fixing Nulls • df.dropna(): Removes any row that has a missing value. (Use this if you have a lot of data and only a few missing spots). • df.fillna(value): Replaces missing values with a specific number or text. • Pro Tip: Often, we fill missing numbers with the Mean or Median of that column: df['Age'] = df['Age'].fillna(df['Age'].mean()) Data Collection & Pre-processing Before you can apply any fancy AI models or statistical analysis, you have to get your data ready. In the industry, it is often said that 80% of a data scientist's job is data cleaning and preprocessing, while only 20% is actual modeling. Here is the breakdown of how we move from raw, "dirty" data to a clean dataset ready for an algorithm. 1. Data Collection Data doesn't always come in a neat .csv file. It can be collected from various sources: • Databases: Pulling data using SQL from structured tables. • Web Scraping: Using Python libraries like BeautifulSoup or Scrapy to extract information from websites. • APIs: Requesting data from services like Google Maps, Twitter (X), or weather stations. • Sensors/IoT: Real-time data from devices (GPS, temperature sensors, etc.). 2. Data Pre-processing (The 4 Key Steps) Once you have the data, you must perform "Data Wrangling" to make it usable. A. Data Cleaning This is where you handle the "mess." • Handling Missing Values: Using .isnull() to find gaps and .fillna() or .dropna() to fix them. • Removing Duplicates: Using df.drop_duplicates() to ensure no redundant data biases the model. • Outlier Detection: Identifying extreme values (like a "Student Age" of 200) that will break your Mean and Standard Deviation. B. Data Transformation Converting data into a format the computer prefers (mostly numbers). • Categorical Encoding: Machines can't read "Red," "Green," or "Blue." We convert these to numbers: o Label Encoding: Red=1, Green=2, Blue=3. o One-Hot Encoding: Creating separate columns for each color with 0s and 1s. • Scaling & Normalization: If one column is "Salary" (50,000) and another is "Age" (25), the machine might think Salary is more "important" just because the number is bigger. Scaling brings both ranges to a scale of 0 to 1. C. Data Reduction If you have 1,000 columns, your model will be very slow and "confused." • Feature Selection: Keeping only the columns that actually matter (e.g., "Shoe Size" probably doesn't help predict "Exam Score"). • PCA (Principal Component Analysis): A mathematical way to merge multiple columns into a few "principal" ones without losing the core information. D. Data Integration Combining data from different sources (e.g., merging a "Sales" table with a "Customer" table) using the pd.merge() function in Pandas. Exploratory Data Analysis (EDA) is the "detective work" of data science. Before you build an AI model, you perform EDA to understand the stories your data is telling, find hidden patterns, and spot anomalies that might ruin your results. Think of it as getting to know a person before you decide to work with them; you want to know their habits, their strengths, and if they have any "red flags." 1. The Three Main Goals of EDA 1. Understand the Data Structure: How many rows/columns? What are the data types (Integers, Floats, Strings)? 2. Find the "Shape" of the Data: Is it a Normal Distribution (Bell Curve)? Is it skewed to one side? 3. Detect Relationships: Does "Study Hours" actually correlate with "Exam Score"? EDA proves or disproves your assumptions. 2. Univariate Analysis (One Variable) We look at a single column at a time to see its distribution. • For Numerical Data: We use Histograms or Box Plots. o Box Plots are amazing for spotting Outliers. If you see a "dot" far away from the whiskers of the box, that's an outlier. • For Categorical Data: We use Bar Charts or Pie Charts to see the frequency of different groups. 3. Bivariate & Multivariate Analysis (Multiple Variables) This is where we see how columns interact with each other. • Scatter Plots: Perfect for seeing the relationship between two numbers (e.g., Height vs. Weight). • Correlation Matrix (Heatmap): This gives you a "score" between -1 and 1 for every pair of columns. o 1.0: Perfect positive relationship (as X goes up, Y goes up). o 0.0: No relationship at all. o -1.0: Perfect negative relationship (as X goes up, Y goes down). In EDA, we often use the IQR to mathematically define an outlier. • IQR = Q3 (75th percentile) - Q1 (25th percentile) • Anything 1.5x the IQR above Q3 or below Q1 is usually considered an outlier and should be investigated. Supervised Learning (Learning with a Teacher) In Supervised Learning, the data is labeled. This means for every input, the computer already knows the correct answer (the "Ground Truth"). It learns a mapping function from input to output. • Regression: Predicting a continuous number (e.g., House Price, Temperature). o • Algorithms: Linear Regression, Support Vector Regression (SVR). Classification: Predicting a discrete label or category (e.g., Spam vs. Not Spam). o Algorithms: Logistic Regression, Decision Trees, K-Nearest Neighbors (KNN), Support Vector Machines (SVM). 2. Ensemble Learning (The Power of the Crowd) Ensemble Learning is a technique where you combine multiple models (often called "weak learners") to create one "strong learner" that is more accurate and stable. • Bagging (Bootstrap Aggregating): Training multiple models independently and taking the average or majority vote. o • Boosting: Training models sequentially, where each new model tries to fix the errors made by the previous one. o • Algorithm: Random Forest (A collection of many Decision Trees). Algorithms: XGBoost, AdaBoost, Gradient Boosting. Stacking: Using a "meta-model" to combine the predictions of several different types of models. 3. Unsupervised Learning (Finding Hidden Patterns) In Unsupervised Learning, the data is unlabeled. The computer doesn't know the "right" answer; its job is to find interesting structures or groupings within the data on its own. • Clustering: Grouping similar data points together. o • Association: Finding rules that describe your data (e.g., "People who buy milk also tend to buy bread"). o • Algorithms: K-Means Clustering (grouping by distance), DBSCAN (grouping by density), Hierarchical Clustering. Algorithm: Apriori Algorithm. Dimensionality Reduction: Compressing data while keeping important info. o Algorithm: PCA (Principal Component Analysis). Comparison of Key Algorithms Category Algorithm Use Case Supervised Linear Regression Predicting sales based on advertising budget. Category Algorithm Use Case Supervised Decision Trees Classifying if a bank loan should be approved. Ensemble Random Forest High-accuracy medical diagnosis (very robust). Ensemble XGBoost Winning Kaggle competitions and complex predictions. Unsupervised K-Means Segmenting customers into "High Spenders" and "Bargain Hunters." Unsupervised PCA Speeding up AI models by reducing 100 features to 5. We are now at the final and most critical stages of the AI pipeline. This is where you transform your data into high-performing models and then prove that they actually work. Model Evaluation & Validation Once you train a model, how do you know it's any good? You can't just test it on the same data it learned from (that's like giving a student the exam answers before the test). A. Train-Test Split We usually split data into 80% Training (to learn) and 20% Testing (to evaluate). • Overfitting: The model is too "smart" for its own good—it memorizes the training data but fails on new data. • Underfitting: The model is too simple and fails to learn any patterns at all. B. Evaluation Metrics • For Regression: * Mean Absolute Error (MAE): The average physical distance from the true value. o • R-Squared ($R^2$): How much of the variance is explained by the model (0 to 1). For Classification: * Accuracy: Total correct predictions / Total predictions. o Confusion Matrix: A table showing True Positives, True Negatives, False Positives (Type I Error), and False Negatives (Type II Error). o F1-Score: The balance between Precision (exactness) and Recall (completeness). Feature Engineering & Selection This is the "art" of Machine Learning. It involves creating better inputs for your model. • Feature Selection: Removing "noisy" or useless columns (e.g., a student's "Roll Number" doesn't help predict their "Grade"). • Feature Engineering: Creating new columns from existing ones. o • Example: If you have "Birth Date," you can calculate "Age." Age is much more useful for a model than a specific date. Handling Skewness: Using Log Transformations to make data look more like a Normal Distribution. Deep Learning Fundamentals Deep Learning is a subset of AI inspired by the human brain. While standard ML uses formulas like Linear Regression, Deep Learning uses Neural Networks. The Architecture of a Neural Network 1. Input Layer: Receives the raw data (pixels, text, or numbers). 2. Hidden Layers: This is where the "Deep" in Deep Learning comes from. These layers perform complex mathematical transformations using Weights and Biases. 3. Output Layer: Provides the final prediction (e.g., "This image is a cat"). Key Concepts • Activation Functions (ReLU, Sigmoid): These decide whether a "neuron" should fire or stay silent. • Backpropagation: The "learning" mechanism. The model looks at its error at the end and goes backward through the layers to adjust the weights. • Epochs: One full pass of the entire dataset through the neural network. Summary Roadmap 1. Collect data. 2. Clean it (handle .isnull()). 3. Explore it (EDA). 4. Engineer features (make it "smart"). 5. Train a model (Supervised/Unsupervised). 6. Evaluate (Check the Confusion Matrix).
0
You can add this document to your study collection(s)
Sign in Available only to authorized usersYou can add this document to your saved list
Sign in Available only to authorized users(For complaints, use another form )