Module II:
Statistical Foundations
Outline
 Descriptive Statistics
 Statistical Features
 Summarizing the Data
 Outlier Analysis
 Understanding Distributions and Plots
 Univariate Statistical plots and usage
 Bivariate and Multivariate statistics
 Dimensionality Reduction
 Over and Under Sampling
 Bayesian Statistics
 Statistical Modeling for Data Analysis
Statistical Foundations (Methods)
Descriptive Statistics (What does the data look like?)
 Descriptive
statistics
has
been
applied
for
summarizing,
organizing,
and
presenting
data meaningfully and concisely.
 It focuses on describing and analyzing a dataset's main features and characteristics without making
any generalizations or inferences to a larger population.
 The primary goal of descriptive statistics is to provide a clear and concise summary of the data,
enabling researchers or analysts to gain insights and understand patterns, trends, and distributions
within the dataset.
 This summary typically includes measures such as central tendency (e.g., mean, median, mode),
dispersion (e.g., range, variance, standard deviation), and shape of the distribution (e.g.,
skewness, kurtosis).
 Descriptive statistics also involves a graphical representation of data through charts, graphs, and
tables, which can further aid in visualizing and interpreting the information.
Descriptive Statistics
Fig.1: Descriptive Statistics
https://www.scribbr.com/statistics/descriptive-statistics
Descriptive Statistics
1. Distribution (Frequency Distribution)
 Datasets consist of a distribution of scores or values.
 Statisticians use graphs and tables to summarize the frequency of every possible
value of a variable, rendered in percentages or numbers.
Example (Dice Rolls)
Value (x)
Frequency (f)
1
2
3
4
5
6
5
4
6
4
6
5
Ungrouped frequency distribution tables
and Used when: data size is small and values
repeat clearly.
Example (Exam marks)
Class Interval
Frequency
0–10
3
10–20
7
20–30
12
30–40
8
Grouped frequency distribution tables and
Used when: data is large and difficult to analyze
individually.
Distribution (Frequency Distribution)- Example
The following are the daily step counts (in thousands) recorded by 12
employees using a fitness tracking app over one week: 6, 8, 10, 6, 9, 10, 8, 7,
6, 9, 8, 10. Represent this data in the form of a frequency distribution table
and determine how many employees recorded the same number of steps
during the week.
Frequency (Number of
Step counts
employees)
6
3
7
1
8
3
9
2
10
3
Total
12
Distribution (Frequency Distribution)- Example
During a mid-semester assessment, a teacher recorded the marks obtained by
20 students in a class to analyze their performance pattern. The marks scored
by the students (out of 20) are as follows:
5, 10, 20, 15, 5, 20, 20, 15, 15, 15, 10, 10, 10, 20, 15, 5, 18, 18, 18, 18
To better understand how the students’ marks are distributed across score
ranges, the teacher decides to organize the data into a grouped frequency
distribution.
Class Interval
0–5
6 – 10
11 – 15
16 – 20
Total
Frequency
3
4
5
8
20
Descriptive Statistics
2. Measures of Central Tendency: A measure of central tendency is a single value that
attempts to describe a set of data by identifying the central position within that set of data
estimate a dataset's average or center, finding the result using three methods: mean,
mode, and median.
Mean: The mean is the most common method for finding averages. You get the mean by
adding all the response values together, and dividing the sum by the number of responses
Mode: The mode is just the most frequent response value. Datasets may have any
number of modes, You can find the mode by arranging your dataset's order from the lowest
to highest value and then looking for the most common response.
Median: Finally, we have the median, defined as the value in the precise center of the
dataset. Arrange the values in ascending order (like we did for the mode) and look for the
number in the set‘s middle.
Mean of Ungrouped Data - Example
The following data represents the number obtained when a standard six-sided
dice is rolled 25 times:
2, 4, 6, 1, 3, 5, 2, 6, 4, 3, 1, 5, 2, 4, 6, 3, 2, 5, 1, 4, 6, 3, 2, 5, 4
Find the mean of the observations.
Mean of Grouped Data - Example
In a city hospital, the time (in minutes) patients wait before meeting a doctor was
recorded for 40 patients. The distribution of waiting times is given below.
Find the mean waiting time.
Waiting Time (minutes) Number of Patients
fi : frequency of data in a class
interval
xi : midpoint of the interval data
= (lower limit + upper limit) /2
0 – 10
6
10 – 20
10
20 – 30
12
30 – 40
8
40 – 50
4
Mean of Grouped Data - Example
In a city hospital, the time (in minutes) patients wait before meeting a doctor was
recorded for 40 patients. The distribution of waiting times is given below.
Find the mean waiting time.
fi : frequency of data in a class
interval
xi : midpoint of the interval data
= (lower limit + upper limit) /2
Waiting Time (min)
(xi)
Frequency (fi) (fi xi)
0 – 10
10 – 20
20 – 30
30 – 40
40 – 50
5
15
25
35
45
6
10
12
8
4
40
xˉ=490/40​=23.5
30
150
300
280
180
490
Mean of Grouped Data - solve
The histogram below represents the marks obtained by 340 students in a Data
science test. Calculate the mean marks.
Mode
Mode is the value that occurs most frequently in a dataset.
A dataset may have one mode (unimodal), two modes (bimodal), multiple
modes (multimodal), or no mode.
Unlike mean, mode is not influenced by outliers.
Widely used in market analysis, education, and fashion to identify most
common choices.
Mode of Ungrouped data
Mode of Ungrouped data
Mode of Grouped data
Mode of Grouped data
Mode of Grouped data
Mode of Grouped data
Studying time (minutes)
1–10
11–20
21–30
31–40
41–50
Number of students
2
10
6
4
3
For the mode formula, the classes must be continuous.
The class intervals are:
1–10, 11–20, 21–30, …
These have gaps between classes (e.g., 10 to 11).
To convert into Continuous (Inclusive) Classes
Class width:
h=11−10=1
Here, h/2=0.5
Subtract h/2 from the lower limit
Add h/2​ to the upper limit
Mode of Grouped data
Studying time (minutes)
1–10
11–20
21–30
31–40
41–50
Number of students
2
10
6
4
3
Median
Median of Ungrouped data
Median of Ungrouped data
Median of Grouped data
Median of Grouped data
Median of Grouped data
Descriptive Statistics
3. Measure of Variability (Also Called Dispersion): The measure of variability gives the
statistician an idea of how spread out the responses are. The spread has three aspects
— range, standard deviation, and variance.
Range: Use range to determine how far apart the most extreme values are. Start by
subtracting the dataset‘s lowest value from its highest value.
Standard Deviation: The standard deviation (s) is your dataset‘s average amount of
variability, showing you how far each score lies from the mean. The larger your
standard deviation, the greater your dataset‘s variable.
Variance: Variance reflects the dataset‘s degree spread. The greater the degree of
data spread, the larger the variance relative to the mean. You can get the
variance by just squaring the standard deviation.
Descriptive Statistics - Range
Statistical Features
 Statistical features are those features of the dataset that can be defined and
calculated via statistical analysis.
 It is the statistical concept that is probably most used in data science.
 When we start exploring a dataset, the first thing we do is to find out the statistical
features of the dataset.
 Following are the some crucial statistical features for the data science.
1. Bias
2. Min/Max
3. Variance
4. Median
5. Mode
6. Quartiles and Inter quartile range (IQR)
Statistical Features - Bias
 In statistics, when there is a systematic difference between the true parameters and the results we estimated,
it is called a bias.
 It essentially means that our statistic is not representing the population properly. There are many sources of
bias, some of them are:
 Selection bias is when there is no proper randomization while selecting data which results in the sample being
unrepresentative of the population.
 Omitted variable bias is when we do not have a required variable/attribute in a model.
 Survivorship bias is when only people or data that made it past some selection criteria are considered and
others are overlooked.
 Observer bias is when the observer's prejudice, opinions and expectations influence how they assess or
record subjective criteria.
 Recall bias is when participants of an experiment do not remember previous events.
Bias Type
Main Cause
Example
Selection Bias
Non-random sampling
Survey only toppers
Omitted Variable Bias
Missing key variable
House price without
location
Survivorship Bias
Ignoring failures
Studying only successful
startups
Observer Bias
Researcher influence
Biased grading
Memory errors
Self-reported past
behavior
Recall Bias
Statistical Features - Variance
 Variance gives us a measure of how much the data points differ from the mean.
 It can also be treated as a measure that tell us how far the data points are spread out.
 It is represented as σ2 and its formula is
 Here, n is the sample size, Xi is the ith data point and μ is the mean.
 The square root of variance is the standard deviation.
Variance and Standard Deviation - Simple Ungrouped Data (Population)
Problem: Find the variance and standard deviation of the data:
2, 4, 6, 8, 10
Variance and Standard Deviation - Grouped Data (Population)
Variance and Standard Deviation - Grouped Data (Population)
Statistical Features - Median
 In a sorted list of numbers (either ascending or descending), median is the middle number
separating the higher half from the lower half.
 We tend to use median instead of mean when we have outliers in the dataset that skews the
average as a median is less affected by outliers than the mean.
 When our data sample consists of odd number of values, then the median is the middle number
such that there are same number of entries above and below it.
 When our data sample consists of even number of values, then the median is the average of the
middle pair of values.
Statistical Features - Quartiles and Inter quartile range (IQR)
 Quartiles are the values that divide the number of observations in our dataset in to four
equal parts. They mark each 25% of the data.
 Q1 marks 25%, Q2 marks the median or 50%, Q3 marks 75%.
 The inter quartile range contains the second and third quartile and tells us how far
is the middle portion of the data is spread.
 Minimum (min) value is calculated as Q1 - 1.5(IQR) and
The maximum (max) value is calculated as Q3 + 1.5(IQR).
Statistical Features
Quartiles and Inter quartile range (IQR)
 All these are perfectly illustrated by a box plot.
 If the median line is not in the middle, then we can conclude that the data is skewed.
 The length of the box tells us how similar or different the data points are.
 A smaller box corresponds to similar data points and a taller box tells us that the data points are wide
spread.
 The length of the whiskers tell us about the variance and standard deviation.
 If they are too long, it means that our data is highly varying.
Summarizing the Data
 When you want to measure something in the natural world you usually have to take
several measurements.
 This is because things are variable, so you need several results to get an idea of the situation.
 Once you have these measurements you need to summarize them in some way because
sets of raw numbers are not easily interpreted by most people.
 There are four key areas to consider when summarizing a set of numbers:
1. Centrality: the middle value or average.
2. Dispersion: how spread out the values are from the average.
3. Replication: how many values there are in the sample.
4. Shape: the data distribution, which relates to how ―evenly the values are spread either side
of the average.
Summarizing the Data
 Centrality(Average):An average is a measure of the middle point of a set of values.
 This central
tendency
(centrality)
is an important
measure
and is usually what you
are
comparing when looking at differences between samples for example.
 There are three main kinds of average:
 Mean: The arithmetic mean is calculated by adding together the values in the sample. The
sum is then divided by the number of items in the sample (the replication).
 Median: The median is the middle value, taken when you arrange your numbers in order
(rank). This measure of the average does not depend on the shape of the data.
 Mode: The Mode is the most frequent value in a sample. It is calculated by working out how
many there are of each value in your sample. The one with the highest frequency is the
mode. It is possible to get tied frequencies, in which case you report both values. The sample
is then said to be bimodal.
Summarizing the Data
 Dispersion: The dispersion of a sample refers to how
spread out the values are around the average.
 If the values are close to the average, then your sample
has low dispersion.
 If the values are widely scattered about the average
your sample has high dispersion.
 There are various measures of dispersion, such as:
 Standard deviation:
 Variance
 Standard Error
 Confidence Interval
 Inter-Quartile Range
 Range
Summarizing the Data
 Replication: This is the simplest of the summary statistics but it is still important.
 The replication is simply how many items there are in your sample (that is, the
number of observations).
 The value n, the replication, is used in calculating other summary statistics, such as
standard deviation and IQR, but it is also helpful in its own right.
 You should look at the dispersion and replication together.
Summarizing the Data
 Shape: The shape of the data affects the type of summary statistics that best summarize them.
 The shape refers to how the data values are distributed across the range of values in the sample.
 Generally you expect there to be a cluster of values around the average.
 It is important to know if the values are more or less symmetrically arranged around the average,
or if there are more values to one side than the other.
 There are two main ways to explore the shape (distribution) of a sample of data values:
I.
Graphically: using frequency histograms or tally plots draws a picture of the sample shape.
II.
Shape statistics: such as skewness and kurtosis. These give values to how central the average
is and how clustered around the average the data are.
 Types of data distribution: There are many ―
 Normal (also called Gaussian)
 Poisson
 Binomial
shapes of data, commonly encountered ones are: