20/07/2024, 19:06 A cheat sheet for statistical estimators | by Thibaut | Medium archive.today webpage capture Webpage Saved from https://thibaut-deveraux.medium.com/a-cheat-sheet-for-statistical-estimators- search 15 Oct 2023 06:46:47 UTC no other snapshots from this url All snapshots from host thibaut-deveraux.medium.com Screenshot share Search download .zip Write report bug or abuse Sign up Buy me a coffee Sign In Member-only story A cheat sheet for statistical estimators What distribution and parameters fit our data? What is the confidence interval of the estimated parameters? Thibaut · Follow 10 min read · Sep 22, 2021 72 Photo by Launde Morel on Unsplash The above article contains the formulas with minimal explanations. If you have seen this topic some time ago, it is probably a good way to refresh memories. However, this would be a bit too harsh to start from scratch with it. This is, for a big part, formulas from my notes of the course at the DSTI, as we had an excellent teacher, with some parts from other sources. Hypothesis and notations X is a random variable x is a realization of this random variable https://archive.is/lHyJF 1/18 20/07/2024, 19:06 A cheat sheet for statistical estimators | by Thibaut | Medium f is the associated density function F is the associated cumulative distribution function g and h are any functions of X i, j, k, n are integers θ in an unknown parameter ₙ ₙ θ̂ is an estimator of θ according to X₁, … X . It is a random variable. ₙ A specific realization of θ according to the observations is referred to as an estimation. Reminders Photo by Ferenc Horvath on Unsplash Just some friendly reminders to make sure we have the formulas on hand. If some notions need more explanations, see the article: Descriptive statistics and probability formulas. Expected value Variance Transfert formula https://archive.is/lHyJF 2/18 20/07/2024, 19:06 A cheat sheet for statistical estimators | by Thibaut | Medium Moments Law of large numbers ₖ X₁, …, Xᵢ, …, X being independent and identically distributed random variables with an expectation μ. It even tends almost surely to μ if each Xᵢ has a moment of order 1. Central limit theorem ₖ X₁, …, Xᵢ, …, X being independent and identically distributed random variables, with an expectation μ and variance σ². N is the normal distribution. Some distributions we will need Before entering into estimators, we need to define some distributions. Underlined titles link to Wikipedia. Normal distribution https://archive.is/lHyJF 3/18 20/07/2024, 19:06 A cheat sheet for statistical estimators | by Thibaut | Medium By Inductiveload, Wikimedia Commons, Public domain. Change of variable: Chi-squared distribution By Geek3, Wikimedia Commons, CC BY 3.0 https://archive.is/lHyJF 4/18 20/07/2024, 19:06 ₖ A cheat sheet for statistical estimators | by Thibaut | Medium X₁, …, Xᵢ, …, X being independent random variables with a gaussian distribution. k is referred to as the degree of freedom. Student distribution By Skbkekas, Wikimedia Commons, CC BY 3.0 U and C are random variables. U has a standard normal distribution and C a Chi-Squared distribution with k degrees of freedom. Fisher distribution https://archive.is/lHyJF 5/18 20/07/2024, 19:06 A cheat sheet for statistical estimators | by Thibaut | Medium By IkamusumeFan, Wikimedia Commons, CC BY-SA 4.0 C₁ and C₂ being Chi-Squared distributions with k₁ and k₂ degrees of freedom. Point estimators https://archive.is/lHyJF 6/18 20/07/2024, 19:06 A cheat sheet for statistical estimators | by Thibaut | Medium Photo by Heidi Walley on Unsplash ₙ A point estimator θ̂ is a random variable that represents the estimated value of a parameter θ according to the data. It is a single value, in contrast to Interval Estimators. Method of moments ₙ X₁, …, Xᵢ, …, X being independent and identically distributed random variables. We are looking for a function g of θ that verifies: Note: the expected value is the same for all Xᵢ. ₙ Then, knowing this function g, we calculate an estimator θ̂ of θ by finding a solution of: For example, the estimated value of the exponential distribution being 1/λ, we can take k=1: Method of moments ₙ ₙ X₁, …, Xᵢ, …, X being independent random variables and x₁, …, xᵢ, …, x specific realizations. https://archive.is/lHyJF 7/18 20/07/2024, 19:06 A cheat sheet for statistical estimators | by Thibaut | Medium The associated likelihood is: With independent random variables: We can use this to expresses the likelihood as a function of θ: ₙ Then, knowing l, we try to find the value of θ̂ that maximize the likelihood to have these observations by solving: For example, the estimated value of the exponential distribution being 1/λ, we can solve, for xᵢ > 0 using: Bias The bias is the difference between the real estimated value and the expected value of the estimator. We prefer to use unbiased estimators, which mean: Or at least asymptotically unbiased estimators: https://archive.is/lHyJF 8/18 20/07/2024, 19:06 A cheat sheet for statistical estimators | by Thibaut | Medium When we have a biased estimator, it is preferable to transform it to have an unbiased one. Consistency ₙ An estimator θ̂ is consistent if it tends to θ in probabilities. This formula is quite unpractical. Fortunately, we can use a theorem on the variance: Once again, we prefer to use consistent estimators. An unbiased and consistent estimator of the expectation An unbiased and consistent estimator of the variance Quadratic error The quadratic error is the expectation of the squared difference between the estimator and the true value. It can also be expressed as a function of the variance and the bias. So for an unbiased estimator, it is just the variance. It is often used to compare estimators. It is desirable to have a lower quadratic error. Cramer Rao bound It is the inverse of the Fisher Information. It is a lower bound for the variance and, then, for the quadratic error, of an unbiased estimator. https://archive.is/lHyJF 9/18 20/07/2024, 19:06 A cheat sheet for statistical estimators | by Thibaut | Medium Interval Estimators Photo by Patricia Serna on Unsplash With interval estimators, we are interested in the interval of plausible values according to the sample data. Confidence interval ₙ ₙ ₙ A and B are functions of the random variables X₁, …, X , whose distribution depends on the parameter θ. They are the bonds of a confidence interval for g(θ) with confidence level (1-α) such that: The strategy is to define g such that it makes it easy to calculate the interval. We usually take g such that the distribution of the point estimator ĝ is known, at least approximately. Confidence interval for µ in the Gaussian case: N(µ,σ²) If σ is known. A good candidate for µ̂, whose distribution is known, is: https://archive.is/lHyJF 10/18 20/07/2024, 19:06 ₙ ₙ A cheat sheet for statistical estimators | by Thibaut | Medium ₙ We express A and B using X . This is like saying that the real µ is ₙ ₙ somewhere near µ̂ minus m or plus M . What can also be understood as bounds on the difference: That can be transformed to introduce the standard gaussian: We have to decide how we center the interval. Let’s say that we want that the probabilities to be > or < to be equal: α₁ = α₂ = α/2. Thanks to the symmetry of ₙ ₙ the Gaussian density function, we also have m = M . Using the distribution function of the standard gaussian, we find that: With z the quantile for the gaussian: The resulting confidence interval for µ with a confidence level (1-α) is: How to compute the quantiles of the gaussian distribution? In R, the quantile z can be calculated with qnorm, In Python with scipy.stats.norm.ppf, and in Julia with quantile.(Normal(), …). https://archive.is/lHyJF 11/18 20/07/2024, 19:06 A cheat sheet for statistical estimators | by Thibaut | Medium Aside from being more readable, Julia is the most practical as we can enter any interval. If σ is unknown. When σ is also unknown, we can estimate it: Then we can do a little magic by plugging in the estimator of σ in the central term of the previous case’s inequality, which changes the normal distribution into a student distribution. Similarly, we will find the confidence interval: How to compute the quantiles of the student distribution? In R, the quantile t can be calculated with qt, In Python with scipy.stats.t.ppf, and in Julia with quantile.(TDist(), …). Non-gaussian case Thanks to the central limit theorem, the result is still valid. The difference is that we have an exact confidence interval in the gaussian case and an asymptotic confidence interval in the general case. This means that it is valid when the number of observations is high. When σ is unknown, we cannot rely on the student function. However, σ tends towards its estimator when n tends to ∞. In the asymptotic case, the ₙ above formula is also valid, replacing σ with σ̂ . Intuitively, we remark that a student tends towards a gaussian when its degree of liberty tends towards ∞. https://archive.is/lHyJF 12/18 20/07/2024, 19:06 A cheat sheet for statistical estimators | by Thibaut | Medium Unilateral confidence intervals Similarly, we can decide to ensure a lower or a higher bound. We use the same framework, yet take α₁ = 0 and α₂ = α or 1-α. Similarly, if σ is unknown, we can plug its estimated value. We need to replace the quantiles of the gaussian distribution with the ones of the student distribution to get an exact confidence interval in the gaussian case. Confidence interval for σ in the Gaussian case: N(µ,σ²) We consider the non-biased and consistent point estimator for σ². ₙ ₙ ₙ ₙ ₙ ₙ We take A = σ̂² . m and B = σ̂² . M . What can be transformed into: The central term is a Chi-Squared distribution of degree n-1. So, similarly, we can use its quantiles. We will let the formula in its general form with α₁+α₂=α. Delta Method ₙ X₁, …, X being independent and identically distributed random variables of expectation µ and variance σ². Thanks to the law of large numbers, we know that: https://archive.is/lHyJF 13/18 20/07/2024, 19:06 A cheat sheet for statistical estimators | by Thibaut | Medium We take g, a derivable function such that g’(μ) ≠ 0. Then we can use: For example, in the case of the exponential random variable. The expectation is 1/λ. So we could try to build a confidence interval for λ using g(x)=1/x. Using a centered interval, we would conclude that λ is in: Case of a proportion p ₙ X₁, …, X being independent and identically distributed random variables with a Bernoulli distribution of parameter p. Rather than relying on the precedent intervals, we usually redo the things from scratch with specific approaches. Option 1 We can rely on the Beinaymé — Tchebychev theorem: What we can write, in our case, as: There, ξ is just chosen to correspond to a function of α. By the end, we get the tolerance interval for p with a confidence level 1-α: https://archive.is/lHyJF 14/18 20/07/2024, 19:06 A cheat sheet for statistical estimators | by Thibaut | Medium Option 2 We can also rely on the central limit theorem with: Then, we can use the quantiles of the gaussian distribution: Relying on the fact that p(1-p) ≤ 1/4 we can write an asymptotic tolerance interval for p with a confidence level 1-α: Or, by plugging the excepted value of p: Statistics Statistical Analysis Data Science Estimations Mathematics 72 Written by Thibaut Follow 326 Followers Publications in English & French about Data Science, Artificial Intelligence, and Innovation. More from Thibaut https://archive.is/lHyJF 15/18 20/07/2024, 19:06 A cheat sheet for statistical estimators | by Thibaut | Medium Thibaut Thibaut Common special characters and Unicode symbols to copy-paste f… A Docker Compose file to install WordPress with a Traefik reverse… Because it is sometime useful… If you are planning to host your blog or create a website, Wordpress is the solution of… · 4 min read · May 3, 2021 · 4 min read · Jun 5, 2020 3 104 Thibaut Thibaut 6 How to install Neo4j with DockerCompose? Introduction to simulation with NetLogo: how to create a small… I guess if you landed there, it’s because you already have your idea of what property… NetLogo was designed as an introduction to a type of simulation that allows putting into… · 5 min read · Dec 1, 2019 · 13 min read · Feb 7, 2022 51 73 2 See all from Thibaut Recommended from Medium Ann Mary Shaju in Towards AI https://archive.is/lHyJF ajaymehta 16/18 20/07/2024, 19:06 A cheat sheet for statistical estimators | by Thibaut | Medium Univariate, Bivariate, and Multivariate Analysis “Unpacking the Basics of Statistical Hypothesis Testing for… A beginner guide to exploratory data analysis using Matplotlib and Seaborn Hypothesis testing is a statistical method used to determine whether a hypothesis or… 6 min read · May 18 13 min read · Apr 16 159 19 1 Lists Predictive Modeling w/ Python New_Reading_List 174 stories · 149 saves 20 stories · 488 saves Practical Guides to Machine Learning Coding & Development 11 stories · 216 saves 10 stories · 557 saves Rahul S Wojciech kowalczyk An Intuitive Exploration of Probability: Approaches,… The Gamma function; explanation and usage. This article delves into the multifaceted realm of probability, unraveling its fundamental… Maybe you’ve seen this strange Γ floating around. I’ve personally used it in my other… · 9 min read · Jun 17 5 min read · Jul 17 8 Manoj Mangam Multicollinearity Problems in Linear Regression. Clearly… De-coded: Transformers explained in plain English A behind-the-scenes look at the infamous multicollinearity No code, maths, or mention of Keys, Queries and Values 11 min read · Mar 21 15 min read · 5 days ago 289 https://archive.is/lHyJF Chris Hughes in Towards Data Science 7 626 5 17/18 20/07/2024, 19:06 A cheat sheet for statistical estimators | by Thibaut | Medium See more recommendations Help https://archive.is/lHyJF Status About Careers Blog Privacy Terms Text to speech Teams 18/18
