Introduction to Multiple Regression
Shirali Kadyrov
1 / 14
Learning objectives
Extend the simple regression idea from one predictor to several predictors.
Interpret a coefficient in multiple regression as a partial slope: the expected change in Y
when one predictor changes and the others are held fixed.
State the main requirements / assumptions for ordinary least squares (OLS), including
the no perfect multicollinearity requirement.
Recognize what collinearity does: it does not create bias by itself, but it can make
estimates unstable and standard errors large.
Read a basic multiple regression model, assess fit, and answer conceptual questions about
it.
2 / 14
Why move beyond simple regression?
Simple regression
Yi = β0 + β1 Xi + εi
This is useful when one predictor is the main focus.
Multiple regression
Yi = β0 + β1 X1i + β2 X2i + · · · + βp Xpi + εi
Now we can model an outcome using several predictors at the same time.
Real outcomes usually depend on more than one factor.
Including relevant predictors helps us control for confounding variables.
Running several separate simple regressions is not the same as one multiple regression,
because each simple regression leaves out other predictors.
3 / 14
Model and coefficient interpretation
Yi = β0 + β1 X1i + β2 X2i + · · · + βp Xpi + εi
Interpretation of βj
βj is the expected change in the mean of Y for a one-unit increase in Xj , holding all other
included predictors fixed.
β0 is the expected value of Y when all predictors equal 0. It may or may not be
substantively meaningful.
Each slope is a conditional or partial effect within the model.
Be careful: association is not automatically causation. Causal interpretation requires
stronger design and identification assumptions.
Key idea: in simple regression, the slope is a marginal relationship; in multiple regression, the
slope is a relationship after adjusting for other predictors in the model.
4 / 14
Requirements / assumptions for OLS
Core requirements for the OLS coefficients to be unbiased
1
Linearity in parameters: the model is linear in the coefficients β0 , β1 , . . . , βp .
2
Independent observations / appropriate sampling design: observations should be
independent, or dependence must be modeled appropriately.
3
Variation in the predictors: each predictor must vary in the sample.
4
No perfect multicollinearity: no predictor can be an exact linear combination of the others.
5
Zero conditional mean: E[εi | X1i , . . . , Xpi ] = 0. This means the omitted influences in εi are
not systematically related to the included predictors.
Additional conditions often used for inference
For the usual standard errors, t-tests, and confidence intervals, we also often assume constant
variance of errors (homoscedasticity). In small samples, approximately normal errors make the usual
t and F procedures more reliable.
5 / 14
What is collinearity?
Definition
Multicollinearity means two or more predictors carry overlapping information because they
are highly linearly related.
Perfect collinearity
High (but not perfect) collinearity
Example: include both Hours studied and
Minutes studied.
OLS still runs.
Then one predictor is an exact linear
transformation of the other.
Standard errors increase.
OLS cannot estimate unique coefficients.
Coefficients can become unstable.
Signs and magnitudes may change a
lot when a variable is added or
removed.
Important
High collinearity does not by itself bias OLS coefficients; its main effect is to make estimation
less precise.
6 / 14
Diagnosing and handling collinearity
Look for predictors that are conceptually redundant or very strongly correlated.
Warning signs: large standard errors, wide confidence intervals, unstable coefficients, or
variance inflation factors (VIFs) that are clearly elevated.
Common responses:
remove one of two nearly redundant predictors,
combine related predictors into an index or composite,
collect more data, especially with more independent variation,
center variables when using polynomial or interaction terms (helps interpretation and can
reduce nonessential collinearity).
Prediction can still be good even when individual coefficients are hard to interpret.
Do not overreact
If your goal is explanation, collinearity matters a lot. If your goal is pure prediction, it matters
mainly when it hurts out-of-sample performance or interpretability.
7 / 14
Fit, inference, and model reading
R2 : proportion of variation in Y explained by the model.
Adjusted R2 : penalizes adding predictors that do not improve fit enough.
t-test for one coefficient: asks whether a specific βj is plausibly 0 after controlling for
the other predictors.
F -test: asks whether a group of coefficients, or all slopes together, are jointly 0.
Residual checks: look for nonlinearity, unequal variance, influential observations, or
patterns suggesting a missing variable.
Two facts students often mix up
1
2
Adding a predictor can never decrease ordinary R2 .
Adding a predictor can decrease adjusted R2 if the new predictor adds little useful
information.
8 / 14
Example 1: STEM education context
Suppose we model a student’s final physics score using:
X1 = prior math diagnostic score,
X2 = number of lab sessions attended,
X3 = percentage of weekly problem sets completed.
\ = 12 + 0.30X1 + 2.4X2 + 0.18X3
FinalScore
Interpretations
+1 diagnostic point ⇒ about +0.30 final-score points, others fixed.
+1 lab session ⇒ about +2.4 final-score points, others fixed.
+10 completion points ⇒ about +1.8 final-score points, others fixed.
Possible collinearity issue: prior math score and problem-set completion may move together,
increasing standard errors.
9 / 14
Example 2: Economics & Data Science context
Suppose we model a store’s weekly sales revenue using:
X1 = product price (in dollars),
X2 = digital advertising spend (in dollars),
X3 = holiday-week indicator (1 = holiday week, 0 = otherwise).
[ = 5200 − 180X1 + 0.90X2 + 650X3
Sales
Interpretations
+$1 in price ⇒ about −$180 in sales revenue, others fixed.
+$100 in ad spend ⇒ about +$90 in sales revenue, others fixed.
Holiday week = 1 ⇒ about +$650 in sales revenue, others fixed.
Possible collinearity issue: price discounts often coincide with holidays, increasing coefficient
uncertainty.
10 / 14
A practical workflow for using multiple regression
1
Start with a clear outcome variable Y and predictors motivated by the question.
2
Check units and coding carefully, especially for indicators such as 0/1 variables.
3
Fit the model and interpret coefficients as holding the other predictors fixed.
4
Check whether signs, magnitudes, and standard errors make substantive sense.
5
Look for collinearity, nonlinearity, unequal variance, influential points, and
omitted-variable concerns.
6
Report uncertainty with confidence intervals or standard errors, not just point estimates.
Remember
Multiple regression is powerful because it lets us adjust for several predictors at once, but it is
only as trustworthy as the model assumptions and the quality of the data.
11 / 14
Check for understanding: MCQ 1-2
1. In a multiple regression model, what does β2 mean?
1
The total effect of X2 without considering any other predictor.
2
The expected change in Y for a one-unit increase in X2 , holding the other included predictors
fixed.
3
The correlation between Y and X2 .
4
The change in X2 caused by a one-unit increase in Y .
2. Which statement about multicollinearity is correct?
1
It always biases OLS coefficients.
2
It means the response variable is measured with error.
3
It can inflate standard errors and make coefficients unstable.
4
It implies that R2 must be small.
12 / 14
Check for understanding: MCQ 3-4
3. Which condition is required for OLS to estimate unique coefficients?
1
No perfect multicollinearity among predictors.
2
Perfect correlation between each predictor and the response.
3
Equal sample sizes in every subgroup.
4
A very large R2 .
4. What happens to ordinary R2 when a new predictor is added to a regression model?
1
It always decreases.
2
It can increase or decrease.
3
It never decreases.
4
It always stays exactly the same.
13 / 14
Answers and take-home messages
MCQ answers: 1 = B, 2 = C, 3 = A, 4 = C.
Take-home messages
Multiple regression extends simple regression by using several predictors at once.
Each coefficient is interpreted conditionally: change in Y for a one-unit change in that predictor,
holding the others fixed.
The core requirements include a sensible linear specification, independent observations or
appropriate dependence modeling, variation in predictors, no perfect multicollinearity, and
E[ε | X] = 0.
High collinearity mainly harms precision and interpretability, not unbiasedness by itself.
Always interpret results together with diagnostics, uncertainty, and substantive context.
14 / 14