Lecture 6: Multiple
Linear Regression II
Dr. Irene Vrbik
University of British Columbia Okanagan
Outline
1. Interaction
The first consideration will involving allowing predictors to
“interact” with one another (related to the marketing term
synergy, in regression Interaction)
2. Categorical Predictors
3. Interaction with Categorical Predictors
Example: Clock Auction
The data give the selling price, Price at
auction of 32 antique grandfather clocks.
Also recorded is the age of the clock (Age) and
the number of people who made a bid
(Bidders).
This data (uploaded to Canvas) is tab delimited
so it gets read into R using read.delim
1 dat <- read.delim("data/clockauction", sep="\t")
Clock Auction Data
Age
<int>
Bidders
<int>
Price
<int>
1
127
13
1235
2
115
12
1080
3
127
7
845
4
150
9
1522
5
156
6
1047
6
182
11
1979
6 rows
3D Scatter plot
Correlation
Another way that you could investigate correlation between
predictors is through the correlation matrix (see ?cor)
1 cor(dat)
Age
Bidders
Price
Age
1.0000000 -0.2537491 0.7302332
Bidders -0.2537491 1.0000000 0.3946404
Price
0.7302332 0.3946404 1.0000000
The reinforces our observation that Price seems to be
linearly related to the Age and Bidders (and that Age and
Bidders don’t have a high correlation).
SLR with Age
1 agelm <- lm(Price~Age)
2 summary(agelm)
...
Call:
lm(formula = Price ~ Age)
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) -191.66
263.89 -0.726
0.473
Age
10.48
1.79
5.854 2.1e-06 ***
--Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Residual standard error: 273 on 30 degrees of freedom
Multiple R-squared: 0.5332,
Adjusted R-squared: 0.5177
F-statistic: 34.27 on 1 and 30 DF, p-value: 2.096e-06
...
SLR with Bidders
1 bidlm <- lm(Price~Bidders)
2 summary(bidlm)
...
Call:
lm(formula = Price ~ Bidders)
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept)
806.40
230.68
3.496 0.00149 **
Bidders
54.64
23.23
2.352 0.02540 *
--Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Residual standard error: 367.2 on 30 degrees of freedom
Multiple R-squared: 0.1557,
Adjusted R-squared: 0.1276
F-statistic: 5.534 on 1 and 30 DF, p-value: 0.0254
...
MLR without Interaction
1 ablm <- lm(Price~Age+Bidders); summary(ablm)
...
Call:
lm(formula = Price ~ Age + Bidders)
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) -1336.7221
173.3561 -7.711 1.67e-08 ***
Age
12.7362
0.9024 14.114 1.60e-14 ***
Bidders
85.8151
8.7058
9.857 9.14e-11 ***
--Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Residual standard error: 133.1 on 29 degrees of freedom
Multiple R-squared: 0.8927,
Adjusted R-squared: 0.8853
F-statistic: 120.7 on 2 and 29 DF, p-value: 8.769e-15
Fitted model:
Price = -1336.72 + Age × 12.74 + Bidders × 85.82
Interpretation of Coefficients
: For a clock with a given amount of Bidders, an increase
of 1 year in the age of the clock is associated with a $12.74
increase in the mean price of the clock.
β1
: For a clock with given age, an increase of 1 Bidder is
associated with a $85.82 increase in the mean selling price.
β2
Attention:
Notice that the specific value of age (resp. bidders) does not
affect the interpretation of β (resp. β )
1
2
Setting Bidders values
Eg. when we have 6 bidders Price is calculated:
= -1336.72 + Age × 12.74 + 6 × 85.82
= -821.8 + Age × 12.74
Eg. when we have 12 bidders Price is calculated:
= -1336.72 + Age × 12.74 + 12 × 85.82
= -306.88 + Age × 12.74
MLR Visualization (no interaction)
with 2 qualitative predictors
3D Visualization (no interaction)
trace 0
A 3D scatterplot of the clock data with the fitted additive MLR
model (no interaction) depicted as a plane. This plane is the
Interaction
Suppose that the affect of Age on Price actually depends on
how many Bidders there are.
Similarly, the affect of Bidders on Price might depend on the
Age of the clock.
We can allow the slope term for a predictor to vary based on
the value(s) of the other predictor(s) by including an
additional interaction term1
1. In marketing, this interaction term is refereed to as a synergy effect
Interaction Model
With 2 predictors
Y = β0 + β1 X1 + β2 X2 + β3 X1 X2 + ϵ
is the response variable at i
Y
Xj
is the j predictor variable
β0
is the true intercept (unknown)
βj
ϵ
th
are the true slopes or regression coefficients (unknown)
is the true error, assumed ϵ
i
∼ N (0, σ
2
)
.
MLR with Interaction
1
~
β1
More generally the slope for X ( ) is a function of X so that
the association between X and Y is no longer constant.
1
2
1
Y = β0 + β1 X1 + β2 X2 + β3 X1 X2
= β0 + β2 X2 + (β1 + β3 X2 )X1
~
~
= β0 + β1 X1
where β̃
1
= (β1 + β3 X2 )
and β̃
0
= (β0 + β2 X2 )
1. A similar argument can be made for multiplier associated with X
.
R syntax
1 # same as lm(Price~ Age + Bidders + Age*Bidders)
2 ilm <- lm(Price ~ Age*Bidders); summary(ilm)
...
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 322.7544
293.3251
1.100 0.28056
Age
0.8733
2.0197
0.432 0.66877
Bidders
-93.4099
29.7077 -3.144 0.00392 **
Age:Bidders
1.2979
0.2110
6.150 1.22e-06 ***
Residual standard error: 88.37 on 28 degrees of freedom
Multiple R-squared: 0.9544,
Adjusted R-squared: 0.9495
F-statistic: 195.2 on 3 and 28 DF, p-value: < 2.2e-16
...
hierarchical principle: if the interaction effect is significant you
should also keep the so-called main effects even if they have
non-significant p-values
Interpretation of Coefficients
Population Model
Price = β + β Age + β Bidders + β Age ⋅ Bidders
0
1
2
3
Fitted Model
Price = 322.8 + 0.9 Age -93.4 Bidders + 1.3(Age ⋅
Bidders)
β1
is the effect of Age when Bidders = 0.
Since a clock needs to have at least 1 bidder to be sold β is
meaningless by itself.
1
Better: the effect of Age is 0.9 + 1.3 ⋅ Bidders
Fitted Model with 6 bidders
̂ + β̂ 𝙰𝚐𝚎 + β̂ 𝙱𝚒𝚍𝚍𝚎𝚛𝚜 + β̂ 𝙰𝚐𝚎 ⋅ 𝙱𝚒𝚍𝚍𝚎𝚛𝚜
𝙿𝚛𝚒𝚌𝚎 = β0
1
2
3
= 322.75 + 0.87𝙰𝚐𝚎 − 93.41𝙱𝚒𝚍𝚍𝚎𝚛𝚜 + 1.3𝙰𝚐𝚎 ⋅ 𝙱𝚒𝚍𝚍𝚎𝚛𝚜
Notice now the multiplier associated with Age now depends
on the value of Bidders. Eg. Bidders = 6, Price is given:
= 322.75 + 0.87𝙰𝚐𝚎 − 93.41(6) + 1.3(𝙰𝚐𝚎 ⋅ 6)
= 322.75 + (0.87 + 1.3 ⋅ 6)𝙰𝚐𝚎 − 93.41(6)
= 322.75 + (8.66)𝙰𝚐𝚎 − 560.46
= −237.71 + (8.66)𝙰𝚐𝚎
Fitted Model with 12 bidders
̂ + β̂ 𝙰𝚐𝚎 + β̂ 𝙱𝚒𝚍𝚍𝚎𝚛𝚜 + β̂ 𝙰𝚐𝚎 ⋅ 𝙱𝚒𝚍𝚍𝚎𝚛𝚜
𝙿𝚛𝚒𝚌𝚎 = β0
1
2
3
= 322.75 + 0.87𝙰𝚐𝚎 − 93.41𝙱𝚒𝚍𝚍𝚎𝚛𝚜 + 1.3𝙰𝚐𝚎 ⋅ 𝙱𝚒𝚍𝚍𝚎𝚛𝚜
Notice now the multiplier associated with Age now depends
on the value of Bidders. Eg. Bidders = 12, Price is given:
= 322.75 + 0.87𝙰𝚐𝚎 − 93.41(12) + 1.3(𝙰𝚐𝚎 ⋅ 12)
= 322.75 + (0.87 + 1.3 ⋅ 12)𝙰𝚐𝚎 − 93.41(12)
= 322.75 + (16.45)𝙰𝚐𝚎 − 1120.92
= −798.16 + (16.45)𝙰𝚐𝚎
3D Visualization (with interaction)
trace 0
A 3D scatterplot of the clock data with the fitted MLR model
with interaction depicted by a (non-flat) surface.
Interaction Model with set Bidders
Categorical Predictors
So far we’ve assumed that our predictors are continuous
valued when we’ve fit a regression.
But there is no real problem if instead we have categorical
(i.e. qualitative) values
Let’s motivate this through an example.
Example body
This data set contains the following on 507 individuals:
21 body dimension measurements (eg. wrist and ankle girth)
Age, Weight (in kg), Height (in cm), and Gender1 .
You can find this in the gclus library as body.
1 library(gclus)
2 data(body); attach(body)
3 dim(body)
[1] 507
25
Let’s regress Weight on some of these predictors.
1. Gender was recorded as a binary variable (1 - male, 0 - female).
>head(body)
Biacrom
<dbl>
Biiliac
<dbl>
Bitro
<dbl>
ChestDp
<dbl>
1
42.9
26.0
31.5
17.7
2
43.7
28.5
33.5
16.9
3
40.1
28.2
33.3
20.9
4
44.3
29.9
34.0
18.4
5
42.5
29.9
34.0
21.5
6
43.3
27.0
31.5
19.6
6 rows | 1-5 of 26 columns
Scatterplot Weight vs Height
1
2
3
4
slr <- lm(Weight~Height)
par(mar = c(4.9, 3.9, 1, 1))
plot(Weight~Height)
abline(slr, col =4, lwd = 3)
# reduce white space around figure
Incorporating Gender
But we know more, eg. gender.
Question:
how do we
incorporate
categorical
variables
into this
model?
MLR with categorical variables
We need to create dummy variables.
A dummy, or indicator variable takes only the value 0 or 1 to
indicate the absence or presence of some categorical effect
that may be expected to shift the outcome.
This requires making one of the possible responses of the
categorical variable the reference (which means it is
assumed true in the base model), and then creating stand-in
(dummy) variables for the non-reference options.
It’s best understood through examples…
body MLR Model
The MLR model with 2 predictors will not look any different:
Y = β0 + β1 X1 + β2 X2
But now, we will consider having mixed data types:
Height (numeric)
Gender (qualitative/categorical)
If a qualitative predictor (also known as a factor) only has two
possible values (AKA levels), then incorporating it into a
regression model is very simple ….
R syntax (for mixed predictor types)
1 mlr <- lm(Weight ~ Height + factor(Gender))
Letting R know that Gender (1 = male, 0 = female) is a factor
is very important.
Failure to do this will result in R treating Gender as a number
(rather than a category).
While coding this up in R is very simple, we need to
understand that under the hood we are creating a dummy
variable for Male and using Female as our reference
variable to be used in our baseline model.
Under the hood
𝚆𝚎𝚒𝚐𝚑𝚝 = β0 + β1 𝙷𝚎𝚒𝚐𝚑𝚝 + β2 𝙼𝚊𝚕𝚎
We now have a dummy variable Male (1 = yes, 0 = no).
If the male, then Male = 1 and our model becomes:
𝚆𝚎𝚒𝚐𝚑𝚝 = β0 + β1 𝙷𝚎𝚒𝚐𝚑𝚝 + β2 (1)
If female, then Male = 0 and our model becomes:
𝚆𝚎𝚒𝚐𝚑𝚝 = β0 + β1 𝙷𝚎𝚒𝚐𝚑𝚝 + β2 (0)
= β0 + β1 𝙷𝚎𝚒𝚐𝚑𝚝
Parallel lines models
𝚆𝚎𝚒𝚐𝚑𝚝 =
β0 + β2 + β1 𝙷𝚎𝚒𝚐𝚑𝚝,
{β
0
+ β1 𝙷𝚎𝚒𝚐𝚑𝚝,
for males
for females
This is sometimes referred to as the parallel lines (or parallel
slopes) model.
An parallel slopes models include one numeric and one
categorical explanatory variable
This model allows for different intercepts but forces a
common slope.
Plot of Parallel Slope Model
Parallel lines fit
1 summary(mlr)
...
lm(formula = Weight ~ Height + factor(Gender))
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept)
-56.94949
9.42444 -6.043 2.95e-09 ***
Height
0.71298
0.05707 12.494 < 2e-16 ***
factor(Gender)1
8.36599
1.07296
7.797 3.66e-14 ***
--Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Residual standard error: 8.802 on 504 degrees of freedom
Multiple R-squared: 0.5668,
Adjusted R-squared: 0.5651
F-statistic: 329.7 on 2 and 504 DF, p-value: < 2.2e-16
...
Renaming levels of factor
The non-reference level is made more obvious once we give
the levels meaningful names.
1 genderFM <- factor(ifelse(Gender==0, "female", "male"))
2 (sum.out <- summary(lm(Weight ~ Height + genderFM)))
...
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) -56.94949
9.42444 -6.043 2.95e-09 ***
Height
0.71298
0.05707 12.494 < 2e-16 ***
genderFMmale
8.36599
1.07296
7.797 3.66e-14 ***
--Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Residual standard error: 8.802 on 504 degrees of freedom
Multiple R-squared: 0.5668,
Adjusted R-squared: 0.5651
F-statistic: 329.7 on 2 and 504 DF, p-value: < 2.2e-16
...
Interpreting the output
This small p-value associated with the “male” dummy
variable indicates that, after controlling for height, there is
strong statistical evidence to suggest a difference in average
weight based on gender.
It is estimated that males will tend to be 8.37 kg heavier than
females of the same height.
N.B. The level selected as the baseline category is arbitrary,
and the final predictions for each group will be the same
regardless of this choice
Overkill?
This may see like overkill seeing as how Gender was
already coded up as 0 for female and 1 for male.
However, imagine if Gender instead was grouped as: male,
female or other.
In this case, we would need pick a reference level and create
two 1 dummy variables to represent other levels for this
categorical variable.
1. we will always need to set up one less dummy than the number of possible options,
i.e. levels, for the category, i.e. factor.
Gender with >2 Levels
Choosing Female1 as our reference level requires us to make a
dummy variable for Male and a dummy variable for Other.
𝚆𝚎𝚒𝚐𝚑𝚝 = β0 + β1 𝙷𝚎𝚒𝚐𝚑𝚝 + β2 𝙼𝚊𝚕𝚎 + β3 𝙾𝚝𝚑𝚎𝚛
We now have two dummy variables
𝙼𝚊𝚕𝚎 =
1
{0
if male
otherwise
𝙾𝚝𝚑𝚎𝚛 =
1
{0
if Other
otherwise
1. We could have just as easily made Male or Other our reference level (here’s how)
When the individual identifies as a male we have Male = 1,
Other = 0 and so our model becomes
𝚆𝚎𝚒𝚐𝚑𝚝 = β0 + β2 + β1 𝙷𝚎𝚒𝚐𝚑𝚝
When the individual identifies as a female (reference level) we
have Male = 0, Other = 0 and so our model becomes
𝚆𝚎𝚒𝚐𝚑𝚝 = β0 + β1 𝙷𝚎𝚒𝚐𝚑𝚝
When the individual identifies as Other we have Male = 0,
Other = 1 and so our model becomes
𝚆𝚎𝚒𝚐𝚑𝚝 = β0 + β3 + β1 𝙷𝚎𝚒𝚐𝚑𝚝
Parallel Lines with >2 Levels
As before, we have parallel lines1 (one line for each level in our
Gender factor)
⎧ β0 + β2 + β1 𝙷𝚎𝚒𝚐𝚑𝚝,
⎪
𝚆𝚎𝚒𝚐𝚑𝚝 = ⎨ β0 + β3 + β1 𝙷𝚎𝚒𝚐𝚑𝚝,
⎪
⎩ β0 + β1 𝙷𝚎𝚒𝚐𝚑𝚝,
for males
for other
for females
β2
describes the shift from Females (our reference) to Males
β3
describes the shift from Females (our reference) to Other
1. Depending on the sign of the coefficient, the parallel line for the non-reference level
may appear above or below the reference level.
Interaction with Mixed
Predictors
We can include interactions with mixed-type predictors as well.
𝚆𝚎𝚒𝚐𝚑𝚝 = β0 + β1 𝙷𝚎𝚒𝚐𝚑𝚝 + β2 𝙼𝚊𝚕𝚎 + β3 (𝙷𝚎𝚒𝚐𝚑𝚝 × 𝙼𝚊𝚕𝚎)
Now for males (Male =1) we have:
Weight = β0 + β1 (Height) + β2 (1)
+ β3 (Height × 1)
= (β0 + β2 ) + (β1 + β3 )(Height)
For female (Male = 0) we have:
Weight = β0 + β1 (Height) + β2 (0)
+ β3 (Height × 0)
= β0 + β1 × Height
We see that they have different intercepts and slopes.
Non-parallel lines
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
# fit the model w/ interaction
intlm <- lm(Weight ~ Height*Gender
# store coeffiecents
icoefs <- intlm$coefficients
# Plot ------par(mar = c(4.9, 3.9, 1, 1))
plot(Weight~Height, col=Gender+1)
legend("topleft",col=c(2,1),pch=1,
legend=c("Male","Female"))
# plot the line for males
abline(a = icoefs[1]+ icoefs[3],
b = icoefs[2] + icoefs[4],
col=2, lwd=2)
# plot the line for female
abline(a = icoefs[1],
b= icoefs[2], lwd=2)
Interaction Model: Height*Gender
1 intlm <- lm(Weight ~ Height*genderFM); summary(intlm)
...
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept)
-43.81929
13.77877 -3.180 0.00156 **
Height
0.63334
0.08351
7.584 1.63e-13 ***
genderFMmale
-17.13407
19.56250 -0.876 0.38152
Height:genderFMmale
0.14923
0.11431
1.305 0.19233
--Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Residual standard error: 8.795 on 503 degrees of freedom
Multiple R-squared: 0.5682,
Adjusted R-squared: 0.5657
F-statistic: 220.7 on 3 and 503 DF, p-value: < 2.2e-16
...
Issues to look out for
1. Non-linear relationships
2. Correlation of error terms
3. Non-constant variance of error terms
4. Outliers/Influential observations (high-leverage points)
5. Collinearity of predictors
Some of these topics will be covered in Lab 3 (read more in
ISLR 3.3.3, Lab 3.6 and [1][2])
Non-linearity Examples
Suppose we have a case where the response has a nonlinear relationship with the predictor(s).
For example, what if there’s a quadratic relationship?
We can extend the linear model in a very simple way to
accommodate non-linear relationships, using polynomial
regression.
Polynomial Regression
Easy. Fit a model of the form
Y = β0 + β1 X + β2 X
2
+ ϵ
Basically, if we square (or otherwise transform) the original
predictor, we can still fit a “linear” model for the response.
In line with the hierarchical principle, if you keep X in your
model, you should also keep X .
2
Though it’s important to keep in mind the change in
interpretation for β and β , for example
1
2
Example: Quadratic Simulation
We simulate 30 values from the following model where ϵ is
standard normally distributed.
Y = 15 + 2.3x − 1.5x
2
+ ϵ
Example: SLR fit
1 summary(linmod)
...
Call:
lm(formula = y ~ x)
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 13.3206
0.3592 37.081 < 2e-16 ***
x
1.7866
0.3248
5.501 7.06e-06 ***
--Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Residual standard error: 1.967 on 28 degrees of freedom
Multiple R-squared: 0.5194,
Adjusted R-squared: 0.5023
F-statistic: 30.26 on 1 and 28 DF, p-value: 7.059e-06
...
1 summary(quadmod)
...
Call:
lm(formula = y ~ x + x2)
Coefficients:
Estimate Std. Error t value
(Intercept) 15.3556
0.2509
61.20
x
2.2773
0.1529
14.89
x2
-1.6702
0.1578 -10.59
--Signif. codes: 0 '***' 0.001 '**' 0.01
Pr(>|t|)
< 2e-16 ***
1.54e-14 ***
4.14e-11 ***
'*' 0.05 '.' 0.1 ' ' 1
Residual standard error: 0.8829 on 27 degrees of freedom
Multiple R-squared: 0.9067,
Adjusted R-squared: 0.8998
F-statistic: 131.2 on 2 and 27 DF, p-value: 1.244e-14
While it is clear from the fitted line that the simple linear
regression (SLR) model is too simple, we can do some
Regression Model Diagnostics to verify.
Diagnostic Plots
1. Residuals vs Fitted: checks for linearity. A “good” plot will
have a red horizontal line, without distinct patterns.
2. Normal Q-Q: checks if residuals are normally distributed. It’s
“good” if points follow the straight dashed line.
3. Scale-Location: checks the equal variance of the residuals.
“Good” to see horizontal line with equally spread points.
4. Residuals vs Leverage: identifies influential points1.
1. extreme values that might influence the regression results when included or excluded
from the analysis.
Example: SLR Residuals
1 plot(linmod)
Example: Quadratic Fit
1 summary(quadmod)
...
Call:
lm(formula = y ~ x + x2)
Coefficients:
Estimate Std. Error t value
(Intercept) 15.3556
0.2509
61.20
x
2.2773
0.1529
14.89
x2
-1.6702
0.1578 -10.59
--Signif. codes: 0 '***' 0.001 '**' 0.01
Pr(>|t|)
< 2e-16 ***
1.54e-14 ***
4.14e-11 ***
'*' 0.05 '.' 0.1 ' ' 1
Residual standard error: 0.8829 on 27 degrees of freedom
Multiple R-squared: 0.9067,
Adjusted R-squared: 0.8998
F-statistic: 131.2 on 2 and 27 DF, p-value: 1.244e-14
Example: Quadratic Residuals
1 plot(quadmod)
Example: Cars
1 library(ISLR)
2 scar <- lm(mpg~horsepower,data=Auto); plot(scar)
1 pcar <- lm(mpg~horsepower+ I(horsepower^2),data=Auto); plot(pcar)
Exemplary Residual Plot
1 xx <- runif(500); eps <- rnorm(500); yy = 8 + 6*xx + eps
2 simfit <- lm(yy~xx); plot(simfit)
Non-parametric model
In this context, lm() assumed a linear functional form for
f (X)
What about a nonparametric alternative?
Let’s return to one of the simplest and best-known nonparametric methods, K-nearest neighbors, but this time lets’
use it for regression rather than classification.
You can probably guess what this method looks like from
based on our first discussions on KNN classification, but let’s
spell it out …
KNN Regression
Given positive integer K (chosen by user) and observation x :
0
1. Identify the K closest points to x in training data. Call this
set N
0
0
2. Estimate f (x
0)
using
̂ x ) =
f (
0
1
K ∑
yi
i∈N0
See Ch 7 of Campbell, T., Timbers, T., Lee, M. (2022). Data
Science: A First Introduction. United States: CRC Press.
K = 20
Visualization 1
Visualization 2
K=5
K = 5 Visualization
K=1
Conclusion
Small values of K are more flexible
produce low bias but high variance
At K = 1 the prediction in a given region is entirely
dependent on just one observation.
In contrast, large values of K are “smoother” with less steps
less variance (changing one observation has a smaller effect)
more bias (smoothing masks some of the structure in f (X))
Again we see the bias-variance trade off in action!