3/19/2021
Final Project
MSDS 430 Final Project
Complete the following and submit your notebook, html to Canvas. Your completed notebook should include all output, i.e. run each cell and save your file before
submitting.
In this final project you will continue working with the building prices dataset. You have already run Python code to fix the bad data detected in Milestone 1.
In Milestone 2 you selected a sngle variable that you think is best for predicting sale price. Use your single variable from Milestone 2 to complete this assignment. You can change your mind but tell me what you
are doing and why.</font>
The code provided below uses age_home for the input variable. This is just so the code will run. Replace age_home with the name of the variable you selected in Milestone 2.
Section 1 Import and Explore
Start by rebuilding the dataframes you used in Milestone 1 and 2.
Changing single variable from age_home to taxes due to the higher correlation coefficients we identified in Milestone 2
In [13]:
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import statsmodels.formula.api as smf
import seaborn as sns
sns.set(style="ticks", palette="bright")
In [14]:
df = pd.read_csv('building prices 2018-1.csv')
df.columns = [s.lower() for s in df.columns]
df.head()
price
taxes
bathrooms
lot_size
living_space
num_garage
num_rooms
num_bedrms
age_home
num_fplaces
0
25.9
4.918
1.0
3.472
0.998
1.0
7
4
42
0
1
29.5
5.021
1.0
3.531
1.500
2.0
7
4
62
0
2
27.9
4.543
1.0
2.275
1.175
1.0
6
3
40
0
3
25.9
4.557
1.0
4.050
1.232
1.0
6
3
54
0
4
29.9
5.060
1.0
0.000
1.121
1.0
6
3
0
0
Out[14]:
In [15]:
def scat(dataframe,var1,var2):
plt.scatter(dataframe[var1],dataframe[var2], s = 30)
plt.title('Scatter')
plt.xlabel(var1)
plt.ylabel(var2)
scat(df, 'taxes', 'price')
Section 2 Base Model and Residual Mean Square
Run a base model and calculate the residual mean square for the model. Kaggle is set up to score your models using a different calculation called root mean square error (RMSE). Using residual mean square here
because statsmodels least squares calculates it for you.
Using the statsmodels API for this linear model and age_home for the single prediction variable. age_home will be used for the single prediction variable for each of the sections where the change will be
improvement in the data due to the EDA. Result will be model residual mean square improvement due to the EDA rather than changes to the input variable.
In [16]:
base_model = smf.ols(formula='price ~ taxes', data=df).fit()
predictions = base_model.fittedvalues
predictions.head()
33.003663
Out[16]: 0
1
33.163371
2
32.422201
3
32.443909
4
33.223843
dtype: float64
Join the actual price with the model predicted price from the model so you can look at a plot later.
In [17]:
d = {'p_price': predictions}
d2=pd.DataFrame(data=d)
base_join = pd.concat([df,d2],axis = 1).reindex(df.index)
Note, statsmodels has a feature that calculates some very useful things for you regarding your model. Use dir(base_model) to see what these are.
In [18]:
dir(base_model)
Out[18]: ['HC0_se',
'HC1_se',
'HC2_se',
'HC3_se',
'_HCCM',
'__class__',
'__delattr__',
'__dict__',
'__dir__',
'__doc__',
'__eq__',
'__format__',
This study source
was downloaded
'__ge__',
by 100000824742498 from CourseHero.com on 02-22-2022 19:25:36 GMT -06:00
file:///E:/Documents/Downloads/Final Project.html
https://www.coursehero.com/file/84994658/Final-Projectpdf/
1/4
3/19/2021
Final Project
'__getattribute__',
'__gt__',
'__hash__',
'__init__',
'__init_subclass__',
'__le__',
'__lt__',
'__module__',
'__ne__',
'__new__',
'__reduce__',
'__reduce_ex__',
'__repr__',
'__setattr__',
'__sizeof__',
'__str__',
'__subclasshook__',
'__weakref__',
'_abat_diagonal',
'_cache',
'_data_attr',
'_data_in_cache',
'_get_robustcov_results',
'_is_nested',
'_use_t',
'_wexog_singular_values',
'aic',
'bic',
'bse',
'centered_tss',
'compare_f_test',
'compare_lm_test',
'compare_lr_test',
'condition_number',
'conf_int',
'conf_int_el',
'cov_HC0',
'cov_HC1',
'cov_HC2',
'cov_HC3',
'cov_kwds',
'cov_params',
'cov_type',
'df_model',
'df_resid',
'eigenvals',
'el_test',
'ess',
'f_pvalue',
'f_test',
'fittedvalues',
'fvalue',
'get_influence',
'get_prediction',
'get_robustcov_results',
'initialize',
'k_constant',
'llf',
'load',
'model',
'mse_model',
'mse_resid',
'mse_total',
'nobs',
'normalized_cov_params',
'outlier_test',
'params',
'predict',
'pvalues',
'remove_data',
'resid',
'resid_pearson',
'rsquared',
'rsquared_adj',
'save',
'scale',
'ssr',
'summary',
'summary2',
't_test',
't_test_pairwise',
'tvalues',
'uncentered_tss',
'use_t',
'wald_test',
'wald_test_terms',
'wresid']
Find the residual mean square and r squared for this model so you can use it to compare with your other models.
In [19]:
base_mse = base_model.mse_resid
base_mse = np.around([base_mse], decimals = 2)
base_mse
Out[19]: array([23.16])
In [20]:
base_rsquared = np.around([base_model.rsquared], decimals = 2)
base_rsquared
Out[20]: array([0.35])
Look at a scatter plot of price vs p_price to see if this model looks reasonable (hope your model is better than my example).
In [21]:
scat(base_join, 'price', 'p_price')
This study source was downloaded by 100000824742498 from CourseHero.com on 02-22-2022 19:25:36 GMT -06:00
file:///E:/Documents/Downloads/Final Project.html
https://www.coursehero.com/file/84994658/Final-Projectpdf/
2/4
3/19/2021
Final Project
Section 3 Residual Mean Square (after removing duplicate records)
In [22]:
df1 = df.drop_duplicates(keep='first')
print('Number of Records:', len(df1))
Number of Records: 24
Model the data with the nodups dataset df1:
In [23]:
nodups_model = smf.ols(formula='price ~ taxes', data=df1).fit()
nodups_preds = nodups_model.fittedvalues
nodups_preds.head()
33.046073
Out[23]: 0
1
33.197726
2
32.493937
3
32.514551
4
33.255148
dtype: float64
Calculate the NODUPS residual mean square and r squared.:
In [24]:
d = {'p_price': nodups_preds}
d2=pd.DataFrame(data=d)
nodups_join = pd.concat([df,d2],axis = 1).reindex(df.index)
In [25]:
nodups_mse = nodups_model.mse_resid
nodups_mse = np.around([nodups_mse], decimals = 2)
nodups_mse
Out[25]: array([25.08])
In [26]:
nodups_rsquared = np.around([nodups_model.rsquared], decimals = 2)
nodups_rsquared
Out[26]: array([0.34])
In [27]:
scat(nodups_join, 'price', 'p_price')
Section 4 Residual Mean Square & R Squared (after EDA, fixing outliers, and missing data)
In [28]:
df2 = df1.copy()
df2=df2.replace({'living_space': {9999: 0}})
df2=df2.replace({'age_home': {9999: 0}})
m = np.median(df2.taxes[df2.taxes>0])
df2=df2.replace({'taxes': {0: m}})
n = np.median(df2.lot_size[df2.lot_size>0])
df2=df2.replace({'lot_size': {0: n}})
p = np.median(df2.living_space[df2.living_space>0])
df2=df2.replace({'living_space': {0: p}})
q = np.median(df2.age_home[df2.age_home>0])
df2=df2.replace({'age_home': {0: q}})
Use df2 with outlier and missing data fixes in place.
In [34]:
taxes_model = smf.ols(formula='price ~ taxes', data=df2).fit()
taxes_model.summary()
taxes_preds = age_home_model.fittedvalues
taxes_preds.head()
29.044122
Out[34]: 0
1
29.408913
2
27.716001
3
27.765584
4
29.547037
dtype: float64
Calculate taxes_mse for the nodups, no outliers and no missing values data df2.
In [35]:
d = {'p_price': taxes_preds}
d2=pd.DataFrame(data=d)
taxes_join = pd.concat([df,d2],axis = 1).reindex(df.index)
In [36]:
taxes_mse = taxes_model.mse_resid
taxes_mse = np.around([taxes_mse], decimals = 2)
taxes_mse
Out[36]: array([8.62])
In [37]:
taxes_rsquared = np.around([taxes_model.rsquared], decimals = 2)
taxes_rsquared
Out[37]: array([0.77])
In [38]:
scat(taxes_join, 'price', 'p_price')
# you hope this will look like a line group.
#
Your plot should look better than mine
This study source was downloaded by 100000824742498 from CourseHero.com on 02-22-2022 19:25:36 GMT -06:00
file:///E:/Documents/Downloads/Final Project.html
https://www.coursehero.com/file/84994658/Final-Projectpdf/
3/4
3/19/2021
Final Project
In your Python career you will see instances where others have offered a predictive model without
doing any EDA and no data fixes. You know how to fix this. Be kind in sharing your skill.
Summarize the Residual Mean Square and r squared for each of the models.
In [39]:
models_df = ({'Base Model':base_mse,
'nodups Model':nodups_mse,
'taxes Model':taxes_mse})
models = pd.Series(models_df)
models
Out[39]: Base Model
nodups Model
taxes Model
dtype: object
In [40]:
[23.16]
[25.08]
[8.62]
models_df = ({'Base Model':base_rsquared,
'nodups Model':nodups_rsquared,
'taxes Model':taxes_rsquared})
models = pd.Series(models_df)
models
Out[40]: Base Model
nodups Model
taxes Model
dtype: object
[0.35]
[0.34]
[0.77]
Bonus Points
Submit your model to Kaggle.com for scoring. Use the code below to create the csv file for Kaggle.
You may have to add the column variable names to the csv file to submit to kaggle. Kaggle expects
two columns labeled index and predict. Use Excel to change the column names.
In [ ]:
age_home_preds.to_csv('c:/data/predictions.csv')
Everything below is optional
age_home model with removal of duplicate records and fixes for outliers and missing data show major improvement for the simple single variable model. Still more you can do by including other variables.
One more look at a pairplot for price vs p_price with the single variable model:
In [ ]:
sns.set(style="ticks", palette="bright")
sns.pairplot(x_vars=["price"], y_vars=["p_price"], data=age_home_join, height = 6)
In [ ]:
import seaborn as sns
sns.set(style="ticks", palette="bright")
sns.pairplot(x_vars=["age_home"], y_vars=["price"], data=df2, hue="bathrooms",
palette=["g", "b"],height = 6)
Bathrooms may be a good addition to the model. Maybe something else too. You decide.
This study source was downloaded by 100000824742498 from CourseHero.com on 02-22-2022 19:25:36 GMT -06:00
file:///E:/Documents/Downloads/Final Project.html
https://www.coursehero.com/file/84994658/Final-Projectpdf/
Powered by TCPDF (www.tcpdf.org)
4/4