ENGINEERING DIVISION | NYU ABU DHABI
ENGR-UH 3332
Applied Machine Learning
Mini Project 1 – Linear & Logistic Regression
Due Date: Refer to Brightspace
ENGINEERING DIVISION | NYU ABU DHABI
Introduction
Linear regression was the first type of regression analysis to be studied rigorously, and to be
used extensively in many applications. This is because models which depend linearly on their
unknown parameters are easier to fit than models which are non-linearly related to their
parameters and because the statistical properties of the resulting estimators are easier to
determine.
Logistic regression is a classification algorithm used to assign observations to a discrete set of
classes. Unlike linear regression which outputs continuous number values, logistic regression
transforms its output using the logistic sigmoid function to return a probability value which can
then be mapped to two or more discrete classes.
This project contains two parts:
Part 1: Linear Regression
Part 2: Logistic Regression
Part A: Linear Regression (50 pts)
Dataset
In this project, we are going to use “Boston house prices dataset” from Scikit Learn. The
Boston house-price data has been used in many machine learning papers that address
regression problems. Learn more at
https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_boston.html
Boston house prices dataset has 506 instances and for each instance, it has 13 attributesand
one target value.
● CRIM per capita crime rate by town
● ZN proportion of residential land zoned for lots over 25,000 sq.ft.
● INDUS proportion of non-retail business acres per town
● CHAS Charles River dummy variable (= 1 if tract bounds river; 0
otherwise)
● NOX nitric oxides concentration (parts per 10 million)
● RM average number of rooms per dwelling
● AGE proportion of owner-occupied units built prior to 1940
● DIS weighted distances to five Boston employment centres
● RAD index of accessibility to radial highways
● TAX full-value property-tax rate per $10,000
● PTRATIO pupil-teacher ratio by town
● B 1000(Bk - 0.63)^2 where Bk is the proportion of blacks by town
● LSTAT % lower status of the population
● MEDV Median value of owner-occupied homes in $1000’s
ENGINEERING DIVISION | NYU ABU DHABI
Requirements
1. Explore and import Boston house prices dataset:
a. Explore the dataset by using the Scikit Learn library and Numpy.
2. In this problem you will experiment with a linear regression problem based on realworld data. The data is from the Boston Housing dataset in scikit-learn. Your task isto
estimate the price of a house in Boston using 13 attributes.
3. Fit a linear regression model using the closed-form solution presented in class. Use
k-fold cross-validation to estimate the performance of this model. Print the average of
your recorded scores for both the test set and training set.
a. Only Numpy library is allowed. (you may use external API to extract samples for k
folds)
4. Fit a ridge regression model using the closed solution. Use k-fold cross-validation to find
[
].
the best λ∈ 101, 101.5, 102, 102.5, …, 107
a. Use the Numpy function: np.logspace(1, 7, num=13) to get the different
values for λ .
5. For the best λ you found, use k-fold cross-validation to estimate the performance of this
model with this λ . Print the average of your recorded scores for both the test set and
training set.
6. Repeat the previous exercise, but this time, by creating a polynomial transformation
of degree 2 on the features of the dataset.
a. Using PolynomialFeatures(degree=2) in sklearn library
7. Repeat Multivariate Linear Regression using the Gradient Descent method.
8. Implement Lasso Regression (Text book Chapter 4)
a. Cost function of Lasso Regression:
𝑛
𝐽(𝜃) = 𝑀𝑆𝐸(𝜃) + 𝛼 ∑
𝑖=1
|𝜃𝑖 |
9. Implement Elastic Net (Text book Chapter 4)
a. Cost Function of Elastic Net:
𝑛
𝐽(𝜃) = 𝑀𝑆𝐸(𝜃) + 𝑟𝛼 ∑
|𝜃𝑖 | +
𝑖=1
𝑛
1−𝑟
𝛼 ∑ 𝜃2
2
𝑖=1
10. If you are given a choice of predicting future housing prices using one of the models
you have learned above (those optimized with gradient descent), which one would
you choose and why? State the parameters of that model.
ENGINEERING DIVISION | NYU ABU DHABI
Part B: Logistic Regression (50 pts)
Dataset
In the Logistic Regression assignment, you will use the “Breast Cancer Wisconsin dataset” which
you can import from Scikit Learn. The breast cancer dataset is classic and very easy binary
classification dataset has 569 instances and 30 numeric predictive attributes.
Attribute information:
● radius (mean of distances from center to points on the perimeter)
● texture (standard deviation of gray-scale values)
● perimeter area
● smoothness (local variation in radius lengths)
● compactness (perimeter^2 / area - 1.0)
● concavity (severity of concave portions of the contour)
● concave points (number of concave portions of the contour)
● symmetry fractal dimension (“coastline approximation” - 1)
The mean, standard error, and “worst” or largest (mean of the three largest values) of these
features were computed for each image, resulting in 30 features. For instance, field 3 is Mean
Radius, field 13 is Radius SE, field 23 is Worst Radius.
Requirements
1.
Explore and import Breast Cancer Wisconsin dataset:
a) Explore the dataset by using the Scikit Learn library and Numpy.
2.
In the assignment, you will use gradient descent to find the weights for the logistic
regression problem and apply it to the dataset.
3.
The initial hyper-parameters for this assignment are:
a) Threshold=0.5
b) Learning rate=0.5
c) Run your algorithm for 5000 iterations to update weights
4.
Report the coefficient vector w
5.
For the test dataset, determine the
a) Precision
b) Recall
c) F1 score
d) Confusion matrix.
6.
Plot the log loss on every 100th iteration of your gradient descent, with the iteration
number on the horizontal axis and the objective value on the vertical axis
7.
Use the test set as a validation set and see if you can find a better setting of the
hyperparameters. Report the best values you found.
ENGINEERING DIVISION | NYU ABU DHABI
Deliverables
A zip file containing the following:
1. a working project (source code, makefiles if needed, etc)
2. a report for the detailed description of the project
a. Instructions on how to run your project
b. Answers to the programming questions.
Before submitting your project, please make sure to test your program on the given dataset.
Notes
You may discuss the general concepts in this project with other students, but you must implement
the program on your own. No sharing of code or report is allowed. Violation of this policy can
result in a grade penalty.
Late submission is acceptable with the following penalty policy:
10 points deduction for every day after the deadline