TASK - 2 (SUBTASK - 3B (LOGISTIC REGRESSION))
Introduction This report is about the subtask-3B of Task-2 , creating a logistic regression
model to predict whether a certain user purchases a certain commodity from a
social network ad , using parameters like Age , and EstimatedSalary . I have trained the model using
gradient descent, plotted the binary cross‐entropy loss over the number of epochs , and assessed
performance on a held‐out test set via accuracy, precision, recall, and F1‐score. 80% of data has
been used as training data , and 20% has been used for testing .
Approach and Implementation To start , we apply plain linear regression on the data This is the expression for plain
linear regression , now the problem
with this is there is no bound for ‘z’ ,
thus we apply the sigmoid function
to squish the output , into a range
of values between 0 and 1 , so that
it can be compared with supplied
values of “Purchased”.
Code - snippet shows implementation of the sigmoid function . The 2
clauses are done separately to ensure the stability of the parameters so
that the sigmoid doesn’t by default assign x to either 0 or inf.
This code - snippet implements gradient descent:
sw1 and sw2 accumulate the partial derivatives of
the loss w.r.t. each parameter, and the commented
lines update each parameter by subtracting its
gradient times the learning rate L.
So , after the above - mentioned steps , we go on to
implement the Binary-Cross Entropy Loss Function
2
, and plot the loss against the number of iterations . To estimate the efficiency of the model , we
also estimate Accuracy, Precision, Recall and F1 - score .
Plots of the Loss-Curves , Discussion of Improvement This is the binary - cross - entropy loss value obtained for
the logistic regression model . The binary - cross - entropy
loss value converges to “0.34738965430767876” for
50,000 iterations , with a learning rate of 0.1 . There is a
reason for which I’ve chosen such a high learning rate , its
mentioned in detail in the “challenges” section . We
observe , the loss decrease over the 1st 100 epochs is
0.04906286403357918 , whereas , for the last 100
epochs is 6.349020856666332e-06.
So , it is evident that the loss function’s value initially changes rapidly and decreases to finally attain
a very small rate of change .
Accuracy, Precision, Recall and F1 - score -
The predlist() function evaluates these evaluation metrics , and the code for that is above .
3
Observations , Challenges and Improvements 1.)​ A point to be noticed is the weird definition of the sigmoid function . It is done so ,
because the np.log() would return Nan on values <= e-15 , which was occurring for this
data set . Hence , the parameters have been scaled(age has been reduced to 0.01 ,
whereas salary has been reduced to 1/100,000 .
2.)​ To modify the above problem , I had also tried introducing an epsilon = 1e-15 , so that the
sigmoid would never pass a 0 to the np.log() function . But , even that failed, as it led to a
highly unstable model , whose binary-cross entropy was coming as a graph just
oscillating between the high and low bounds in an uneven manner .
3.)​ A possible improvement is to explore polynomial feature expansion (e.g., Age², Salary²)
to capture non‐linear boundaries for the data , that is , we can attempt to apply logistic
regression on a quadratic model for more accuracy.
4.)​ Key - Observation : Feature scaling proved essential: without standardization, gradient
descent failed to converge , and was just oscillating .