Uploaded by Aman Patil


Define Bias and Threshold?What is learning rate paramenter?
Neural Network is conceptually based on actual neuron of brain. Neurons are the basic units of a large
neural network. A single neuron passes single forward based on input provided.
In Neural network, some inputs are provided to an artificial neuron, and with each input a weight is
associated. Weight increases the steepness of activation function. This means weight decide how fast the
activation function will trigger whereas bias is used to delay the triggering of the activation function.
For a typical neuron, if the inputs are x1, x2, and x3, then the synaptic weights to be applied to them are
denoted as w1, w2, and w3.
Equation is y = f(x) = Σxiwi
where i is 1 to the number of inputs.
The weight shows the effectiveness of a particular input. More the weight of input, more it will have
impact on network.
On the other hand Bias is like the intercept added in a linear equation. It is an additional parameter in
the Neural Network which is used to adjust the output along with the weighted sum of the inputs to the
neuron. Therefore Bias is a constant which helps the model in a way that it can fit best for the given data.
The processing done by a neuron is thus denoted as :
output = sum (weights * inputs) + bias
A threshold transfer function is sometimes used to quantify the output of a neuron in the output layer.
Feed-forward networks include Perceptron (linear and non-linear) and Radial Basis Function networks.
Feed-forward networks are often used in data mining.
A feed-back network (e.g., recurrent neural network or RNN) has feed-back paths meaning they can have
signals traveling in both directions using loops. All possible connections between neurons are allowed.
Since loops are present in this type of network, it becomes a non-linear dynamic system which changes
continuously until it reaches a state of equilibrium. Feed-back networks are often used in associative
memories and optimization problems where the network looks for the best arrangement of
interconnected factors.
Learning rate Parameter
In machine learning and statistics, the learning rate is a tuning parameter in an optimization
algorithm that determines the step size at each iteration while moving toward a minimum of a
loss function. Since it influences to what extent newly acquired information overrides old
information, it metaphorically represents the speed at which a machine learning model
"learns". In the adaptive control literature, the learning rate is commonly referred to as gain.
In setting a learning rate, there is a trade-off between the rate of convergence and
overshooting. While the descent direction is usually determined from the gradient of the loss
function, the learning rate determines how big a step is taken in that direction.A too high
learning rate will make the learning jump over minima but a too low learning rate will either
take too long to converge or get stuck in an undesirable local minimum.
In order to achieve faster convergence, prevent oscillations and getting stuck in undesirable
local minima the learning rate is often varied during training either in accordance to a learning
rate schedule or by using an adaptive learning rate. The learning rate and its adjustments may
also differ per parameter, in which case it is a diagonal matrix that can be interpreted as an
approximation to the inverse of the Hessian matrix in Newton's method. The learning rate is
related to the step length determined by inexact line search in quasi-Newton methods and
related optimization algorithms.