Machine
Learning
S10 (Neurons)
ENGR. MARK MANLAPAZ
A Simple Model of a Neuron
The Brain
➢Made of millions of neurons
➢Biological neural network
➢Each neuron performs a very modest cognitive function
➢Combination of these function results in the brain’s mental ability
Neurons
A Simple Model of a Neuron
How a Neuron Classifies
How a Neuron Classifies
Bias Inputs
➢A weighted sum is a linear function
➢It has a weight assigned to each input
➢An independent input is needed to move the line vertically
➢This bias input is always set to 1
➢The bias has its own weight
Bias Term
Bias Term
Bias Term
Refined Model of a Neuron
Weighted Sums
Values aren’t constrained, so a sum may result in:
➢A very large value
➢A very small value
It’s a linear function:
➢The threshold to “fire” is not very well modeled
➢It’s not easily trained
Weighted Sums
Linear Function
Activation Function
Activation Functions
➢ Model the desired threshold behavior
➢Constraint output values
➢Provide nonlinearity to the neuron
➢Enable training by backpropagation
Binary Step Function
➢Limits output between 0 and 1
➢Outputs are exactly 0 or 1
Logistic or Sigmoid Function
➢Limits output between 0 and 1
➢Outputs are real numbers between 0 or 1
Hyperbolic Tangent Function
➢Limits output between -1 and 1
➢Outputs are real numbers between -1 and 1
Rectified Linear Unit Function (ReLU)
➢Limits outputs to positive values
➢It is unbounded for positive values