Single and Multilayer Feed Forward Neural Networks
Feedforward Networks
2
Neuron
3
Neuron
4
Common Activation Functions
5
Two-layer Neural Networks
Two-layer neural networks model
linear classifiers.
However, many real-world
problems are non-linear!
6
Classification with Neural Networks
• Classification: predicting a discrete-valued target
• In this lecture, we focus on binary classification: predicting a
binary-valued target
• Examples
– predict whether a patient has a disease, given the presence or
absence of various symptoms
– classify e-mails as spam or non-spam
– predict whether a financial transaction is fraudulent
Binary linear classification
• classification: predict a discrete-valued target
• binary: predict a binary target t ∈ {0, 1}
– Training examples with t = 1 are called positive examples, and
training examples with t = 0 are called negative examples.
• linear: model is a linear function of x, followed by a threshold:
Some simplifications
• Eliminating the threshold
– We can assume WLOG that the threshold r = 0:
Some simplifications
• Eliminating the threshold
– We can assume WLOG that the threshold r = 0:
• Eliminating the bias
– Add a dummy feature x0 which always takes the value 1. The weight w0
is equivalent to a bias.
Some simplifications
• Eliminating the threshold
– We can assume WLOG that the threshold r = 0:
• Eliminating the bias
– Add a dummy feature x0 which always takes the value 1. The weight w0
is equivalent to a bias.
• Simplified Model
As a neuron
• This is basically a special case of the neuron-like processing
unit
Examples for representing simple functions with linear
classifiers
• Example1:
b>0
b+w<0
As an example, we select b = 1, w = −2
Example2: AND
b<0
b + w2 < 0
b + w1 < 0
b + w1 + w2 > 0
Example: b = −1.5, w1 = 1, w2 = 1
The geometric picture
•
Input space or Data space
•
•
•
•
Here we’re visualizing the NOT example
Training examples are points
Hypotheses are half-spaces whose boundaries pass through the origin
The boundary is the decision boundary
– In 2-D, it’s a line, but think of it as a hyperplane
•
•
If the training examples can be separated by a linear decision rule, they are linearly
separable.
The AND example requires three dimensions (w1,w2,b).
The Perceptron Learning Rule
• Let’s mention a classic classification algorithm from the
1950s: the perceptron
•
- Frank Rosenblatt, with the image sensor (left) of the Mark I Perceptron40
The Perceptron Learning Rule
• The idea:
– If t = 1 and z = wTx > 0
• then y = 1, so no need to change anything.
The Perceptron Learning Rule
• The idea:
– If t = 1 and z = wTx > 0
• then y = 1, so no need to change anything.
– If t = 1 and z < 0
• then y = 0, so we want to make z larger.
• we’d like z’ = w’T x > wT x = z
The Perceptron Learning Rule
• The idea:
– If t = 1 and z = wTx > 0
• then y = 1, so no need to change anything.
– If t = 1 and z < 0
• then y = 0, so we want to make z larger.
• we’d like z’ = w’T x > wT x = z
• Update:
– w’ ← w + x
The Perceptron Learning Rule
• The idea:
– If t = 1 and z = wTx > 0
• then y = 1, so no need to change anything.
– If t = 1 and z < 0
• then y = 0, so we want to make z larger.
• we’d like z’ = w’T x > wT x = z
• Update:
w’ ← w + x
• Justification:
w’Tx = (w + x)Tx
= w Tx + x Tx
= wTx + ||x||2.
The Perceptron Learning Rule
For convenience, let targets be {−1, 1} instead of our usual {0, 1}.
Perceptron Learning Rule:
Repeat:
For each training case (x(i), t(i)),
z(i) ← wT x(i)
If z(i)t(i) ≤ 0,
w ← w + t(i)x(i)
Stop if the weights were not updated in this epoch.
The Perceptron Learning Rule
• Under certain conditions, if the problem is feasible, the
perceptron rule is guaranteed to find a feasible solution after a
finite number of steps.
• If the problem is infeasible, all bets are off.
– Stay tuned. . .
• The perceptron algorithm caused lots of hope in the 1950s,
then people got disillusioned and gave up on neural nets.
• People were discouraged about fundamental limitations of
linear classifiers.
Limits of Linear Classification
• Why XOR is not linearly separable?
Positive cases (different):
(0,1) -> 1; (1,0) -> 1
Negative cases (same):
(1,1) -> 0; (0,0) -> 0
x1
0
0
1
1
x2
0
1
0
1
t
0
1
1
0
The four input-output pairs give four inequalities that are impossible to satisfy:
b < 0,
w2 + b > 0,
w1 + b > 0,
w1 + w2 + b < 0
Add 1st and 4th, and 2nd and 3rd to see that it is impossible to satisfy the inequalities.
The geometric picture
• XOR is not linearly separable: Positive and negative
cases cannot be separated by plane.
Learning with hidden units
• Networks without hidden units are very limited in the input-output
mappings they can learn to model.
– More layers of linear units do not help. It's still linear.
• We need multiple layers of adaptive, non-linear hidden units. But
how can we train such nets?
– We need an efficient way of adapting all the weights, not just the last layer.
– Learning the weights going into hidden units is equivalent to learning
features.
– This is difficult because nobody is telling us directly what the hidden units
should do.
LET’S GET MULTI-LAYER
Multi-layer Networks
Information flow is unidirectional
Data is presented to Input layer
Passed on to Hidden Layer
Passed on to Output layer
Information is distributed
Information processing is parallel
Internal representation (interpretation) of data
Multi-layer Networks
• We can connect lots of units
together into a directed
acyclic graph (DAG).
• This gives a feed-forward
neural network. That’s in
contrast to recurrent neural
networks, which can have
cycles.
• Typically, units are grouped
together into layers.
Multi-layer Perceptrons
•
•
Each layer connects N input units to M output units.
•
•
Note: the inputs and outputs for a layer are distinct from the inputs and outputs to the network.
•
A multilayer network consisting of fully connected layers is called a multilayer perceptron.
In the simplest case, all input units are connected to all output units. We call this a fully connected
layer.
The output units are a function of the input units:
y = f (x) = φ (Wx + b)
Multi-layer Perceptrons
• Designing a network to compute XOR:
• Assume hard threshold activation function (threshold at 0)
Multi-layer Perceptrons
Designing a network to compute XOR:
Minsky & Papert (1969) offered solution to XOR problem by combining perceptron unit
responses using a second layer of Units. Piecewise linear classification using an MLP with
threshold (perceptron) units
1
3
2
What do each of the layers do?
2nd layer combines the boundaries
1st layer draws linear
boundaries
Next layers can generate arbitrarily complex boundaries
With enough hidden layers, the network is able to
approximate any mapping from the network’s input to
the expected output.
Properties of the MLP Architecture
• No connections within a layer
• No direct connections between input and output layers
• Fully connected between layers
•Often more than 3 layers
•Number of output units need not equal number of input units
•Number of hidden units per layer can be more or less than input or output units
•Different layers may have different activation functions
Often include bias as an extra weight
Conceptually: Forward Activity Backward Error
Forward Propagation of Activity
(Steps 1-2)
• Step 1: Initialise weights at random, choose a learning rate α
• Until network is trained:
• For each training example i.e. input pattern and target output(s):
• Step 2: Do forward pass through net (with fixed weights) to
produce output(s)
– i.e., in Forward Direction, layer by layer:
• Inputs applied
• Multiplied by weights
• Summed
• ‘Squashed’ by an activation function, e.g., sigmoid
• Output passed to each neuron in next layer
– Repeat above until network output(s) produced
Back-propagation of error
(Steps 3-4)
Deep Learning / Advanced Deep Learning
Training
Sample labeled
data
(batch)
Forward it
through the
network, get
predictions
Backpropagate the
errors
Update the
network
weights
Use error signal to change the weights and get more accurate predictions
Subtracting a fraction of the gradient moves you towards the (local)
minimum of the cost function
40
Expressive Power
• Multilayer feed-forward neural nets with nonlinear activation
functions are universal approximators: they can
approximate any function arbitrarily well.
• The more hidden layers you stack together, the more
complex functions you will be able to model.
Expressive Power Example: Universality for
binary inputs and targets
• We’re given a function mapping input vectors to
outputs; we will need to produce a neural network,
which matches that function.
• The function can be given to us as a table which lists
the output corresponding to every possible input
vector.
* Inputs are -1/+1 as opposed
to 0/1
* There are 2D rows
Expressive Power Example: Universality for
binary inputs and targets
• Strategy: 2D hidden units, each of which responds to one
particular input configuration
• Hard threshold at each hidden units
• Only requires one hidden layer, though it needs to be extremely
wide (2D units)!
Limits of Universality
• Universality is a neat property, but it has major catches:
– It is not compact !
– If you can learn any function, you’ll just overfit.
• We desire compact representations for two reasons:
1.
2.
Computing predictions in a reasonable amount of time.
Training a network to generalize from a limited number of training
examples (no memorization)
Learning
Design an objective function (or loss function) that is • Minimized when the model does what you want it to do.
• Easy to minimize (smooth, well-behaved).
Here we want y = wx + b to be close to t, for every training case.
Learning
Design an objective function (or loss function) that is • Minimized when the model does what you want it to do.
• Easy to minimize (smooth, well-behaved).
Here we want y = wx + b to be close to t, for every training case.
• Loss function: squared error (or, called squared loss)
• y − t is the residual, and we want to make this small in
magnitude
• The factor is just to make the calculations convenient.
Learning
• Cost function: loss function averaged over all training
examples
• Need to find w, b such that ɛ(w, b) is minimized.
Inference
• To make a prediction about a new player, just use y = wx +b
Solving the Optimization Problem
• 3 methods:
1. Random search
2. Follow the slope with analytic gradient
3. Gradient descent
1. Random Search
• A first very bad idea !
• No guarantee to reach global min.
• # of search iterations can be huge
• Setting random weights on a Cifar-10 classification problem
with 10 classes yields 15% accuracy (SOTA is > 98%)!
2. Numerical Gradient
• Use calculus to compute the analytic gradient
2. Follow the slope with analytic gradient
Since ɛ is a nonnegative quadratic function in w and b, any critical point
is a minimum. Therefore, we minimize ɛ by setting
2. Follow the slope with analytic gradient
• Now we have 2 linear equations and 2 unknowns w and b. Solve!
• This method is known as direct solution with analytic gradient
3. Gradient Descent
Now let’s consider another way to minimize the cost function which is more broadly
applicable: gradient descent.
3. Gradient Descent
• Gradient descent is an iterative algorithm, which means we
apply an update repeatedly until some criterion is met.
• We initialize the weights to something reasonable and
repeatedly adjust them in the direction of steepest descent.
Gradient Descent
• Observe:
• The following update decreases the cost function:
• α is a learning rate. The larger it is, the faster w changes.
Values of α are typically small, e.g. 0.01 or 0.0001
Gradient Descent
• Why gradient descent (GD), if we can find the optimum
directly?
– GD can be applied to a much broader set of models
– GD can be easier to implement than direct solutions, especially with
automatic differentiation software
– For regression in high-dimensional spaces, GD is more efficient
than direct solution
Example: Fitting Polynomials
Example: Fitting Polynomials
• We get polynomial regression for free!
• Define the feature map
• Polynomial regression model:
• All of the derivations and algorithms so far in this lecture
remain exactly the same!
Example: Fitting Polynomials
Example: Fitting Polynomials
Example: Fitting Polynomials
Example: Fitting Polynomials
Underfitting/Overfitting
•
Underfitting: The model is too simple - does not fit the data.
•
Overfitting: The model is too complex - fits perfectly, does not generalize.
Generalization
• We would like our models to generalize to data they haven’t seen before
• The degree of the polynomial is an example of a hyperparameter,
something we can’t include in the training procedure itself
Setting Hyperparameters
• Idea #1: Choose hyperparameters that work best on the data
• BAD: This does not generalize
• Idea #2: Split data into train and test, choose hyperparameters that work
best on test data
• BAD: No idea how algorithm will perform on new data
• Idea #3: Split data into train, val, and test; choose hyperparameters on val
and evaluate on test
Setting Hyperparameters
• Idea #4: Cross-Validation: Split data into folds, try each fold as validation
and average the results
• Useful for small datasets, but not used too frequently in deep learning