Neural network architectures
and learning algorithms
Author : Bogdan M. Wilamowski
Source : IEEE INDUSTRIAL ELECTRONICS MAGAZINE
Date : 2011/11/22
Presenter : 林哲緯
1
Outline
•
•
•
•
•
Neural Architectures
Parity-N Problem
Suitable Architectures
Use Minimum Network Size
Conclusion
2
Neural Architectures
Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e © The MIT Press (V1.0)
3
Neural Architectures
Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e © The MIT Press (V1.0)
4
Neural Architectures
Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e © The MIT Press (V1.0)
5
error back propagation(EBP) algorithm
• error back propagation(EBP) algorithm
– multilayer perceptron (MLP)
Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e © The MIT Press (V1.0)
6
multilayer perceptron (MLP)
MLP-type architecture 3-3-4-1(without connections across layers)
Neural network architectures and learning algorithms, Wilamowski, B.M.
7
neuron by neuron(NBN) algorithm
• neuron by neuron(NBN) algorithm
– bridged multilayer perceptron (BMLP)
– fully connected cascade (FCC)
arbitrarily connected network
Neural network architectures and learning algorithms, Wilamowski, B.M.
8
neuron by neuron(NBN) algorithm
• Levenberg–Marquardt(LM) algorithm
– Improve nonlinear function of least square
– Forward & Backward Computation
• Jacobian Matrix
– Forward-Only Computation
9
bridged multilayer perceptron (BMLP)
BMLP architecture 3=3=4=1(with connections across layers marked by dotted lines)
Neural network architectures and learning algorithms, Wilamowski, B.M.
10
fully connected cascade (FCC)
Bipolar neural network for parity-8 problem in a FCC architecture
Neural network architectures and learning algorithms, Wilamowski, B.M.
11
Outline
•
•
•
•
•
Neural Architectures
Parity-N Problem
Suitable Architectures
Use Minimum Network Size
Conclusion
12
parity-8 problem
MLP 8*9 + 9 = 81 weights
BMLP 4*9 + 8 + 4 + 1 = 49 weights
Neural network architectures and learning algorithms, Wilamowski, B.M.
13
parity-8 problem
9 + 10 + 11 + 12 = 42 weights
Neural network architectures and learning algorithms, Wilamowski, B.M.
14
parity-17 problem
• MLP architecture needs 18 neurons
• BMLP architecture with connections across
hidden layers needs 9 neurons
• FCC architecture needs only 5 neurons
15
parity-N problem
• MLP architectures
nn = neurons
nw = weights
• BMLP architectures
• FCC architectures
Neural network architectures and learning algorithms, Wilamowski, B.M.
16
Outline
•
•
•
•
•
Neural Architectures
Parity-N Problem
Suitable Architectures
Use Minimum Network Size
Conclusion
17
suitable architectures
• For a limited number of neurons, FCC neural
networks are the most powerful architectures,
but this does not mean that they are the only
suitable architectures
18
suitable architectures
• if the two weights marked by red dotted lines
– signal has to be propagated by fewer layers
Neural network architectures and learning algorithms, Wilamowski, B.M.
19
Outline
•
•
•
•
•
Neural Architectures
Parity-N Problem
Suitable Architectures
Use Minimum Network Size
Conclusion
20
Use Minimum Network Size
• receive a close-to-optimum answer for all
patterns that were never used in training
• generalization abilities
21
Case Study
TSK fuzzy controller:
(a) Required control surface
(b) 8*6 = 48 defuzzification rules
TSK fuzzy controller:
(a) Trapezoidal membership functions
(b) Triangular membership functions
Neural network architectures and learning algorithms, Wilamowski, B.M.
22
Case Study
(a) 3 neurons in cascade (12 weights), training error = 0.21049
(b) 4 neurons in cascade (18 weights), training error = 0.049061
(a) 5 neurons in cascade (25 weights), training error = 0.023973
(b) 8 neurons in cascade (52 weights), training error = 1.118E-005
Neural network architectures and learning algorithms, Wilamowski, B.M.
23
time complexity
NBN algorithm can train neural
networks 1,000 times faster than
the EBP algorithm.
(a) EBP algorithm, average solution time of 4.2s, and average 4188.3 iterations
(b) NBN algorithm, average solution time of 2.4ms , and average 5.73 iterations
Neural network architectures and learning algorithms, Wilamowski, B.M.
24
two-spiral problem
NBN algorithm using FCC architecture
244 iterations and 0.913s
EBP algorithm using FCC architecture
30,8225 iterations and 342.7s
Neural network architectures and learning algorithms, Wilamowski, B.M.
25
Outline
•
•
•
•
•
Neural Architectures
Parity-N Problem
Suitable Architectures
Use Minimum Network Size
Conclusion
26
Conclusions
• FCC or BMLP architectures are not only more
powerful but also easier to train
• use networks with a minimum number of
neurons
• NBN have to invert a nw*nw matrix, but 500
weights are limit now.
27