Artificial Neural
Networks
KONG DA, XUEYU LEI & PAUL MCKAY
Neural Networks
What is Neural Networks
An Artificial Neural Network is a computational simulation of a
biological neural network.
 composed of a large number of highly interconnected processing
elements(neurons)
an information processing paradigm
learn by example
What is Neural Networks
Inspiration from brain
Figure1 Neuron structure
Figure 2 ANN structure
Figure1: http://www.studyblue.com/notes/note/n/biological-foundations-neuron-communication-/deck/1025438
Figure2: http://www.gdl.cinvestav.mx/~edb/students/evazquez/index.html
Warren McCulloch and
Walter Pitts modeled a
Bernard Widrow and
simple neural network
Marcian Hoff of Stanford
Donald
Hebb
pointed
out that
with
electrical
circuits
developed models called
neural pathways are strengthened
"ADALINE" and
The IBM research
laboratories
ledare used in
each time
that they
"MADALINE."
1943
•The first
the first effort
to Organization
simulate a neural
The
of Behavior
recurrent
network
•ADALINE Could predict the next
network
bit
1949
Bayesian network
1950’s
•MADALINE TheLearning
first neural
Vector Quantization
1959
network applied
a realterm
world
Longtoshort
memory network
1980
"deep learning" gained
traction in thephysical neural network
1982
problem
Convolutional
neural
mid-2000s
after a publication
Hierarchical temporal memory (HTM)
networks
were
introduced
in
by
Geoffrey
Hinton and Ruslan
……
1985
John Hopfield
of
Caltech
created
a 1980 paper by
Kunihiko
Salakhutdinov
more useful
machines
by using
Fukushima
Companies are working
bidirectional lines (Hopfield
A Boltzmann machine, a
on three types of
Network)
type
Mid 2000s
neuro chips - digital,
of stochastic recurrent
analog, and optical
neural network is
History of ANNs
invented by Geoffrey
Hinton and Terry
Sejnowski
NOW
History of ANNs
Figure 3 MADALINE structure
Figure 4 An example Boltzmann machine
Figure3:http://www.drdobbs.com/the-foundation-of-neural-networks-the-ad/184402585
Figure4:http://en.wikipedia.org/wiki/Boltzmann_machine
Figure5: http://www.schraudolph.org/teach/NNcourse/lstm.html
Figure 6 Neural chip http://www.gizmag.com/neuromorphic-chips/28586/
Figure 5 a maximally simple LSTM network
Types of ANNs
ANNs
Feed-forward
Neural
Networks
Singlelayer
Multilayer
Recurrent Neural Networks
Hopfield
network
Long short
term
memory
network
…
Others
Kohonen
selforganizing
network
…
Perceptron
𝑛
x1
x2
w1
𝑋=

𝑥𝑖 𝑤𝑖
𝑖=1
y
w2

𝑌=
−1,
+1,
𝑥<𝜃
𝑥≥𝜃
Activation function
Step function
Sigmoid function
WikiBooks. Artificial Neural Networks/Activation Functions. 25 August 2014.
Example 1
Çelebi, Ömer Cengiz. Neural Networks and Pattern Recognition Using MATLAB. Retrieved 25 August 2014.
Example 2
Çelebi, Ömer Cengiz. Neural Networks and Pattern Recognition Using MATLAB. Retrieved 25 August 2014.
Input signals
Output signals
Neural network topology
Input
layer
First
hidden
layer
Second
hidden
layer
Output
layer
Multilayer
Interconnected
Feed forward/recurrent
Deep learning
Training
Back propagation
Learning rate
Batch learning
Stochastic gradient
descent
Evolutionary algorithms
Reference: LeCun et al. Efficient BackProp. 1998.
Application areas
Function approximation
http://www.eweb.unex.es/eweb/fisteor/santos/sby.html
Classification
Screenshot from
https://www.youtube.com/watch?v=KuPai0ogiHk
Application areas
Data processing.
http://www.cslu.ogi.edu/tutordemos/nnet_recog/recog.html
Control
http://www.seit.adfa.edu.au/research/details2.php?page_id
=410&topic=Adaptive_Flight_Control
Application areas
Robotics
Screenshots from https://www.youtube.com/watch?v=V2ADU8YWIug