ImageNet Classification with
Deep Convolutional Neural Networks
Alex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton, NIPS 2012
Eunsoo Oh
(오은수)
ILSVRC
● ImageNet Large Scale Visual Recognition Challenge
● An image classification challenge with 1,000
categories (1.2 million images)
Processing…
Deep
Convolutional
Neural Network
(ILSVRC-2012 Winner)
reference : http://www.image-net.org/challenges/LSVRC/2013/slides/ILSVRC2013_12_7_13_clsloc.pdf
2
Why Deep Learning?
● “Shallow” vs. “deep” architectures
Learn a feature hierarchy all the way from pixels to classifier
reference : http://web.engr.illinois.edu/~slazebni/spring14/lec24_cnn.pdf
3
Background
● A neuron
Input
(raw pixel)
x1
Weights
x3
w1
w2
w3
…
wd
x2
Output: f(w·x+b)
f
xd
reference : http://en.wikipedia.org/wiki/Sigmoid_function#mediaviewer/File:Gjl-t(x).svg
4
Background
● Multi-Layer Neural Networks
● Nonlinear classifier
● Learning can be done
by gradient descent
 Back-Propagation
algorithm
5
Input
Layer
Hidden
Layer
Output
Layer
Background
● Convolutional Neural Networks
● Variation of multi-layer neural networks
● Kernel (Convolution Matrix)
reference : http://en.wikipedia.org/wiki/Kernel_(image_processing)
6
Background
● Convolutional Filter
.
.
.
Input
Feature Map
reference : http://cs.nyu.edu/~fergus/tutorials/deep_learning_cvpr12/fergus_dl_tutorial_final.pptx
7
Proposed Method
● Deep Convolutional Neural Network
● 5 convolutional and 3 fully connected layers
● 650,000 neurons, 60 million parameters
● Some techniques for boosting up performance
● ReLU nonlinearity
● Training on Multiple GPUs
● Overlapping max pooling
● Data Augmentation
● Dropout
8
Rectified Linear Units (ReLU)
reference : http://www.image-net.org/challenges/LSVRC/2012/supervision.pdf
9
Training on Multiple GPUs
● Spread across two GPUs
● GTX 580 GPU with 3GB memory
● Particularly well-suited to cross-GPU parallelization
● Very efficient implementation of CNN on GPUs
10
Pooling
● Spatial Pooling
● Non-overlapping / overlapping regions
● Sum or max
Max
Sum
reference : http://cs.nyu.edu/~fergus/tutorials/deep_learning_cvpr12/fergus_dl_tutorial_final.pptx
11
Data Augmentation
256x256
Training Image
12
Horizontal Flip
224x224
224x224
224x224
224x224
224x224
224x224
Dropout
● Independently set each hidden unit activity to
zero with 0.5 probability
● Used in the two globally-connected hidden layers
at the net's output
reference : http://www.image-net.org/challenges/LSVRC/2012/supervision.pdf
13
Overall Architecture
● Trained with stochastic gradient descent on two NVIDIA GPUs
for about a week (5~6 days)
● 650,000 neurons, 60 million parameters, 630 million connections
● The last layer contains 1,000 neurons which produces a
distribution over the 1,000 class labels.
14
Results
● ILSVRC-2010 test set
ILSVRC-2010 winner
Previous best
published result
Proposed Method
15
reference : http://image-net.org/challenges/LSVRC/2012/ilsvrc2012.pdf
Results
● ILSVRC-2012 results
Runner-up
Top-5 error rate : 26.172%
16
Proposed method
Top-5 error rate : 16.422%
Qualitative Evaluations
17
Qualitative Evaluations
18
ILSVRC-2013 Classification
reference : http://www.image-net.org/challenges/LSVRC/2013/slides/ILSVRC2013_12_7_13_clsloc.pdf
19
ILSVRC-2014 Classification
22 Layers
20
19 Layers
Conclusion
● Large, deep convolutional neural networks for
large scale image classification was proposed
● 5 convolutional layers, 3 fully-connected layers
● 650,000 neurons, 60 million parameters
● Several techniques for boosting up performance
● Several techniques for reducing overfitting
● The proposed method won the ILSVRC-2012
● Achieved a winning top-5 error rate of 15.3%,
compared to 26.2% achieved by the second-best entry
21
Q&A
22
Quiz
● 1. The proposed method used hand-designed
features, thus there is no need to learn features
and feature hierarchies. (True / False)
● 2. Which technique was not used in this paper?
① Dropout
② Rectified Linear Units nonlinearity
③ Training on multiple GPUs
④ Local contrast normalization
23
Appendix
Feature Visualization
● 96 learned low-level(1st layer) filters
24
Appendix
Visualizing CNN
25
reference : M. D. Zeiler and R. Fergus. Visualizing and understanding convolutional neural networks.
arXiv preprint arXiv:1311.2901, 2013.
Appendix
Local Response Normalization
●
: the activity of a neuron computed by applyuing kernel
i at position (x, y)
● The response-normalized activity
is given by
● N : the total # of kernels in the layer
● n : hyper-parameter, n=5
● k : hyper-parameter, k=2
● α : hyper-parameter, α=10^(-4)
● This aids generalization even though ReLU don’t require it.
● This reduces top-5 error rate by 1.2%
26
Appendix
Another Data Augmentation
● Alter the intensities of the RGB channels in training images
● Perform PCA on the set of RGB pixel values
● To each training image, add multiples of the found principal
components
● To each RGB image pixel
add the following quantity
●
●
,
: i-th eigenvector and eigenvalue
: random variable drawn from a Gaussian with mean 0
and standard deviation 0.1
● This reduces top-1 error rate by over 1%
27
Appendix
Details of Learning
● Use stochastic gradient descent with a batch size of 128 examples,
momentum of 0.9, and weigh decay of 0.0005
● The update rule for weight w was
● i : the iteration index
●
: the learning rate, initialized at 0.01 and reduced three times
prior to termination
●
: the average over the i-th batch Di of the derivative of the
objective with respect to w
● Train for 90 cycles through the training set of 1.2 million images
28
29