1
CS334
Deep Learning Intro Part 2
Convolution Neural Networks
And its Applications for Recognition, Detection,
and Segmentation
Ahmed Elgammal
Department of Computer Science
Rutgers University
1
Modern CNNs
• From MLP to CNNs
• Modern CNNs: AlexNet,…
• CNN for Detection: Ex: YOLO
• CNN for Segmentation
2
Object Recognition: a core vision task
Object Recognition
Object Detection
Semantic segmentation
Visual Question Answer
3
Hidden layer
Flatten
ℎ[𝑖, 𝑗]
Input layer
• Why Fully Connected MLPs not suitable for images?
• MLP cannot learn translation invariant features
• MLPs are good classifiers, they need features extracted from images before
classifying them.
• MLPs do not utilize the spatial relations between pixels. We can permute the pixels
of the images in our dataset and the MLP will behave similarly
!
!
• The number of parameters in a Fully Connected MLP is of 𝑂(∑! 𝑁"#
𝑁$%&
) , leads to
poor generalization
• Deeper MLPs suffers from vanishing gradients
From Fully Connected MLP to CNN
!"%$ !"%$
ℎ 𝑢, 𝑣 = 𝑏 + - - 𝑊 𝑖, 𝑗 . 𝑋[𝑢 + 𝑖, 𝑣 + 𝑗]
!"#$ &"#$
Weighing pixels (u+i, v+j) in
the vicinity of (u,v) with
coefficient W[i,j] to obtain
the value h[u,v]
Hidden layer
Fully Connected
Layer
Hidden layer
Convolution Layer
ℎ[𝑢, 𝑣]
ℎ[𝑢, 𝑣]
Input layer
Input layer
From Fully Connected MLP to CNN
!"%$ !"%$
Weighing pixels (u+i, v+j) in
the vicinity of (u,v) with
coefficient W[i,j] to obtain
the value h[u,v]
ℎ 𝑢, 𝑣 = 𝑏 + - - 𝑊 𝑖, 𝑗 . 𝑋[𝑢 + 𝑖, 𝑣 + 𝑗]
!"#$ &"#$
Hidden layer
Fully
Connected
Layer:
Input for each
hidden unit
comes from all
inputs
Hidden layer
ℎ[𝑢, 𝑣]
ℎ[𝑢, 𝑣]
Convolution Layer:
Input for each
hidden unit is
localized
h=WX+b
W
W
X
X
Input layer
Input layer
All hidden units
share the same set
of weights
Example of a biologically motivated recognition system
A convolutional neural network, LeNet; the layers filter, subsample, filter,
subsample, and finally classify based on outputs of this process.
Figure from “Gradient-Based Learning Applied to Document
Recognition”, Y. Lecun et al Proc. IEEE, 1998 copyright 1998, IEEE
CNN terms
• Convolution
• Receptive Field
• Kernel = Filter
• Pooling
• Stride
• Padding
• Weight sharing
• Feature map
• Etc.
9
Same / Valid Padding
• Given input 5x5x1 image, with Stride (1, 1) and Kernel (3, 3)
• Same padding
•
To preserve the size by
•
Introducing the padding area to make 6x6x1 image
•
Resulting feature map is 5x5x1
• Valid padding (same as leaving margin)
•
•
Do nothing
Feature map becomes 4x4x1
10
efficiency or because we wish to downsample, we move our window more than one pixel at a time,
skipping the intermediate locations.
Stride
We refer to the number of rows and columns traversed per slide as the stride. So far, we have
used strides of 1, both for height and width. Sometimes, we may want to use a larger stride. Fig.
6.3.2 shows a two-dimensional cross-correlation operation with a stride of 3 vertically and 2 horizontally. We can see that when the second element of the first column is output, the convolution
• Stride
(Moving
Step)rows. The convolution window slides two columns to the right when
window
slides
down three
• element
How many
pixels
move
to get the
next receptive
field window slides three columns
the second
of the
firsttorow
is output.
When
the convolution
to the right on the input, there is no output because the input element cannot fill the window
(unless we add another column of padding).
Stride
(2, 3) of 3 and 2 for height and width respectively. The shaded
Fig. 6.3.2: Cross-correlation
with= strides
portions are the output element and the input and core array elements used in its computation:
11
15
LeNet Architecture
Full Connection = Dense layer = Linear layer
Fig. 6.6.2: Compressed notation for LeNe
Fig. 6.6.2: Compressed notation for LeNe
AlexNet
17
• LeNet didn’t work well for larger, and realistic datasets
• Until 2012, SVM was more preferred
• Data and GPU
• Proved, for the first time, that the learned features can
transcend manually-designed features (on ImageNet)
AlexNet: Krizhevsky, A., Sutskever, I., &
Hinton, G. E. (2012). Imagenet classification
with deep convolutional neural networks.
NeuroIPS 2012
AlexNet
needed in 2012 to make the model fit on two small GPUs.
AlexNet vs LeNet
• Much deeper
•
5 conv layers, 3 dense layers,
• ReLU
•
No expensive exp() operation
• Why much larger kernel size?
• Multiple Input and Output Channels
• Regularization (avoid over fitting)
•
•
LeNet only uses weight decay
AlexNet used dropout (for dense layers) and image
augmentation
• Data Augmentation: Flipping, cropping, color changes,
etc.
LeNet
AlexNet
Fig. 7.1.2:
LeNet (left) and AlexNet
(right)
19
Pooling Layer
• To reduce the size of the feature map
• To reduce computation, obtain high-level features
• Local Translation Invariance
•
Approximately invariant to small translations to the input
• Max pooling
•
•
Extract maximum value in the receptive field
Noise Suppressant
• Average (Mean) pooling
•
Average the values in the receptive field
• In practice and general, max pooling works better than avg pooling
20
Since the input and convolution kernel each have ci channels, we can perform a cross-correlation
operation on the two-dimensional array of the input and the two-dimensional kernel array of the
convolution kernel for each channel, adding the ci results together (summing over the channels)
to yield a two-dimensional array. This is the result of a two-dimensional cross-correlation between
multi-channel input data and a multi-input channel convolution kernel.
Multiple Input Channels
21
In Fig. 6.4.1, we demonstrate an example of a two-dimensional cross-correlation with two input
•channels.
when cThe
> 1,
we need
a kernel
a tensor
of shape
kh × and
kw for
every
shaded
portions
are thethat
first contains
output element
as well
as the input
kernel
array
elements
used in its computation:
(1 ×these
1 + 2 ×c2 +tensors
4 × 3 + 5together
× 4) + (0 ×yields
0 + 1 ×a
1+
3 × 2 + 4 × 3) =
56.
input channel.
Concatenating
convolution
kernel
of shape c × kh × kw .
Fig. 6.4.1: Cross-correlation computation with 2 input channels. The shaded portions are the first
output element as well as the input and kernel array elements used in its computation: (1 × 1 +
2 × 2 + 4 × 3 + 5 × 4) + (0 × 0 + 1 × 1 + 3 × 2 + 4 × 3) = 56.
Multiple Input Channels
22
When the input data contains multiple channels, we need to construct a convolution kernel with
the same number of input channels as the input data, so that it can perform cross-correlation with
the input data. Assuming that the number of channels for the input data is ci , the number of input
channels of the convolution kernel also needs to be ci . If our convolution kernelʼs window shape
is kh × kw , then when ci = 1, we can think of our convolution kernel as just a two-dimensional
array of shape kh × kw .
Convolution Layer with Multi-channel Inputs
However, when ci > 1, we need a kernel that contains an array of shape kh × kw for each input
channel. Concatenating these ci arrays together yields a convolution kernel of shape ci × kh × kw .
Since the input and convolution kernel each have ci channels, we can perform a cross-correlation
depend on
(i, j),oni.e.,
we have V array
[i, j,ofa,theb]input
=V
[a,
and u is a kernel
constant.
As a
operation
the two-dimensional
and
theb]
two-dimensional
array of the
convolution
the definition
forkernel
h. for each channel, adding the ci results together (summing over the channels)
to yield a two-dimensional array. This is the result of a two-dimensional cross-correlation between
multi-channel input
)data and a multi-input channel convolution kernel.
• From C input channels (e.g., ‘RBG’)
h[i,
j] =
u+
V [a, b]
x[i + ofa,ajtwo-dimensional
+ b].
In Fig.
6.4.1,
we demonstrate
an·example
cross-correlation with two (6.1.2)
input
channels. The shaded
a,b portions are the first output element as well as the input and kernel array
elements used in its computation: (1 × 1 + 2 × 2 + 4 × 3 + 5 × 4) + (0 × 0 + 1 × 1 + 3 × 2 + 4 × 3) = 56.
We are effectively weighting pixels (i + a, j + b) in the vicinity of (i, j) with
obtain the value h[i, j]. Note that V [a, b] needs many fewer coefficients than
egapixel image it has at most 1 million coefficients. This is 1 million fewer
longer depends on the location within the image. We have made significant
econd principle—locality. As motivated above, we believe that we shouldnʼt
way from (i, j) in order to glean relevant information to assess what is going
ns that outside
range computation
|a|, |b| >with
∆,2 we
setshaded
V [a,portions
b] = 0.
EquivaFig. 6.4.1: some
Cross-correlation
inputshould
channels. The
are the
first
output
element
as
well
as
the
input
and
kernel
array
elements
used
in
its
computation:
(1
×
1+
h[i, j] as
2 × 2 + 4 × 3 + 5 × 4) + (0 × 0 + 1 × 1 + 3 × 2 + 4 × 3) = 56.
∆
∆
)
)
To make sure we really understand what is going on here, we can implement cross-correlation
h[i, j]operations
= u + with multiple input
V [a,
b] · x[i
+ a, Notice
j + b].
(6.1.3)
channels
ourselves.
that all we are doing is performing
one
cross-correlation
operation
per
channel
and
then
adding
up
the
results
using
the
add_n
function.
a=−∆ b=−∆
import d2l
23
K
𝑂 𝑙, 𝑖, 𝑗 = ' ' 𝑋 𝑐, 𝑖 + 𝑎, 𝑗 + 𝑏 𝐾(𝑙, 𝑐, 𝑎, 𝑏)
!∈!! #,%
Kernel for one
output channel
HI
X: Input Tensor
One output channel
Ho
kh
wI
cI
kw
cI
wo
Has to match the input channels
cI
Ho
kh
kh
kh
kw
kw
kw
cI
cI
Stack all kernels for form a kernel tensor of size 𝑐' × 𝑐( × 𝑘) ×𝑘*
wo
co
O : Output Tensor
Multiple Output channels
• Denote by 𝑐' and 𝑐( the number of input and output channels, respectively,
• let 𝑘) and 𝑘* be the height and width of the kernel,
• To get an output with multiple channels, we can create a kernel tensor of shape
𝑐' × 𝑘) ×𝑘* for every output channel
• so that the shape of the convolution kernel is 𝑐( × 𝑐' × 𝑘) ×𝑘*
• The result on each output channel is calculated from the convolution kernel
corresponding to that output channel and takes input from all channels in the input
tensor.
AlexNet
• Learned features
Fig. 7.1.1: Image filters learned by the first layer of AlexNet
gher layers in the network might build upon these representations to represent larger struces, like eyes, noses, blades of grass, etc. Even higher layers might represent whole objects like
ople, airplanes, dogs, or frisbees. Ultimately, the final hidden state learns a compact represen-
Modern CNNs
ResNet
• AlexNet
• VGG
• NiN
• GoogLeNet
• ResNet
• DenseNet
• …
Fig. 7.2.1: Designing a network from building blocks
Fig. 7.6.4: ResNet 18
VGG
•
we provide a slightly streamlined version of AlexNet removing some of the design
34
needed in 2012 to make the model fit on two small GPUs.
VGG: Visual Geometry Group at Oxford Univ.
• Simonyan, K., & Zisserman, A. (2014). Very deep
convolutional networks for large-scale image
recognition
Motivation: How to go deeper!
•
In successive convolution and pooling, the spatial
resolution decreases quite rapidly (half every time).
•
This imposes a hard limit of log 𝑑 convolutional layers on
the network before all dimensions, d , are used up.
•
For instance, in the case of AlexNet, it would be
impossible to have more than 8 convolutional layers in
this way.
AlexNet
Fig. LeNet
7.1.2: LeNet (left) and AlexNet
(right)
VGG:
Simonyan, K., & Zisserman, A. (2014). Very deep convolutional networks for large-scale
image recognition
7.2.1
Blocks
• VGG
VGG
Block
• Ablock
sequence
of convolutional
conv layers networks is a sequence of the following layers: (i)
The basic building
of classic
a convolutional
padding to maintain the resolution), (ii) a nonlinearity such as a ReLU,
• layer
Max(with
pooling
One VGG block consists of a sequence of convolutional layers, followed by a max pooling layer
for spatial
downsampling.
original
VGG paper (Simonyan & Zisserman, 2014), the authors
• The
sequence Inofthe
conv
layers
employed convolutions with 3 × 3 kernels and 2 × 2 max pooling with stride of 2 (halving the reswithin
block
not
reduce
olution after
eachthe
block).
In thedo
code
below,
we define a function called vgg_block to implement
one VGGthe
block.
The function takes two arguments corresponding to the number of convolutional
dimension.
layers num_convs and the number of output channels num_channels.
• One reduction in size per block
import d2l
at the
last
in the block
from mxnet
import
np, layer
npx
from mxnet.gluon import nn
(pooling)
npx.set_np()
def vgg_block(num_convs, num_channels):
blk = nn.Sequential()
for _ in range(num_convs):
blk.add(nn.Conv2D(num_channels, kernel_size=3,
padding=1, activation='relu'))
blk.add(nn.MaxPool2D(pool_size=2, strides=2))
return blk
35
VGG-11
Ex: VGG-16
VGG
• Main Conclusion: Found that several layers of deep and narrow
convolutions were more effective than fewer layers
of wider convolutions.
39
GoogLeNet (Inception V1): Inception Blocks
• Basic blocks in GoogLeNet
• Consists of four parallel paths
1. 1x1 Conv
2. 3x3 Conv
3. 5x5 Conv
4. …
• To extract information from different spatial sizes
• Appropriate padding is required to each path to give the same output size
• Concatenation at the final layer
40
question of which sized convolutional kernels are best. After all, previous popular networks
ployed choices as small as 1 × 1 and as large as 11 × 11. One insight in this paper was that somees it can be advantageous to employ a combination of variously-sized kernels. In this section,
will introduce GoogLeNet, presenting a slightly simplified version of the original model—we
it a few ad hoc features that were added to stabilize training but are unnecessary now with
ter training algorithms available.
.1 Inception
• Blocks
Inception Block
e basic convolutional block in GoogLeNet is called an Inception block, likely named due to a
te from the movie Inception (“We Need To Go Deeper”), which launched a viral meme.
• Input and output have same spatial
dimensions
• All branches have same spatial dimension
to facilitate concatenation of feature maps
Fig. 7.4.1: Structure of the Inception block.
Reduce
# of input
channels
Networks with Parallel
Concatenations
(GoogLeNet)
277
1x1 convolution
• Convolution on the channel dimension
only
• Goal: reduce the number of channels
• Equivalent to FC layer on the channels
• GoogLeNet: different kernel sizes at each layer.
ResNet
44
• ResNet (inductive bias)
§ Every additional layer should contain the identity function as one of its elements
§ If we train a newly-added layer into an identity mapping f(x) = x, the new model will be
as effective as the original model.
§ As the new model may get a better solution, the added layer might make it better.
§ This idea sounds quite complex but a very simple solution exists
d that
particular
layer
and
we
would
mu
Residual Blocks
line box in the right image now only n
e return x + f (x). In practice, the residu
f (x) = 0. The right image in Fig. 7.6.2
hitectures were later proposed for sequ
45
7.6.2 Residual Blocks
Letʼs focus on a local neural network, as depicted below. Denote the input by x. W
the ideal mapping we want to obtain by learning is f (x), to be used as the input to
function. The portion within the dotted-line box in the left image must directly fit
f (x). This can be tricky if we do not need that particular layer and we would much rat
input x. The portion within the dotted-line box in the right image now only needs t
the deviation from the identity, since we return x + f (x). In practice, the residual ma
easier to optimize. We only need to set f (x) = 0. The right image in Fig. 7.6.2 illustr
Residual Block of ResNet. Similar architectures were later proposed for sequence
we will study later.
• On the left, f(x) needs to learn the whole function
• F(x) only needs to model the deviation from the identity
•
Easier to learn not to use or delay a layer
Regular Block
Residual Block
Fig. 7.6.2: The difference between a regular block (left) and a residual block (right)
case, we can short-circuit the convolutions.
def forward(self, X):
Y = npx.relu(self.bn1(self.conv1(X)))
Y = self.bn2(self.conv2(Y))
if self.conv3:
X = self.conv3(X)
return npx.relu(Y + X)
ResNet Block
This code generates two types of networks: one where we add the input to the output before applying the ReLU nonlinearity, and whenever use_1x1conv=True, one where we adjust channels and
resolution by means of a 1 × 1 convolution before adding. Fig. 7.6.3 illustrates this:
Fig. 7.6.3: Left: regular ResNet block; Right: ResNet block with 1x1 convolution
46
Moving to the age of Deep NN
●
Common object recognition (1000 categories)
Image credit to GORDON COOPER
Progress in Recognition using CNNs
CNNs for Detection
From Classification to Detection
Object Localization
Is there an object
(Y/N)
Pc
bx
by
bh
bw
C1
C2
.
.
CN
CNN
Predict
Bounding box
& class probabilities
Bounding box
Class prob.
Grid-based Bounding box prediction
Is there an object
(Y/N)
Pc
bx
by
bh
bw
C1
C2
.
.
CN
CNN
Predict
Bounding box
& class probabilities
Bounding box
Class prob.
Grid-based Bounding box prediction – YOLO
Bounding box regression
CNN
YOLO: You Only Look Once: Unified, Real-Time Object Detection
Output tensor
3x3xY
Pc
bx
by
bh
bw
C1
C2
.
.
CN
YOLO: You Only Look Once: Unified, Real-Time Object Detection
CNNs for Image Segmentation
Semantic Segmentation
CS 534 – Segmentation - 58
J Long et al “Fully Convolutional Networks for Semantic Segmentation” (PAMI, 2016)
CS 534 – Segmentation - 59
Encoder-decoder architecture
Special case: Auto-encoder : If the input=output
Encoder and decoder can be fully connected or convolution layer
CS 534 – Segmentation - 62
Transpose convolution – up convolution
Image credit: https://d2l.ai/chapter_computer-vision/transposed-conv.html#sec-transposed-conv
CS 534 – Segmentation - 63
Transpose convolution – up convolution
Image credit: https://d2l.ai/chapter_computer-vision/transposed-conv.html#sec-transposed-conv
CS 534 – Segmentation - 64
Convolution encoder decoder for segmentation
H. Noh, S. Hong, and B. Han, “Learning deconvolution network for semantic
segmentation, 2015
U-Net
Convolutional Encoder-decoder
With skip connections
CS 534 – Segmentation - 66
Ronneberger et al U-Net: Convolutional Networks for Biomedical Image Segmentation (MICCAI, 2015)
CNNs for Audio
• Recall: Digitized audio is 1D signal sampled over time
2-D convolution on Spectrograms
1-D Convolution on raw wave data
Source
Source
CNNs for Video
• Recall: Video is a sequence of frames.
---> 3D Convolution over space and time
source
Sources
• D2L: Dive to Learning