Chemometrics and Intelligent Laboratory Systems 211 (2021) 104269
Contents lists available at ScienceDirect
Chemometrics and Intelligent Laboratory Systems
journal homepage: www.elsevier.com/locate/chemometrics
Investigation of transfer learning for image classification and impact on
training sample size
Wenbo Zhu a, Birgit Braun b, Leo H. Chiang b, *, Jose A. Romagnoli a
a
b
Department of Chemical Engineering, Louisiana State University, Baton Rouge, LA, 70803, United States
Chemometrics and AI, Dow Inc., Lake Jackson, TX, 77566, United States
A R T I C L E I N F O
A B S T R A C T
Keywords:
Deep learning
Transfer learning
Image processing
Classification
Recent developments in deep learning have brought huge breakthroughs in the image processing area, which
triggered numerous successful applications and positively impacted the current big data context of Industry 4.0.
On the other hand, it is widely known that large amounts of training data are required to train a deep learning
model with millions of parameters, which limits its application in many industrial applications where sufficient
data resources are lacking. Transfer learning is one of the practical solutions to reduce the data required for
training, which tries to reuse learned knowledge for similar tasks. Nevertheless, many technical details of transfer
learning implementation are not well documented. Therefore, in this work, two datasets collected from plastics
manufacturing processes were studied to investigate different transfer learning approaches and implementation
details for high-performance model building under the constraint of limited available training data. Different
transfer learning implementations are compared and important technical details are also discussed in this study.
Through this study, the minimum number of training samples can be estimated. Transfer learning is compared
with the newly developed few-shot learning approach as a brief comparative study. Finally, this work summarizes
practical guidelines for the development of image classification models with limited data resources.
1. Introduction
In the past decade, deep learning (DL) approaches have achieved
huge breakthroughs in computer vision areas including classification,
segmentation, and detection problems [1–3]. In contrast to the traditional machine learning methods using a shallow architecture, the DL
method stacks multiple convolutional and pooling layers to form a deep
model to extract multiple granularity features [4]. Compared to many
other image processing methods such as sparse coding [5,6] and
scale-invariant feature transform (SIFT) [7], deep models show promising results in different image processing problems, and outperform
traditional methods in academic benchmarks [8]. In He’s work [9], they
claimed to surpass human-level performance in the benchmark. The
great success of DL approaches in the computer vision area triggered the
bloom of these techniques in a broad number of applications. For
instance, they have been applied in the smart city plan to manage vehicles in the urban area [10]. Video-based security systems also adopted
the novel DL methods to improve their surveillance efficiency, which
presents a lot of good case studies, such as falling detection for the elders
[11], drone-based crowd surveillance [12], and vision-based driver
assistance systems [13]. In chemical industries, Zhu et al. [14] proposed
an infrared image-based monitoring system for an ethylene pyrolysis
reactor using image segmentation techniques. Manee et al. [15] built a
DL-based sensor integrated with high-speed optical microscopy to characterize crystal size distribution in real-time. Rendall et al. [16] developed a DL-based image classification model to inspect the quality of
plastic pellets.
On the other hand, one major concern behind these success cases is
that the parameter amount of the DL method is always much larger than
traditional methods, e.g. Support Vector Machine [17], Decision Tree
[18], and others. For example, the VGG-16 network [19] used in
Ref. [16] for pellets quality inspection has about 138 million parameters.
Even for the lightweight models, such as ResNet-18 [20] and MobileNet
[21], there are still more than one million parameters that need to be
trained. Consequently, to tune these large amounts of model parameters,
a big training dataset is necessary. For instance, the most famous
benchmark for image classification, ImageNet [6] dataset has more than
14 million labeled images. In the industrial application mentioned previously [16], thousands of images were used to train the DL model. The
large training data requirement potentially limits DL applications in
* Corresponding author.
E-mail address: HChiang@dow.com (L.H. Chiang).
https://doi.org/10.1016/j.chemolab.2021.104269
Received 3 November 2020; Received in revised form 29 January 2021; Accepted 2 February 2021
Available online 15 February 2021
0169-7439/© 2021 Elsevier B.V. All rights reserved.
W. Zhu et al.
Chemometrics and Intelligent Laboratory Systems 211 (2021) 104269
Fig. 1. Illustration of a typical deep convolutional neural network.
classification in practice. Rather than the transfer learning approach, the
recent development of few-shot learning [24–26] also aims to build
classification models using very few training samples.
This work contributes to provide the answers to the challenges discussed above, and to have a better understanding of transfer learning and
its ability to yield high-accuracy classification models with minimal
training data. Datasets from two industrial applications were collected
for this study. Different transfer learning implementations are compared
in the scenarios using limited training data, by which the most applicable
transfer learning approach can be determined. The comparison with
metric-based few-shot learning is also presented in this work to give
further insight into methods selection. Through the investigation, we also
determined the minimum number of images required to build a DL-based
image classification model with high accuracy in the given scenarios.
While obviously not directly translatable to other case studies, the provided information is used to develop a procedural guidance to model
building that eases the development process when training data is
limited. This study summarizes useful suggestions and guidelines for
scientists and engineers with little or no machine learning backgrounds
to help them develop DL-based image classification models in their
respective areas.
The remainder of the manuscript is organized as follows. Section 2
introduces the fundamental background of DL models for image processing, namely the deep convolutional neural networks. Section 3 describes the implementation details of transfer learning. Section 4
provides the background of the datasets from two industrial case studies.
Section 5 presents the main results and the corresponding discussions.
Finally, Section 6 summarizes the main conclusions and contributions of
this work.
industry, because, in many industrial applications, large amounts of
training data with balanced labels are not easily available.
Practically, transfer learning [4,22,23] is an approach to address the
large training data requirement issue for image classification problems.
The transfer learning refers to the approach that utilizes models trained
on one problem as a starting point to solve related ones. Using this
approach, the data required for domain-specific tasks can be significantly
reduced [4].
Although the transfer learning approach presents a promising solution to reduce the training data requirement, it should be noticed that
several technical details are not clearly addressed. For example, in every
DL-based classification model, there are two parts inside, namely the
feature extraction parts (stacking of convolutional and pooling layers)
and classification parts (fully connected layers). When using transfer
learning to build a classification model, it is uncertain which parts play a
more important role in successfully applying to a new dataset. Additionally, there are various DL models for image classification, such as
VGGNet, ResNet, and others. The selection of the bare-bones model could
be an important point for the implementation of transfer learning,
particularly for the cases with limited data. Big models such as ResNet152 can achieve extremely high accuracy in the benchmark tests
compared with lightweight models like ResNet-18, but as a trade-off, big
models require much more parameters to be adjusted than lightweight
models. Further, there is no clear answer to the minimum sample
requirement of training data when using transfer learning. Without this
indicator, model development could struggle at the data collection stage.
All these aspects are essential before deploying transfer learning in image
2. Background
2.1. Deep convolutional neural networks
In the deep convolutional neural network (DCNN), which are the
most widely applied DL architectures for image classification, the
convolution operation and the pooling operation are the most important
features. The convolution operation utilizes multiple filters sliding over
the input image to generate invariant local features, which improves the
model generalization capability by weight sharing and local connectivity.
The extracted feature maps from the convolution layers are then activated by nonlinear function, such as rectifying linear unit (ReLU). After
that, the dimension of the feature map is reduced by the pooling operation, in which a sliding window scans the feature map and a single value
is returned for each scanned region by the maximum value or the averaged value. The above steps may repeat multiple times by stacking the
convolution and pooling layers to refine the features. At the end of the
DCNN, the final output feature map is flattened to a 1D vector and fed
Fig. 2. Demonstration of building blocks of deep convolutional neural network.
(a) Plain convolutional structure. (b) ResNet blocks with identity mapping.
Convolutional layers (Conv), ReLU activation (ReLU) and batch normalization
(BN) are the basic elements for the building blocks.
2
W. Zhu et al.
Chemometrics and Intelligent Laboratory Systems 211 (2021) 104269
2.2. Transfer learning
DCNN models are much larger and have many more parameters than
the traditional machine learning methods, and training usually requires a
large training dataset. This data requirement burden potentially limits
application of DCNN models, especially in industry where labeled
training data is often sparse. To address this problem, transfer learning is
a well-known approach that utilizes known knowledge obtained from
one task to apply on other similar tasks, which can significantly reduce
data requirements. According to Pan’s study [23], there are three main
categories in transfer learning: inductive transfer learning, transductive
transfer learning, and unsupervised transfer learning. Since DL models
are based on representation learning, transfer learning in DL models
usually belongs to inductive transfer learning, which allows the application of learned knowledge transfer to different but related tasks.
As shown in Fig. 3, the common transfer learning approach for image
processing problems are usually composed of two parts, namely pretraining and fine-tuning. The ImageNet dataset is most commonly used
for pre-training, by which the convolutional filters in the model can be
tuned to extract useful features [4]. Subsequently, the pre-trained model
is used on other tasks. Based on the detail of the new task, it only requires
minor modifications of the model structure and fine-tuning of the parameters, which reduces the effort of training the model from randomness. For example in Ref. [4], the authors demonstrated their approach to
transfer a classification model from the ImageNet dataset into a new
dataset by replacing the final layer with a support vector machine to
accomplish the classification task with very few training samples.
Fig. 3. Scheme of transfer learning.
into a softmax function for final class prediction. Pooling and filter
window sizes are adjustable hyper parameters along with layer sizes and
counts. A typical DCNN layout can refer to Fig. 1.
After introducing the building blocks of the DCNN, several DCNN
models widely used in the DL community are introduced. The VGGNet
[19] is one of the most popular models in the image processing area,
which achieved the highest score in the ImageNet Large Scale Visual
Recognition Competition(ILSVRC) in 2014. The architecture of the
VGGNet simply follows the same idea in Fig. 1, which stacks multiple
convolutional and pooling layers for the feature extraction and generates
categorical prediction from the flatten features. In the original work of
VGGNet, the authors proposed five configurations with different
numbers of layers with trainable weights. The error rate of the model
decreases as the number of layers increases, but when VGGNet reaches
19 layers, the error rate reaches saturation. After that, a degradation
problem has been reported [20,27] where the accuracy degrades rapidly
with the increasing of the network layers.
To further improve the DCNN performance and relieve the degradation problem, He et al. [20] proposed an alternative structure by introducing the gradient highways into the network, named ResNet. Fig. 2b
demonstrates the design of the ResNet building block. Compared with the
plain convolutional network in Fig. 2a, the main difference comes from a
highway connection that performs the identity mapping of the input and
element-wise addition to the output. Benefiting from the design of the
residual architecture, the degradation problem can be solved and the use
of deeper models can achieve better performance. The strategy of the
residual architecture has a profound influence on subsequent models,
such as Inception [28] and MobileNet [29].
2.3. Few-shot learning
Few-shot learning is a subcategory of deep learning, which aims at
recognizing new concepts using the pre-trained DCNN model with just a
few labeled samples in each class [30]. A few-shot learning problem
usually turns into an N-way-K-shot classification, where N way means the
number of classes and K shots means the number of training samples in
each class. These are also called the support set and are used to build the
model for solving the task. Besides that, additional samples of the same
classes as the support set are used to evaluate the performance on a given
task, known as a query set. Unlike in normal classification problems
where the task for training and testing are the same, in few-shot learning,
the classification tasks can be non-overlapping. An example is shown in
Fig. 4. In the training phase, the task is to distinguish birds and dogs,
Fig. 4. A demonstration of few-shot learning structure.
3
W. Zhu et al.
Chemometrics and Intelligent Laboratory Systems 211 (2021) 104269
3.1. Pellet dataset
The output of chemical processes that convert monomeric building
blocks into polymers are plastic pellets, which are subsequently processed into other products such as films and toys. Pellet shape is an
important quality factor for customers, since deviations from the ideal
spherical shape potentially leads to processing and quality issues at the
converter. For example, elongated pellets can cause bridging in feeding
hoppers and tails can break off resulting in black specks in films due to
oxidation and degradation. Therefore, pellet shape analysis is common in
plastics manufacturing. Commercial analytical equipment is available to
generate silhouette images from grab samples. The dataset consists of
desired pellets (Fig. 5b) and pellets with bad shape (Fig. 5a). Each image
in the dataset is in three-color channels (RGB) and with a resolution of
9696 pixels. This dataset has also been used in Rendall’s study [16]. In
Rendall’s work [16], an image classifier using VGGNet has been proposed, while thousands of images are used. In this study, the main focus is
not on training a classifier, but on applying transfer learning to build a
high accuracy model with as little training data as possible.
Fig. 5. Sample of two types of pellets: (a) A pellet with round shape, (b) A pellet
with bad shape.
3.2. Defect dataset
The second dataset is collected from a polyethylene(PE) production
line for wire and cable grades, for which contamination quantification is
imperative. For this purpose, three at-line pellet sorters from optical
control systems (OCS) are running in parallel. In addition to size and
color, classification of different types of contaminants (embedded defects
and loose defects) is required for some applications such as identifying
the process origin of the defects. Defect examples are shown in Fig. 6. As a
binary classification problem, the DCNN model can be adapted to automate the defect determination. The images for this case study are also in
three-color channels and with a resolution as 245245. Again, the main
objective of this work is to investigate different approaches in transfer
learning, which allows model training with as little training data as
possible.
Fig. 6. Sample of two types of defects: (a) Embedded defect, (b) Loose defect.
while in the testing phase, the task changes to classification between
geese and fish. This framework can generalize the model to distinguish
data classes in general, rather than a specific subset of classes.
The metric-based approach is one of the most popular methods of
few-shot learning. In the metric-based approach, the DCNN models
embed the input images into the feature space, where distance metrics
determine the class assignment. Based on this idea, different approaches
have been proposed. The Siamese network [24] uses the contrastive loss
function. The cosine distance [25] and the Euclidean distance [26] are
utilized in the matching network and prototypical network respectively.
3. Data preparation
3.3. Data augmentation
This section presents the background of the datasets used in this
work. Both of the datasets originated from quality inspection processes at
polyethylene production plants.
Data augmentation (DA) is another widely used approach to address
the DL model training issue with limited data resources. Common data
augmentation methods include cropping, flipping, rotating, noise adding,
and color-changing from the original images [31]. Some other methods
such as stretching, image segmentation and other distortions are not
sensible for this application, since the datasets in this work come from an
industrial analyzer in chemical plants. Therefore, only image flipping,
and rotating are used in this study since they preserve the fundamental
nature of the images.
4. Implementations
This section introduces the experiment designs corresponding to the
questions raised in the introduction part. The ResNet-18 model [20] is
selected as the bare-bones models for this study due to its simplicity in
implementation and relatively high performance compared with others
[31]. To address the size difference of different datasets, spatial pooling
[32] is used instead of a normal pooling operation to regulate the intermediate size of feature maps. The overall idea of the experiment is to
use various amounts of data to train different approaches and calculate
their accuracy through the test data. Thus, we can evaluate the effectiveness of different transfer learning approaches and determine the
minimum training samples of each approach. A classifier with an accuracy of 90% is considered acceptable and useful in reality, particularly
when training samples are limited.
The accuracy is defined as follows:
Fig. 7. Implementation of three common transfer learning approaches. The red
part indicates the trainable parts, while the blue part represents the freezing
layers. Approach (a) replaces the final classification layer by a linear SVM
classifier. Approach (b) replaces the final layer with a new softmax classifier.
Approach (c) allows the fine-tuning over the entire network. (For interpretation
of the references to color in this figure legend, the reader is referred to the Web
version of this article.)
4
W. Zhu et al.
Chemometrics and Intelligent Laboratory Systems 211 (2021) 104269
It should be noted that the fune-tuning parts of the ConvNet is not
considered in this study. According to Ref. [33], fine-tuning part of the
ConvNet does not bring any advantages to the model accuracy compared
with fine-tuning the entire network. In Ref. [33], it also shows that
freezing part of the pre-trained weights can also downgrade the model
performance due to the specificity of the model.
Two data augmentation methods, image flipping (vertically, horizontally, diagonally, and anti-diagonally) and rotation (90 , 180 , and
270 clockwise) are applied in model training as a comparison. Thus, if
data augmentation is applied, it generates 7 times more images for
training.
For the implementation of few-shot learning models, the prototypical
network is chosen for performance comparison in this work due to its
high accuracy reported in Ref. [26]. We simply followed the structure in
Ref. [26], which is a four-layer convolutional network. The prototypical
network is trained on the mini-ImageNet dataset which is a subset of the
original ImageNet dataset for the few-shot learning benchmark. The
trained model is then applied to the two given datasets.
5. Results and discussion
After introducing experimental methods and background information
of the dataset, this section presents the results for two case studies. In
addition, further suggestions and discussions are also summarized in this
section.
5.1. A comparison of three transfer learning methods
The results of the three transfer learning implementations discussed
previously are summarized in this section. All three approaches were
trained multiple times with the various amount of training data ranging
from 20 to 200 samples for each class with and without data augmentation. The average testing accuracy from different runs are reported in
Fig. 8.
For the pellet shape dataset, approach (a) and approach (c) give
similar results within the given range of training data, while approach (b)
can hardly achieve 90% accuracy even with data augmentation. In the
scenario with a very limited amount of training samples (image size 50) without DA, approach (a) outperforms approach (c), while using
slightly more training data, and applying DA during training, approach
(c) can achieve higher accuracy than approach (a). On the other hand, for
the defect dataset, all three approaches work much worse than the pellet
dataset. Only approach (c) with DA can achieve the target accuracy,
while approach (a) and (b) cannot pass the 90% target line in the entire
experiment. Compared with approach (a) and (c), approach (b) has the
worst performance in both datasets, and thus does not seem to be a
practical solution for transfer learning.
Data augmentation brings remarkable improvement for all three approaches, which can increase the testing accuracy by 2%–10% in the
experiments. Hence, for building models with limited data, DA is recommended to improve accuracy. It is imperative to be selective in DA
methods to be used for a given task; not all typically employed image
modifications provide realistic images, such as stretching and cropping in
the use cases discussed here.
Fig. 8. Accuracy comparison of three transfer learning approaches with the
incorporation of data augmentation for (a) pellets dataset, and (b) defects
dataset. Image size in x coordinate indicates the number of images for each class.
Accuracy ¼ ð1 jx 2 X : tðxÞ 6¼ yðxÞj
Þ 100%
jXj
(1)
where t represents the target, y is the prediction and X is test samples. The
test data for both of datasets contains 200 separate images per category.
In transfer learning, the model training includes two steps, pretraining and fine-tuning. The pre-training step helps the model to learn
general features that are parameterized by model weights. The learned
features can be re-used in similar tasks. To train a deep learning model
from scratch with millions of random parameters, a large dataset is
required, particularly for the pre-training part. In general, the ImageNet
[6] dataset that includes more than 14 million labeled images is used for
image processing area. The pre-training step simply follows the supervised learning architecture, where cross entropy is used as a loss function
and the gradient is back propagated for weights updating. The
pre-trained weights are used as the initial value in fine-tuning.
Fine-tuning is the next step to transfer the general feature into a
specific task, where training samples are always limited. Typically, each
DCNN model contains two parts, the feature extraction component
composed of multiple convolutional layers, and fully connected layers
with softmax function for final prediction. To investigate the most
effective part for fine-tuning, we simply set up three common transfer
learning implementations, which are summarized in Fig. 7. For approach
(a) and (b), the feature extraction parts are frozen, and only fine-tuning in
the final classification portion is allowed. The main difference between
approach (a) and (b) is the choice of the classifier, where a linear SVM
classifier is used in approach (a), and a new softmax classifier is used for
approach (b). For approach (c), the final layer is replaced first corresponding to the output size and all parameters in the network are set to be
trainable.
5.2. Feature space inspection
It is noticeable that although approach (c) has much more trainable
parameters than others, approach (c) performs consistently well in both
datasets. For the methods using the pre-trained feature extraction layers,
both approach (a) and (b) present huge performance variance in the two
case studies. Based on this observation, the feature extraction part plays a
more important role in the entire model. A well tuned feature extractor
presents clear separations between classes, which makes it easier for the
classifier to draw decision boundaries between them. It explains the
reason behind the observation that approach (c) can achieve consistently
5
W. Zhu et al.
Chemometrics and Intelligent Laboratory Systems 211 (2021) 104269
Fig. 9. UMAP visualization on the feature space of the pellet dataset: (a) pre-trained ResNet 18 model, (b) fine-tuned ResNet 18 model.
Fig. 10. UMAP visualization on the feature space of the defect dataset: (a) pre-trained ResNet 18 model, (b) fine-tuned ResNet 18 model.
decision boundary, by which a high prediction accuracy can be achieved.
On the other hand, overlapping clusters can make the decision boundary
difficult to determine. In Equation (2), the numerator is increased with
the distance between different clusters, and the denominator is used to
normalize feature values from different input. Thus, this ratio can be a
robust indicator to represent the feature space distribution. To demonstrate the ratio values corresponding to different distributions, dummy
datasets are created with 5% noise and 512 feature dimensions which is a
common dimension number used in many convolutional models. Fig. 11
presents the UMAP [34] visualization of the dataset with calculated ratio
values in 2D space. Clear separation can be observed at around 0.7 of the
ratio, which makes the classifier to easily draw the decision boundary.
We also tested the ratio values versus the accuracy on various
methods, including various bare-bones models (VGG, ResNets, etc.),
different transfer learning implementations (approach (a) to (c)), and
different classifiers (liner-SVM, RBF-SVM, Nearest-Neighbor, etc.). As
summarized in Fig. 12, the accuracy is positively correlated with the
ratio. Although it is not linearly proportional since the final accuracy
could also be affected by classifiers, a small value (ratio 0.75) indicates
a failure of the feature extractor to represent the dataset. Hence, for these
small ratio values, it is highly recommended to fine-tuning the entire
model (approach (c)) instead of directly using the pre-trained models.
high accuracy in two different datasets. To verify this assumption, Fig. 9
and Fig. 10 visualize the feature space projection from the last convolutional layer using UMAP [34]. The pre-trained feature extractor
represents the pellet dataset well while it fails to separate the defect
samples. It indicates the risk that the knowledge learned from the
ImageNet dataset may not be useful for other case studies. Hence, it is
necessary to verify the consistency between the pre-trained models with
the target dataset before directly using the pre-trained models. In case of
good compatibility between the pre-trained feature extractor and the
target dataset, it is possible to quickly construct a classifier by just adding
a linear classifier at the end of the pre-trained feature extractor (approach
(a)). Fine-tuning the entire network is required to address incompatibility issues between the feature extractor and data set but requires more training samples.
For the determination of compatibility various methods can be used.
As this work shows, direct visualization, using dimensionality reduction
methods, is an applicable method. Besides the qualitative verification,
distance measurements on the feature space could also be useful to
provide quantitative information. For example, the ratio between the
inter-class distances with the intra-class distances from the feature layer
outputs could be an indicator. The pairwise distance between the class
centers can be a choice of inter-class distance, and the median value of
the nearest-neighbor distance can be used for intra-class distances:
ratio ¼
Ci Cj
MedianðNNdist Þ
5.3. Necessity of pre-training
(2)
As discussed in previous sections, the success of the training highly
depends on the feature extraction parts of the model, and fine-tuning the
entire model can be necessary to achieve desired performance. Thus, it is
worth evaluating the benefit of using the pre-trained weights, particularly for approach (c). Fig. 13 summarizes the comparison of fine-tuning
the entire model with and without using pre-trained weights. The
This ratio simply quantifies the distribution of data samples in the
feature space. In general, data samples with high similarity or from the
same category are clustered together in the feature space, while samples
from different categories are separated there. A clear separation of
clusters from different categories eases the classifier to determine the
6
W. Zhu et al.
Chemometrics and Intelligent Laboratory Systems 211 (2021) 104269
Fig. 11. UMAP visualization on the feature space of dummy datasets with various ratio values: (a) 0.38, (b) 0.64, (c) 0.78 and (d) 1.5.
Fig. 13. Necessity of pre-training for the two datasets. Sample size in x coordinate indicates the number of images for each class.
Fig. 12. Correlation between the classification accuracy with the proposed ratio.
weights as the initial value in method (c), especially in the case of limited
data resources.
comparison suggests that the pre-trained weights provide remarkable
improvement when tuning the model with very few (100) training
samples, while when feeding with enough samples (200), the difference becomes minor. Therefore, it is recommended to use the pre-trained
5.4. Model selection
The results so far are tested on a ResNet-18 model. The performance
of different bare-bones models could be another important factor. Hence,
7
W. Zhu et al.
Chemometrics and Intelligent Laboratory Systems 211 (2021) 104269
lower top-1 error in the ImageNet benchmark. Since approach (a) and
approach (c) provide promising results in the previous studies, the
comparison between lightweight and large-scale models are only
considered for these two approaches, which is summarized in Fig. 14.
The strategy of model selection in approach (a) and (c) seems the
opposite. For both datasets, approach (a) using a large-scale model has
better performance than the one using a lightweight model. It is understandable because approach (a) builds the classification model on the
pre-trained feature extractor, where high-performance models can have
better feature representability. On the other hand, for approach (c) that
requires fine-tuning the entire network, the large-scale model encounters
training difficulties in the training when training samples are limited.
The accuracy of the ResNet-152 model does not show any advantage
compared with ResNet-18. Therefore, lightweight models are more
suitable for fine-tuning the entire model when a sufficient amount of
training data is not available.
5.5. Few-shot learning comparison
The results of the prototypical network on the two datasets are
summarized in Table 1, and the comparison with the best score of
transfer learning is also provided. The classification accuracy of the
prototypical network is over 10% lower than the best results of the
transfer learning approaches.
The prototypical network is not comparable with the transfer learning
approach. The poor performance of the prototypical network is not surprising. As pointed out in Section 5.2, the feature extraction part is more
important than the final classification part, and the knowledge learned
from the ImageNet dataset may not be useful for other cases. While in the
metric-based few-shot learning approach, the feature extraction parts are
learned from the ImageNet dataset and keep fixed in new tasks without
further fine-tuning. The adaptability to new tasks only depends on the
subsequent metric matching part. This could explain the failure of fewshot learning in these two applications. Therefore, the few-shot
learning approach is not a good solution, compared to the transfer
learning approach.
Fig. 14. Comparison between the lightweight model and large-scale model in
(a) pellets dataset, and (b) defects dataset. Sample size in x coordinate indicates
the number of images for each class.
Table 1
Comparison of the prototypical network with the best results in transfer learning.
Accuracy(%)
Pellet 20
shots
Pellet 40
shots
Defect 20
shots
Defect 40
shots
5.6. Practical guidelines
ProtoNet
Best in transfer
learning
74.1
90.2
77.1
91.3
61.3
85.1
66.0
90.8
Based on the experiments and results in the previous sections, we can
condense the key findings into a practical guideline, as shown in Fig. 15.
Approach (a) does not always have competitive results due to the
compatibility issue of pre-trained models, but if the pre-trained models
can properly represent the distribution of the target, it can be an
acceptable choice. The benefit in this approach is that the required
training data size is small and a good classifier can be constructed with a
the ResNet-152 model is used in the comparison, which has more parameters than the ResNet-18 model (60 M vs. 11 M) and achieves a 10%
Fig. 15. The practical guideline.
8
W. Zhu et al.
Chemometrics and Intelligent Laboratory Systems 211 (2021) 104269
small image data set using less than 50 images from each class. If the pretrained feature extractor is unable to represent the target dataset, then
approach (c) is the preferred option, where the entire model is fine-tuned
using the pre-trained weights as the initial values. The required training
data size is larger with at least 100 images for each class.
[8] A. Krizhevsky, I. Sutskever, G.E. Hinton, Imagenet classification with deep
convolutional neural networks, in: Advances in Neural Information Processing
Systems, 2012, pp. 1097–1105.
[9] K. He, X. Zhang, S. Ren, J. Sun, Delving deep into rectifiers: surpassing human-level
performance on imagenet classification, in: Proceedings of the IEEE International
Conference on Computer Vision, 2015, pp. 1026–1034.
[10] X. Liu, W. Liu, T. Mei, H. Ma, A deep learning-based approach to progressive vehicle
re-identification for urban surveillance, in: European Conference on Computer
Vision, Springer, 2016, pp. 869–884.
[11] W. Feng, R. Liu, M. Zhu, Fall detection for elderly person care in a vision-based
home surveillance environment using a monocular camera, signal, image and video
processing 8 (6) (2014) 1129–1138.
[12] N.H. Motlagh, M. Bagaa, T. Taleb, Uav-based iot platform: a crowd surveillance use
case, IEEE Commun. Mag. 55 (2) (2017) 128–134.
[13] E. Ohn-Bar, A. Tawari, S. Martin, M.M. Trivedi, On surveillance for safety critical
events: in-vehicle video networks for predictive driver assistance systems, Comput.
Vis. Image Understand. 134 (2015) 130–140.
[14] W. Zhu, Y. Ma, M. Benton, J. Romagnoli, Y. Zhan, Deep learning for pyrolysis
reactor monitoring: from thermal imaging toward smart monitoring system, AIChE
J. 65 (2) (2019) 582–591.
[15] V. Manee, W. Zhu, J. Romagnoli, A deep learning image-based sensor for real-time
crystal size distribution characterization, Ind. Eng. Chem. Res. (2019).
[16] R. Rendall, I. Castillo, B. Lu, B. Colegrove, M. Broadway, L.H. Chiang, M.S. Reis,
Image-based manufacturing analytics: improving the accuracy of an industrial
pellet classification system using deep neural networks, Chemometr. Intell. Lab.
Syst. 180 (2018) 26–35.
[17] C. Cortes, V. Vapnik, Support-vector networks, Mach. Learn. 20 (3) (1995)
273–297.
[18] J.R. Quinlan, C4. 5: Programs for Machine Learning, Elsevier, 2014.
[19] K. Simonyan, A. Zisserman, Very Deep Convolutional Networks for Large-Scale
Image Recognition, arXiv preprint, 2014 arXiv:1409.1556.
[20] K. He, X. Zhang, S. Ren, J. Sun, Identity mappings in deep residual networks, in:
European Conference on Computer Vision, Springer, 2016, pp. 630–645.
[21] A.G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand,
M. Andreetto, H. Adam, Mobilenets: Efficient Convolutional Neural Networks for
Mobile Vision Applications, arXiv preprint, 2017 arXiv:1704.04861.
[22] C. Tan, F. Sun, T. Kong, W. Zhang, C. Yang, C. Liu, A survey on deep transfer
learning, in: International Conference on Artificial Neural Networks, Springer,
2018, pp. 270–279.
[23] S.J. Pan, Q. Yang, A survey on transfer learning, IEEE Trans. Knowl. Data Eng. 22
(10) (2009) 1345–1359.
[24] I. Melekhov, J. Kannala, E. Rahtu, Siamese network features for image matching, in:
2016 23rd International Conference on Pattern Recognition (ICPR), IEEE, 2016,
pp. 378–383.
[25] O. Vinyals, C. Blundell, T. Lillicrap, D. Wierstra, et al., Matching networks for one
shot learning, in: Advances in Neural Information Processing Systems, 2016,
pp. 3630–3638.
[26] J. Snell, K. Swersky, R. Zemel, Prototypical networks for few-shot learning, in:
Advances in Neural Information Processing Systems, 2017, pp. 4077–4087.
[27] R.K. Srivastava, K. Greff, J. Schmidhuber, Highway Networks, arXiv preprint, 2015
arXiv:1505.00387.
[28] C. Szegedy, S. Ioffe, V. Vanhoucke, A.A. Alemi, Inception-v4, inception-resnet and
the impact of residual connections on learning, in: Thirty-first AAAI Conference on
Artificial Intelligence, 2017.
[29] M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, L.-C. Chen, Mobilenetv2: inverted
residuals and linear bottlenecks, in: Proceedings of the IEEE Conference on
Computer Vision and Pattern Recognition, 2018, pp. 4510–4520.
[30] X. Li, L. Yu, C.-W. Fu, M. Fang, P.-A. Heng, Revisiting metric learning for few-shot
image classification, Neurocomputing (2020).
[31] T. He, Z. Zhang, H. Zhang, Z. Zhang, J. Xie, M. Li, Bag of tricks for image
classification with convolutional neural networks, in: Proceedings of the IEEE
Conference on Computer Vision and Pattern Recognition, 2019, pp. 558–567.
[32] K. He, X. Zhang, S. Ren, J. Sun, Spatial pyramid pooling in deep convolutional
networks for visual recognition, IEEE Trans. Pattern Anal. Mach. Intell. 37 (9)
(2015) 1904–1916.
[33] J. Yosinski, J. Clune, Y. Bengio, H. Lipson, How Transferable Are Features in Deep
Neural Networks? arXiv preprint, 2014 arXiv:1411.1792.
[34] L. McInnes, J. Healy, J. Melville, Umap: Uniform Manifold Approximation and
Projection for Dimension Reduction, arXiv preprint, 2018 arXiv:1802.03426.
6. Conclusion
This work investigates different implementations of transfer learning
and aims at searching the optimal methodology for landing deep learning
models in industrial scenarios where data might be limited. Two datasets
from manufacturing processes were collected to evaluate three transfer
learning approaches. The results suggest that the success of the transfer
learning highly depends on the feature extraction parts of the model, and
verification of the feature extractor with the target dataset is the key step
before adopting transfer learning. With the correct implementation of
transfer learning, the data sample requirement for building a DL based
classifier can be dramatically reduced, where the DL model can achieve
over 90% accuracy by using less than 100 images in each class. Besides
that, few-shot learning is also tested in this work as a competitor, which
indicates the superiority of the transfer learning approach. A practical
guideline is summarized to help future development of DL based image
classification models for the scenarios with limited data resources.
CRediT authorship contribution statement
Wenbo Zhu: Writing - original draft. Birgit Braun: Project administration. Leo H. Chiang: Supervision. Jose A. Romagnoli: Writing review & editing.
Declaration of competing interest
The authors declare that they have no known competing financial
interests or personal relationships that could have appeared to influence
the work reported in this paper.
References
[1] A. Voulodimos, N. Doulamis, A. Doulamis, E. Protopapadakis, Deep learning for
computer vision: a brief review, Comput. Intell. Neurosci. 2018 (2018).
[2] A. Brunetti, D. Buongiorno, G.F. Trotta, V. Bevilacqua, Computer vision and deep
learning techniques for pedestrian detection and tracking: a survey,
Neurocomputing 300 (2018) 17–33.
[3] A. Esteva, B. Kuprel, R.A. Novoa, J. Ko, S.M. Swetter, H.M. Blau, S. Thrun,
Dermatologist-level classification of skin cancer with deep neural networks, nature
542 (7639) (2017) 115–118.
[4] M.D. Zeiler, R. Fergus, Visualizing and understanding convolutional networks, in:
European Conference on Computer Vision, Springer, 2014, pp. 818–833.
[5] Y. Lin, F. Lv, S. Zhu, M. Yang, T. Cour, K. Yu, L. Cao, T. Huang, Large-scale image
classification: fast feature extraction and svm training, in: CVPR 2011, IEEE, 2011,
pp. 1689–1696.
[6] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang,
A. Karpathy, A. Khosla, M. Bernstein, et al., Imagenet large scale visual recognition
challenge, Int. J. Comput. Vis. 115 (3) (2015) 211–252.
[7] J. S
anchez, F. Perronnin, High-dimensional signature compression for large-scale
image classification, in: CVPR 2011, IEEE, 2011, pp. 1665–1672.
9