minerals
Article
An Enhanced Rock Mineral Recognition Method
Integrating a Deep Learning Model
and Clustering Algorithm
Chengzhao Liu 1 , Mingchao Li 1, * , Ye Zhang 1 , Shuai Han 1 and Yueqin Zhu 2
1
2
*
State Key Laboratory of Hydraulic Engineering Simulation and Safety, Tianjin University,
Tianjin 300354, China
Development Research Center of China Geological Survey, Beijing 100037, China
Correspondence: lmc@tju.edu.cn
Received: 26 May 2019; Accepted: 23 August 2019; Published: 26 August 2019
Abstract: Rock mineral recognition is a costly and time-consuming task when using traditional
methods, during which physical and chemical properties are tested at micro- and macro-scale in
the laboratory. As a solution, a comprehensive recognition model of 12 kinds of rock minerals
can be utilized, based upon the deep learning and transfer learning algorithms. In the process,
the texture features of images are extracted and a color model for rock mineral identification can also
be established by the K-means algorithm. Finally, a comprehensive identification model is made by
combining the deep learning model and color model. The test results of the comprehensive model
reveal that color and texture are important features in rock mineral identification, and that deep
learning methods can effectively improve identification accuracy. To prove that the comprehensive
model could extract effective features of mineral images, we also established a support vector machine
(SVM) model and a random forest (RF) model based on Histogram of Oriented Gradient (HOG)
features. The comparison indicates that the comprehensive model has the best performance of all.
Keywords: rock mineral; classification recognition; deep learning model; transfer learning model;
clustering algorithm
1. Introduction
The classification and identification of rock minerals are indispensable in geological works.
The traditional recognition methods are based on the physical and chemical properties of rock minerals,
which are used to identify rock minerals at macro- and micro-scales [1,2]. With the development of
computer science and artificial intelligence, the recognition model can be established using machine
learning methods [3–7]. Traditional recognition methods have definite physical meaning; while machine
learning methods are driven by data [8,9]. The two methods have their own strengths and weaknesses.
The significant strength of traditional identification model is that the model can be explained.
Vassilev and Vassileva [10,11] analyzed the optical properties of low-temperature ash (LTA) and
high-temperature ash (HTA) of the coal samples to get the weight of each mineral in the crystalline
matter basis. Then they classified the inorganic matter in coal using the composition from the HTA
and got a good result. But the experiment environment and operating procedures of this method
are difficult to implement. Zaini [12] used wavelength position, linear spectral un-mixing and a
spectral angle mapper to explore carbonate mineral compositions and distribution on the rock mineral
surface and got a good result. Moreover, Zaini’s method needs SisuCHEMA SWIR sensor, which is
integrated with a computer workstation to collect the data. Adep [13] developed an expert system
for hyperspectral data classification with neural network technology. Their system was also used to
Minerals 2019, 9, 516; doi:10.3390/min9090516
www.mdpi.com/journal/minerals
Minerals 2019, 9, 516
2 of 17
calculate the depth and distribution of iron ore in a specific engineering project. The scholars chose the
key factors to build the model. However, the professional knowledge and equipment are necessary for
the process. It is not an efficient and economical way to conduct traditional model sometimes.
In recent years, machine learning methods have been applied in image recognition, including
sensing images [14,15] and mineral images [16]. Multi-source and multi-type mineral datasets are
suitable for machine learning methods [17,18]. Moreover, the application of machine learning in the
identification of rock images has been proved useful and efficient [19–23]. Aligholi et al. [24] classified
45 slices of 15 kinds of rock minerals automatically based on the color of the rock minerals and achieved
high accuracy. Li et al. [25] applied feature selection and transfer learning to classify microscopic
images of sandstone. The feature engineering is significant in machine learning model establishment.
However, it is easy to be influenced by subjective factors. As a result, it is necessary to develop an
end-to-end mineral identification model.
At present, deep learning algorithms have been developed and studied widely. Google introduced
the design and application of a deep learning framework, which was flexible to build deep learning
models [26]. Moreover, deep learning methods have achieved a remarkable prominence in recognition
of rock mineral images. Zhang et al. [27] retrained Inception-v3 to identify granite, phyllite and breccia,
and thereby proved that the Inception-v3 model was suitable for rock identification. The neuro-adaptive
learning algorithm was also used to classify the iron ore real-timely [28]. Lots of researchers studied
the application of deep learning in geochemical mapping [29], mineral prospectivity mapping [30] and
micro-scale images of quartz and resin [31]. It proves that deep learning models are outstanding in
complex image data processing and analysis. While, it is hard to improve deep learning model, because
the deep learning architecture is fixed. The model ensemble can be adopted to solve the problem.
In this paper, a comprehensive model for rock mineral identification was developed by combining
the Inception-v3 model and a color model. First, as shown in Section 3.1, the texture features of
rock minerals in the image were extracted based on the color and brightness of rock minerals. Then,
the Inception-v3 model was retrained with these preprocessed images. Moreover, a color model was
proposed (as shown in Section 3.2) to recognize the mineral images based on the result of Inception-v3
model. Finally, the comprehensive identification model based on the deep learning model and color
model can be established (as shown in Section 4). We also constructed two models using SVM
and RF-based on Histogram of Oriented Gradient (HOG) features. The comparison of the three
models indicates that the comprehensive model can achieve significant performance. In this research,
the elaborate mineral images are adopted, whose features are obvious. We want to study whether the
comprehensive model can extract the obvious features, such as cleavage and luster. If effective features
can be extracted, the model can be used in a larger domain.
2. Methodology
This research is based on the deep learning method, transfer learning and clustering algorithm.
The overall of the method is shown in Figure 1. The process can be divided into two parallel ones. One
of them cut the raw images (images without any pre-process) into pieces. Then T1 is calculated, and the
color model is established using these image slices. T1 is broadcasted to the other process at the same
time. The other process will calculate and outline the binary points in the raw images once received T1 .
Then the processed images are going to be trained using the transfer learning method based on deep
learning method. Finally, a comprehensive model is established, combining the Inception-v3 model
and color model. T and T1 are thresholds defined in Section 3.1.
Minerals 2019, 9, 516
3 of 17
Minerals 2019, 9, x FOR PEER REVIEW
3 of 16
Raw image data
Mineral image
slices
Calcula te the color
value of each point
Calcula te the gray
value of each point
K-mea ns
calculation
T1
If the value changes
more than T1, set the
point as binary point
If the value changes
more than T, set the
point as binary point
Preprocesse d
image data
Transfe r learning
Comprehensive model
Inception-v3
model
Image data
Identific ation by
Inception-v3 model
Color
model
Results from
Inception-v3
model
Comprehensive
result
Identific ation based
on color model and
results from
Inception-V3 model
Figure 1.
1. The
The training
training process
process and
and identification
identification process
process of a comprehensive model.
Figure
2.1. Deep Learning Algorithm
2.1. Deep Learning Algorithm
Deep learning was proposed by Hinton and Salakhutdinov [32] and has been widely applied
Deep learning was proposed by Hinton and Salakhutdinov [32] and has been widely applied in
in semantic recognition [33], image recognition [34], object detection [35], and the medical field [36].
semantic recognition [33], image recognition [34], object detection [35], and the medical field [36]. In
In image recognition, the end-to-end training mode is adopted in the training process. In each iteration
image recognition, the end-to-end training mode is adopted in the training process. In each iteration
of a training process, each layer can adjust its parameters according to the feedback from training data
of a training process, each layer can adjust its parameters according to the feedback from training
to improve the accuracy of the model. Moreover, the nonlinear function mapping ability is added to
data to improve the accuracy of the model. Moreover, the nonlinear function mapping ability is
the deep learning model. As a consequence, it reduces the computational time for extracting features
added to the deep learning model. As a consequence, it reduces the computational time for
from images and improves the accuracy of the deep learning model.
extracting features from images and improves the accuracy of the deep learning model.
In this paper, the Inception-v3 model [37] was set as a pre-trained model. In the Inception-v3
In this paper, the Inception-v3 model [37] was set as a pre-trained model. In the Inception-v3
model, there are five convolutional layers and two max-pooling layers in front of the net structure.
model, there are five convolutional layers and two max-pooling layers in front of the net structure.
To reduce the computation cost, the convolution kernel is converted from one 5 × 5 matrix to two
To reduce the computation cost, the convolution kernel is converted from one 5 × 5 matrix to two 3 ×
3 × 3 matrixes. Then, 11 mixed layers follow. These mixed layers could be divided into three categories,
3 matrixes. Then, 11 mixed layers follow. These mixed layers could be divided into three categories,
named block module 1, block module 2 and block module 3. Block module 1 has three repetitive mixed
named block module 1, block module 2 and block module 3. Block module 1 has three repetitive
layers, and each layer is connected by a contact layer. The input size of first mixed layer is (35,35,192),
mixed layers, and each layer is connected by a contact layer. The input size of first mixed layer
and output size is (35,35,256). The rest mixed layers’ input, and output size are (35,35,288) in the block
is(35,35,192), and output size is (35,35,256). The rest mixed layers’ input, and output size are
module 1. Batch size is the number of training samples in each step. In the front of block module 2,
(35,35,288) in the block module 1. Batch size is the number of training samples in each step. In the
there is a small mixed layer, whose input size is (35,35,288) and the output size is (17,17,768). The rest
front of block module 2, there is a small mixed layer, whose input size is (35,35,288) and the output
of the mixed layers’ input and output size are (17,17,768) in the block module 2. In block module 3,
size is (17,17,768). The rest of the mixed layers’ input and output size are (17,17,768) in the block
the input size is (17,17,768) and output size is (8,8,2048). After the mixed layer, a convolutional layer,
module 2. In block module 3, the input size is (17,17,768) and output size is (8,8,2048). After the
mean pooling layer and dropout layer is followed to extracted image features. Finally, the softmax
mixed layer, a convolutional layer, mean pooling layer and dropout layer is followed to extracted
classifier is trained at the end of the net.
image features. Finally, the softmax classifier is trained at the end of the net.
In the process of convolution, a color image is converted into a 3D matrix including all RGB
values; then, each pixel is evaluated using the kernel and pooling during each iteration. The average
pooling layer and max-pooling layer could get the average value and max value of a matrix,
Minerals 2019, 9, 516
4 of 17
Minerals 2019, 9, x FOR PEER REVIEW
4 of 16
In the process of convolution, a color image is converted into a 3D matrix including all RGB values;
then,
each pixel
is evaluated
using the
kernelcan
andbe
pooling
during
each
iteration.
Theof
average
pooling
respectively.
Finally,
the common
features
extracted
from
a large
number
images.
In the
layer
and
max-pooling
layer
could
get the
average
value andwhich
max value
a matrix,
Finally,
mixed
layer,
convolution
and
pooling
are
synchronized,
is ableofto
achieverespectively.
better performance
the
common
features can be extracted from a large number of images. In the mixed layer, convolution
in feature
extraction.
and pooling are synchronized, which is able to achieve better performance in feature extraction.
2.2. Transfer Leaning Method
2.2. Transfer Leaning Method
According to transfer learning, the parameters in trained models from clustering domains can
According
to transfer
learning,
parameters
trained
fromrecognition,
clustering domains
can be
be utilized
to establish
a new
modelthe
efficiently.
In in
a new
taskmodels
of images
the parameters
utilized
to establish
a new
a new task Transfer
of imageslearning
recognition,
the parameters
of a
of a pre-trained
model
aremodel
set asefficiently.
the initialInparameters.
can help
scholars avoid
pre-trained
model
are
set
as
the
initial
parameters.
Transfer
learning
can
help
scholars
avoid
building
a
building a model from scratch and make full use of the data, which reduces computational cost and
model
from
scratch
and
make
full
use
of
the
data,
which
reduces
computational
cost
and
dependence
dependence on big datasets.
on bigIndatasets.
this research, the Inception-v3 model was selected as the pre-trained model for rock mineral
In recognition.
this research,The
the pre-trained
Inception-v3deep
model
was selected
the pre-trained
model for rockand
mineral
image
learning
model as
is able
to reduce computational
time
image
recognition.
The
pre-trained
deep
learning
model
is
able
to
reduce
computational
and
time
cost when a new model is trained. Therefore, transfer learning based on a deep learning modelcost
has
when
a new model
is trained.
transfer
based onmodel
a deepcontains
learningabout
model1.2
has
been
been widely
used [38,39].
TheTherefore,
original data
set oflearning
the Inception-v3
million
widely
The 1000
original
data set The
of the
Inception-v3
about
1.2 million
images
imagesused
from[38,39].
more than
categories.
model
involvesmodel
about contains
25 million
parameters.
However,
from
than 1000
categories.
Thethe
model
involves
about
25 million
However,
the
with more
the transfer
learning
method,
training
process
based
on thisparameters.
model can be
quickly with
finished
transfer
learning
method,
the
training
process
based
on
this
model
can
be
quickly
finished
even
though
even though the computer has no GPU in general. Figure 2 shows the process of transfer learning
the
computer has no GPU in general. Figure 2 shows the process of transfer learning method.
method.
Images of
ImageNet
Convolution layers
and pooling layers
Labels of
ImageNet
Softmax
Transfer learning
Rock mineral
images
Feature
extraction
Features of rock
mineral images
Softmax
Result of rock mineral
image classification
Figure2.
2. Transfer
Transferlearning
learningwith
withInception-v3.
Inception-v3.
Figure
2.3.
2.3. Clustering
Clustering Algorithm
Algorithm
The
The clustering
clustering algorithm
algorithm has
has aa mass
mass application
application in
in pattern
pattern recognition
recognition (PR)
(PR) and
and is
is based
based on
on the
the
Euclidian
Distance.
Absolute
distance
is
the
main
point
of
Euclidian
Distance,
which
measures
Euclidian Distance. Absolute distance is the main point of Euclidian Distance, which measures the
the
clustering
of of
datasets
by absolute
distance.
Reasonably,
a rankawill
be will
proposed
by the distance
clusteringofoflots
lots
datasets
by absolute
distance.
Reasonably,
rank
be proposed
by the
size
of thesize
dataset.
distance
of the dataset.
In
thethe
color
model
is established
based on
the K-means
algorithm.algorithm.
In the algorithm,
Inthis
thisresearch,
research,
color
model
is established
based
on the K-means
In the
the
absolute
distances
of
objects
are
calculated,
and
the
objects
that
share
the
closest
absolute
distances
algorithm, the absolute distances of objects are calculated, and the objects that share the closest
are
in the distances
same category.
a silhouette
used to evaluate
theisclustering
effect: the
absolute
are inFinally,
the same
category.coefficient
Finally, a is
silhouette
coefficient
used to evaluate
clustering effect:
b𝑏i −−a𝑎i
𝑖
𝑖 .
Si =
(1)
𝑆𝑖 = max(ai , bi ) .
(1)
𝑚𝑎𝑥( 𝑎𝑖 , 𝑏𝑖 )
In Equation
Equation (1),
(1), aai is
is the
the average
average distance
distance from
from the
the ith
ith point
point to
to other
other points
points in
in the
the same
same kind,
kind,
In
i
which
is
called
intra-cluster
dissimilarity;
b
i is the average distance from the ith point to the points in
which is called intra-cluster dissimilarity; bi is the average distance from the ith point to the points
any
kind,
which
is called
inter-cluster
dissimilarity.
When
the the
number
of kinds
is more
thanthan
3, the
in
any
kind,
which
is called
inter-cluster
dissimilarity.
When
number
of kinds
is more
3,
average
value
is
set
as
a
silhouette
coefficient.
The
silhouette
coefficient
is
in
[−1,
1].
In
the
training
the average value is set as a silhouette coefficient. The silhouette coefficient is in [−1, 1]. In the training
colormodel,
model,the
themodel
modelreceiving
receivingthe
themaximum
maximumsilhouette
silhouettecoefficient
coefficientisisgoing
goingtotobe
beestablished.
established.
color
Minerals 2019, 9, 516
5 of 17
2.4. Support Vector Machine
Support Vector Machine (SVM) is a supervised learning method. When given a binary classification
N
D
dataset xn , yn N
n=1 with xn ∈ R and yn ∈ [−1, 1], for each (xn , yn ) belongs to xn , yn n=1 . An SVM
model can be established by the equation:
f (x) = ωT x + b.
(2)
The optimal weight vector wT and bias b are obtained by the Hinge Loss Function [40]. In this
research, the Histogram of Oriented Gradient (HOG) algorithm was used to extract the feature of rock
mineral images and then an SVM model was trained with these features.
2.5. Random Forest
Random Forest (RF) is proposed by Leo Breiman. The steps of establishing an RF model are
as following:
Step 1: Input T as training dataset size and M as feature numbers.
Step 2: Input the feature number of each node (m, m << M) to get the decision result at a node in
the tree.
Step 3: Select K samples with playback from a dataset N randomly. Then, K taxonomic trees are
constructed by these selected samples. The rest of the dataset is used to test the model.
Step 4: Randomly select m features at each node and find the best splitting mode by the
selected features.
Step 5: Each tree grows completely without pruning and may be adopted after a normal tree
classifier is built.
3. Algorithm Implementation
Rock minerals have unique physical properties, such as color, streak, transparency, aggregate
shape and luster [41]. By analyzing their properties, rock minerals can be identified more accurately.
Table 1 shows the properties of 12 kinds of rock minerals.
Table 1. Twelve kinds of rock mineral’s physical properties.
Minerals
Color
Streak
Transparency
Aggregate Shape
Luster
Cinnabar
Red
Red
Translucent
Granule, massive form
Hematite
Red
Cherry red
Opaque
Adamantine luster
From metallic luster to
soil state luster
Calcite
Colorless or
white
White
Translucent
Malachite
Green
Pale green
From
translucent to
opaque
Azurite
Navy blue
Pale blue
Opaque
Aquamarine
Determine by
its component
White
Transparent
Augite
Black
Magnetite
Black
Pale green,
black
Black
Molybdenite
Gray
Light gray
Opaque
Stibnite
Gray
Dark gray
Opaque
Cassiterite
Crineus, yellow,
black
White, pale
brown
Gyp
White, colorless
White
From opaque to
transparent
Transparent,
translucent
Opaque
Opaque
Multi-form
Granule, massive form,
threadiness, stalactitic
form, soil state
Emulsions, massive,
incrusting, concretion
forms or threadiness
Granule, stalactitic,
incrusting, soil state
Cluster form
Granule, radial pattern,
massive
Granule, massive form
Clintheriform, scaly
form or lobate form
Massive, granule or
radial pattern
Irregular granule
Clintheriform, massive,
threadiness
Glassy luster
Waxy luster, glassy
luster, soil state luster
Glassy luster
Glassy luster
Glassy luster
Metallic luster
Metallic luster
Strong metallic luster
Adamantine luster, sub
adamantine luster
Glassy luster, nacreous
luster
Minerals
Minerals 2019,
2019, 9,
9, 516
x FOR PEER REVIEW
of 17
66 of
16
In Table 1, color is a key feature of rock minerals in rock mineral image recognition, because
In Table 1, color is a key feature of rock minerals in rock mineral image recognition, because
different compositions of rock minerals lead to different colors. Also, the composition of different
different compositions of rock minerals lead to different colors. Also, the composition of different rock
rock minerals usually changes, and the colors vary widely. Some rock minerals with a single
minerals usually changes, and the colors vary widely. Some rock minerals with a single composition
composition have a pure color. For example, cinnabar is usually red, malachite is green, and calcite is
have a pure color. For example, cinnabar is usually red, malachite is green, and calcite is white.
white. A rock mineral often shows a mixed color on the surface because of mixed composition,
A rock mineral often shows a mixed color on the surface because of mixed composition, which is often
which is often indicated by binomial nomenclature, such as lime-green. However, whether a
indicated by binomial nomenclature, such as lime-green. However, whether a particular type of rock
particular type of rock mineral has a pure or mixed composition, those in that classification always
mineral has a pure or mixed composition, those in that classification always share some certain kinds
share some certain kinds of colors. Therefore, color is an important classification criterion.
of colors. Therefore, color is an important classification criterion.
The texture is also an important feature for rock mineral image recognition. The main
The texture is also an important feature for rock mineral image recognition. The main components
components of rock mineral texture are aggregate shape and cleavage. Different rock minerals have
of rock mineral texture are aggregate shape and cleavage. Different rock minerals have different
different distinctive textures. For example, cinnabar’s aggregate is granular, with perfect cleavage
distinctive textures. For example, cinnabar’s aggregate is granular, with perfect cleavage and uneven
and uneven to sub-conchoidal fracture; aquamarine’s aggregate is cluster form, with imperfect
to sub-conchoidal fracture; aquamarine’s aggregate is cluster form, with imperfect cleavage and
cleavage and conchoidal to irregular fracture; malachite’s aggregate is multi-form, as shown in
conchoidal to irregular fracture; malachite’s aggregate is multi-form, as shown in Figure 3. The texture
Figure 3. The texture is also helpful in recognizing rock minerals. Therefore, the combination of color
is also helpful in recognizing rock minerals. Therefore, the combination of color and texture can
and texture can improve the accuracy of the rock mineral recognition model.
improve the accuracy of the rock mineral recognition model.
(a)
(b)
(c)
Figure
ofof
rock
mineral
characteristics
[42].
(a)(a)
Aquamarine;
(b)(b)
cinnabar;
(c)(c)
hematite.
Figure3.3.Samples
Samples
rock
mineral
characteristics
[42].
Aquamarine;
cinnabar;
hematite.
3.1.
3.1. Mineral
Mineral Texture
Texture Feature
Feature Extraction
Extraction
The
The visual
visual features
features of
of rock
rock minerals
minerals also
also include
include the
the texture
texture of
of rock
rock mineral
mineral aggregates
aggregates and
and
cleavage.
In
light
conditions,
the
reflection
of
different
cleavages
is
different.
Meanwhile,
the
reflection
cleavage. In light conditions, the reflection of different cleavages is different. Meanwhile, the
of
a given of
rock
mineral
follows
a pattern
ofbecause
its own of
characteristics.
Therefore, Therefore,
the texturethe
of
reflection
a given
rock
mineral
followsbecause
a pattern
its own characteristics.
rock
minerals
can
be
extracted
based
on
the
brightness
and
color
variation
of
images.
texture of rock minerals can be extracted based on the brightness and color variation of images.
In
In an
an area
area where
where the
the brightness
brightness changes
changes significantly
significantly in
in the
the image,
image, the
the rock
rock mineral
mineral texture
texture can
can
be
outlined.
According
to
the
RGB
color
system,
color
images
can
be
converted
to
grayscale
be outlined. According to the RGB color system, color images can be converted to grayscale to
to show
show
brightness
brightness variations
variations more
more effectively.
effectively.The
Theconversion
conversionrule
rule[43]
[43]isisas
asfollows:
follows:
(3)
𝑔𝑟𝑎𝑦 = 0.299 × 𝑅 + 0.587 × 𝐺 + 0.114 × 𝐵,
gray = 0.299 × R + 0.587 × G + 0.114 × B,
(3)
where, gray is the grayscale value of a pixel, and R, G and B are the three primary color values of the
pixel. The
value
of thevalue
image
be calculated
by Equation
In primary
the lightcolor
condition,
where,
graygray
is the
grayscale
ofcan
a pixel,
and R, G and
B are the(3).
three
valuesthe
of
brightness
in the
area
a rock
image by
is distinctly
different
of the
the
pixel. The
grayopaque
value of
theofimage
canmineral
be calculated
Equation (3).
In thefrom
lightthat
condition,
surrounding
pixels.
the rockarea
mineral’s
luster
is brilliant
in the
image, some
pixels from
achieve
a higher
the
brightness
in theIf opaque
of a rock
mineral
image
is distinctly
different
that
of the
brightness value
than
thatrock
of their
surrounding
while
of the
pixels
do achieve
not. If the
luster
surrounding
pixels.
If the
mineral’s
luster isarea,
brilliant
inthe
the rest
image,
some
pixels
a higher
of
a
rock
mineral
is
dull
in
the
image,
which
indicates
that
it
has
a
low
reflection,
pixels
achieve
brightness value than that of their surrounding area, while the rest of the pixels do not. If the luster ofa
value
their surrounding
pixels,
in Figure
4. pixels achieve a lower
alower
rock brightness
mineral is dull
inthan
the image,
which indicates
thatasitshown
has a low
reflection,
brightness value than their surrounding pixels, as shown in Figure 4.
Minerals 2019, 9, 516
Minerals 2019, 9, x FOR PEER REVIEW
7 of 17
7 of 16
(a)
(b)
(c)
(d)
Figure
images. (a)
(a) Calcite;
Calcite;(b)
(b)brightness
brightnessdistribution
distributionof
Figure4.4.The
Thebrightness
brightnessdistribution
distribution of
of rock
rock mineral images.
ofcalcite
calciteimage;
image;(c)
(c)cinnabar;
cinnabar;(d)
(d)brightness
brightnessdistribution
distributionofofcinnabar
cinnabarimage.
image.
the same
same light
brightness
value
fluctuates
around
a fixed
and theand
changed
InInthe
lightcondition,
condition,the
the
brightness
value
fluctuates
around
a value,
fixed value,
the
value
of
the
fluctuation
is
set
as
The
brightness
value
dramatically
changes
if
it
is
beside
|∆Z
|.
i as |Δ𝑍𝑖 |. The brightness value dramatically changes if it the
changed value of the fluctuation is set
is
texture
and
the
value
is
far
more
than
variation
a pixel
is calculated:
|∆Zmore
|∆Zi |. The
i |, namely
beside the texture and the value is far
than |∆Z|
|Δ𝑍𝑖>>
|, namely
|Δ𝑍|
>> |Δ𝑍𝑖 |.forThe
variation
for a
pixel is calculated:
∆Zi = grayi − gray,
Δ𝑍𝑖 =
𝑔𝑟𝑎𝑦𝑖 − 𝑔𝑟𝑎𝑦,
(4)
(4)
where,
𝑖 iisisthe
where,𝛥𝑍
∆Z
thevariation
variationofofbrightness
brightnessfor
foreach
eachpixel
pixeland
andthe
thegrayscale
grayscalevalue
valuefor
forpixel
pixelpoints
pointsisis
determined
by
Equation
(3).
Moreover,
gray
is
the
mean
value
of
the
grayscale
of
the
central
pixel
determined by Equation (3). Moreover, gray is the mean value of the grayscale of the central pixel
and
and
8
pixels
around
the
central
pixel.
In
order
to
eliminate
edge
influence,
the
maximum
and
8 pixels around the central pixel. In order to eliminate edge influence, the maximum and minimum are
minimum
removed,
as shown (5)
in Equations
removed,are
as shown
in Equations
and (6). (5) and (6).
n 9
gray  ( grayi  graymax  graymin ) / 7 ,
(5)
n
=9
X
i 1
gray = (
grayi − graymax − gray7min )/7,
(5)
1
i=1|
|Δ𝑍𝑖 |′ = ||Δ𝑍
(6)
𝑖 − |Δ𝑍|| = ||Δ𝑍𝑖 | − ∑|Δ𝑍𝑘 ||,
7
𝑘=1
7
1 X 𝑔𝑟𝑎𝑦
where, the grayi is the value|∆Z
ofi |0the
ith i |pixel’s
and 𝑔𝑟𝑎𝑦𝑚𝑖𝑛 are the
= |∆Zi | − value,|∆Z
(6)
= |∆Z
− |∆Z| grayscale
k | , 𝑚𝑎𝑥
7 calculated,
maximum and minimum grayscale values of the pixels being
|𝛥𝑍 |′ is the variation of
k =1
𝑖
the
ith grayscale
is selected
as pixel’s
the threshold
value.
After
experimentation,
we found that a
where,
the grayi pixel.
is the T
value
of the ith
grayscale
value,
gray
max and graymin are the maximum
′
0
value
of
T
=
15
was
suitable.
If
the
point
satisfies
|Δ𝑍
|
>
T,
it
will
be
set
as
a
feature
point.
𝑖
and minimum grayscale values of the pixels being calculated,
|∆Zi | is the variation of the ith grayscale
In Taddition,
theastexture
featuresvalue.
couldAfter
be extracted
from color
at the
interface
pixel.
is selected
the threshold
experimentation,
wechanges
found that
a value
of T =of15rock
was
0
minerals.
Therefore,
the
color
change
at
the
interface
of
a
rock
mineral
could
be
measured
by
the
suitable. If the point satisfies |∆Zi | > T, it will be set as a feature point.
variation of RGB proportion. Three linearly independent indexes, namely C1, C2 and C3 were
selected, which were calculated in Equation (7). Moreover, n was set as 0.01 to avoid a denominator
of zero.
𝐶1 = 𝑅/(𝐺 + 𝑛), 𝐶2 = 𝐺/(𝐵 + 𝑛), 𝐶3 = 𝐵/(𝑅 + 𝑛),
(7)
Minerals 2019, 9, 516
8 of 17
In addition, the texture features could be extracted from color changes at the interface of rock
minerals. Therefore, the color change at the interface of a rock mineral could be measured by the
variation of RGB proportion. Three linearly independent indexes, namely C1 , C2 and C3 were selected,
which were calculated in Equation (7). Moreover, n was set as 0.01 to avoid a denominator of zero.
C1 = R/(G + n), C2 = G/(B + n), C3 = B/(R + n),
(7)
where, C was calculated to evaluate the variation of C1 , C2 and C3 comprehensively, as shown in
Equation (7).
p
C = K C1 2 + C2 2 + C3 2 ,
(8)
where, K is the expansion coefficient, and in this research, K was set to 1. C will change sharply when
any one of C1 , C2 and C3 changes. ∆C was set as the maximum difference in coefficients:
∆C = max{∆Ci },
(9)
where, ∆Ci wasMinerals
the 2019,
difference
the pixel and the surrounding pixels, and
1 was used as a
9, x FOR PEERbetween
REVIEW
8 ofT16
threshold value to determine whether the point was a texture boundary point; and T’ was a matrix,
where, C was calculated to evaluate the variation of C1, C2 and C3 comprehensively, as shown in
which was used
to store
Equation
(7). the maximum distances within each class. T 1 indicated the max distance
between the 12 rock minerals, which was
contains the
(8)
𝐶 = calculated
𝐾√𝐶1 2 + 𝐶2 2 + by
𝐶3 2 , K-means. First, a matrix which
RGB values of every
of rock mineral
image
calculated
algorithm.
where, Ktype
is the expansion
coefficient, and
in thiswas
research,
K was set toby
1. CK-means
will change sharply
when Then, a result
any one ofthe
C1, Cmax
2 and C3 changes. Δ𝐶 was set as the maximum difference in coefficients:
matrix that contained
distance for these 12 kinds of rock mineral was assigned to T’:
Δ𝐶 = 𝑚𝑎𝑥{ 𝛥 𝐶𝑖 },
(9)
where, Δ𝐶𝑖 was the difference between
the pixel and the surrounding pixels, and T1 was used as a
T0 = {D1 ,point
D2 , was
. . . ,aD
, Dm }, point; and T’ was a matrix,
(10)
i , . . . boundary
threshold value to determine whether the
texture
which was used to store the maximum distances within each class. T1 indicated the max distance
the 12 rockin
minerals,
which was calculated
by K-means.
a matrix
contains
the
where, Di is thebetween
max distance
each category
from any
point’s First,
C value
to which
the mean
colors’
RGB values;
RGB values of every type of rock mineral image was calculated by K-means algorithm. Then, a result
and m is the number
of
mineral
classes.
Finally,
the
maximum
value
in
T’
was
assigned
to
T
1 . When T 1
matrix that contained the max distance for these 12 kinds of rock mineral was assigned to T’:
is smaller than ∆C, the pixel point will be𝑇 ′ marked
boundary point; and
= {𝐷1 ,D2 , . . . as
, 𝐷𝑖 , a
. . . texture
, 𝐷𝑚 },
(10) in the contrary
case, it is not going
marked.
Theintexture
and from
cleavage
canCbe
outlined
in the
image
where,toDibe
is the
max distance
each category
any point’s
value
to the mean
colors’
RGB by combining
values;color
and m is
the number
of mineral
classes. Finally,
the maximum
valuemineral
in T’ was assigned
to Tcan
1.
the brightness and
change.
The
recognition
accuracy
of rock
images
be increased
When T1 is smaller than 𝛥𝐶, the pixel point will be marked as a texture boundary point; and in the
with the extraction
of texture and the cleavage of rock minerals.
contrary case, it is not going to be marked. The texture and cleavage can be outlined in the image by
In a word,combining
the feature
of rock
images
can be
extracted
by calculating
the brightness
and mineral
color change.
The recognition
accuracy
of rock mineral
images can be the changes in
increased
the values.
extraction of
the cleavage
of rock features
minerals. of rock minerals were extracted
brightness values
and with
color
Intexture
this and
research,
texture
In a word, the feature of rock mineral images can be extracted by calculating the changes in
to strengthen the
texture
features
the Inoriginal
rock
mineral
by marking
and outlining the
brightness
values
and colorof
values.
this research,
texture
featuresimages
of rock minerals
were extracted
to strengthen
features
of the original
rock mineral
images by Inception-v3
marking and outlining
the and the color
texture boundary
points,theastexture
shown
in Figure
5. After
extraction,
model
texture boundary points, as shown in Figure 5. After extraction, Inception-v3 model and the color
model will be trained.
model will be trained.
(a)
(b)
(c)
(d)
Figure 5. Cont.
Minerals 2019, 9, 516
9 of 17
Minerals 2019, 9, x FOR PEER REVIEW
9 of 16
(e)
(f)
Figure 5. Samples of texture extraction for rock mineral images. (a) Cinnabar; (b) texture extraction of
Figure 5. Samples
of texture extraction for rock mineral images. (a) Cinnabar; (b) texture extraction of
cinnabar; (c) calcite; (d) texture extraction of calcite; (e) malachite; (f) texture extraction of malachite.
cinnabar; (c) calcite; (d) texture extraction of calcite; (e) malachite; (f) texture extraction of malachite.
3.2. Color Model of Rock Mineral
3.2. Color Model of Rock
Mineral
The colors
of the 12 types of rock minerals considered here are shown in Table 1. A computer
generated the color through combining red, green, and blue to build RGB color space, in which red,
The colorsgreen,
of the
of rock
considered
here (226
are ×shown
Table 1. A computer
and12
bluetypes
have values
in [0, minerals
255]. As a result,
there are 1,677,216
256 × 256 in
= 1,677,216)
colors
in
RGB
color
space.
Any
change
of
the
value
in
R,
G
or
B
will
lead
to
color
variation.
It is in which red,
generated the color through combining red, green, and blue to build RGB color space,
possible to implement recognition for rock mineral images based on the feature of color. The rock
green, and bluemineral
havelabel
values
[0, 255].
a result,
there are 1,677,216 (226 × 256 × 256 = 1,677,216)
can bein
determined
by As
the following
equation:
2
2
2
(11) variation. It is
𝑆ji = 𝜌ji 2 =of
(𝑅the
R
)
+
(𝐺
G𝑗𝑖 )R,
+ (𝐵
, will lead to color
colors in RGB color space. Any change
value
G𝑖 -B
or𝑗𝑖 )B
𝑖
ji
𝑖 - in
where, ρji are the Euclid Distance. Ri, Gi and Bi are the mean values of RGB value in rock mineral
possible to implement recognition for rock mineral images based on the feature of color. The rock
images. Rji, Gji and Bji are the mean values of the RGB value of the ith color in the jth rock mineral.
mineral label can
be determined
by the
equation:
According
to the color distance
Sji,following
the rock mineral
ranked top 6 are recognized, and shown in the
result. See Equations (11–12).
2𝑚𝑖𝑛{𝑆𝑗𝑖 }
2 2
(𝑖 =G0,1, . . .+
, 𝑘), Bi − B ji ,
S ji = ρ ji 2 =𝑆cor𝑒
Ri ′𝑗−= 1R−ji∑𝑛 +𝑚𝑖𝑛{𝑆
Gi }−
ji
𝑗=1
(12)
𝑗𝑖
(11)
𝑆cor𝑒 ′𝑗
𝑆cor𝑒𝑗 = 𝑛
.
(13)
∑
𝑆cor𝑒
where, ρji are the Euclid Distance. Ri , Gi and B
are
the ′mean
values of RGB value in rock mineral
𝑗
𝑗=1
i
Here a color model for rock mineral will be established based on the colors of rock mineral. The
images. Rji , Gji and
Bji are the mean values of the RGB value of the ith color in the jth rock mineral.
color model will be used to assist inception-v3 model in identifying rock minerals. During
According to the
color distance
rockthe
mineral
ranked
6 are
and
identification
of the typeSji
of, athe
mineral,
inception-v3
model top
identifies
therecognized,
mineral first, then
a shown in the
six-size set
of identification
result. See Equations
(11)
and (12).results is passed to the color model. The final result is given by the color
model after matching the color of the mineral with the results from inception-v3 model.
n o
min S ji
n o (i = 0, 1, . . . , k),
Score0j = 1 − P
n
4.1. Image Processing
j=1 min S ji
4. Experiment and Results
The dataset [42] consists of 4178 images of 12 kinds of elaborate rock minerals. The information
about rock mineral images in each category is shown in0 Table 2. Twenty images were selected
Score j
randomly in each category as a test dataset, and the remaining
images were used to retrain the
Score j = Pn
0.
Inception-v3 model.
j=1 Score j
(12)
(13)
Table 2. The types and numbers of rock mineral images.
Here a color model for rock mineral will be established based on the colors of rock mineral.
Rock Minerals
Number
Rock Minerals
Number
The color model will be used
to assist inception-v3
model
rock minerals. During
Cinnabar
327
Hematite in identifying
324
Aquamarine
380
Calcite
451
identification of the type of Molybdenite
a mineral, the inception-v3
model
identifies
the
mineral
first, then a six-size
316
Stibnite
423
Malachite
335
Azurite
323
set of identification results is
passed to the color
model. Augite
The final result171
is given by the color model
Cassiterite
378
Magnetite
Gyp Inception-v3
415model.
after matching the color of the
mineral with 335
the results from
4. Experiment and Results
4.1. Image Processing
The dataset [42] consists of 4178 images of 12 kinds of elaborate rock minerals. The information
about rock mineral images in each category is shown in Table 2. Twenty images were selected randomly
in each category as a test dataset, and the remaining images were used to retrain the Inception-v3 model.
In establishing the model, numerous factors including the number and clarity of rock mineral
images, background noise and differences between features of minerals all influence its accuracy.
Generally, accuracy increases as the number of rock mineral images increases. However, the imbalance
among different types in quantity may lead to low accuracy. Therefore, we collected the images and
ensured that the total number of each category was at least 150. The proportion of rock minerals in an
image was then adjusted to at least 80%. Finally, the noise of the image, such as complex backgrounds
and labels, was removed to give the images the same size. In the process of training and the color
Minerals 2019, 9, 516
10 of 17
Minerals 2019,
x FOR PEER
REVIEW
10 of
16
model,
the 9,
texture
features
of all images were extracted and strengthened. All images were
also
translated into the RGB matrix. Then, the brightness values and a matrix containing the ∆C values of
In establishing the model, numerous factors including the number and clarity of rock mineral
the points of each image were calculated by Equation (3) and Equations (7)–(9), respectively. If they
images, background noise and differences between features of minerals all influence its accuracy.
were determined to be boundary points, the points were marked and outlined. Some pre-processed
Generally, accuracy increases as the number of rock mineral images increases. However, the
results are shown in Figure 5. Then, the Inception-v3 model was trained.
imbalance among different types in quantity may lead to low accuracy. Therefore, we collected the
images and ensured thatTable
the total
number
of each
category
at least
150. The proportion of rock
2. The
types and
numbers
of rockwas
mineral
images.
minerals in an image was then adjusted to at least 80%. Finally, the noise of the image, such as
Rock
complex backgrounds
and
labels, was removed
same size. In the process of
Rock
Minerals
Number to give the images theNumber
Minerals
training and the color model, the texture features of all images were extracted and strengthened. All
Cinnabarinto the RGB
327 matrix. Hematite
324 values and a matrix
images were also translated
Then, the brightness
Aquamarine
380
Calcite
451
containing the △C values of the points of each image were calculated by Equation (3) and Equations
Molybdenite
316
Stibnite
423
(7)–(9), respectively. IfMalachite
they were determined
points were marked and
335 to be boundary
Azurite points, the
323
outlined. Some pre-processed
shown in Figure
Cassiteriteresults are 378
Augite5. Then, the
171inception-v3 model was
Magnetite
335
Gyp
415
trained.
4.2. Model
4.2.
Model Establishment
Establishment
4.2.1. Model Training
The processed images
images were
were used
used as
as the
the raw
raw data.
data. The
The input
input image
image size
size was
was set
setto
to299
299×× 299.
299. All
the input images were pre-processed for training. There were three input channels for length, width,
width,
and color. The training steps were set to 20,000, and the learning rate was set to
to 0.01.
0.01. The prediction
results
true
label
in in
each
step,
so that
the training
and validation
accuracy
were
results were
werecompared
comparedwith
withthe
the
true
label
each
step,
so that
the training
and validation
accuracy
both
to update
the weights
in the model.
The true
label
the class
name
thefrom
dataset.
were calculated
both calculated
to update
the weights
in the model.
The
trueis label
is the
classfrom
name
the
The
dataset.
primary
The measures
primary measures
used to evaluate
used to training
evaluateeffectiveness
training effectiveness
are the training
are the
accuracy,
trainingvalidation
accuracy,
validationand
accuracy
and train cross-entropy,
shown
accuracy
train cross-entropy,
as shown inasFigure
6.in Figure 6.
Figure 6. Training process with Inception-v3.
The
have an
an influence
To get
The lighting
lighting conditions
conditions have
influence on
on the
the RGB
RGB value
value of
of the
the images.
images. To
get aa better
better color
color
model,
which
is optimized
by the
coefficient,
was used
model, the
theself-adaption
self-adaptionK-means
K-meansalgorithm,
algorithm,
which
is optimized
by silhouette
the silhouette
coefficient,
was
to
train
model
based on
10 image
forslices
each category
differentinlighting
condition
a size
used
tothe
train
the model
based
on 10slices
image
for each in
category
different
lighting(with
condition
of
about
300of× about
300). There
wereThere
aboutwere
900,000
points
in each
category.
points
were
then were
used
(with
a size
300 × 300).
about
900,000
points
in eachThese
category.
These
points
to
train
the color
model
K-means.
the model
was trained,
the silhouette
coefficient
and the
then
used
to train
the by
color
modelWhen
by K-means.
When
the model
was trained,
the silhouette
maximum
distance,
which
is
the
D
in
Equation
(10),
of
each
category
from
point
to
the
mean
i
coefficient and the maximum distance,
which is the Di in Equation (10), of each category from value
point
to the mean value were calculated. The training process ended until the silhouette coefficient was
maximum. The training results of the color model is shown in Table 3. Additionally, Figure 7 shows
samples of rock mineral images which were trained in color model.
Minerals 2019, 9, 516
11 of 17
were calculated. The training process ended until the silhouette coefficient was maximum. The training
results of the color model is shown in Table 3. Additionally, Figure 7 shows samples of rock mineral
images which were trained in color model.
Table 3. Training results of color model.
Colors (R, G, B)
Rock Minerals
Cinnabar
Hematite
Molybdenite
Calcite
Cassiterite
Magnetite
Malachite
Azurite
Aquamarine
Augite
Stibnite
Gyp
Minerals 2019, 9, x FOR PEER REVIEW
(a)
Color1
Color2
Color3
(142,79,69)
(168,96,80)
(121,125,126)
(193,199,193)
(44,47,53)
(36,33,31)
(88,168,129)
(0,19,151)
(0,95,56)
(82,80,82)
(92,104,115)
(161,161,163)
(105,36,31)
(127,157,138)
(93,96,98)
(175,177,167)
(67,144,104)
(44,77,201)
(35,159,112)
(122,120,124)
(230,220,192)
(184,107,102)
(50,108,75)
-
(b)
11 of 16
(c)
Figure7.7.Training
Trainingsamples
samplesofofcolor
colormodel.
model.(a)
(a)Cinnabar;
Cinnabar;(b)
(b)calcite;
calcite;(c)(c)aquamarine.
aquamarine.
Figure
To eliminate the influence of different
lightingresults
condition,
wemodel.
trained numerous images in different
Table 3. Training
of color
lighting conditions. Due to the difference in the reflection of rock minerals, some mineral’s RGB values
Colors (R, G, B)
minerals while some do not, therefore, the number of RGB value kinds of these minerals
changedRock
significantly,
Color1
Color2
Color3
in the colorCinnabar
model is different.
(142,79,69)
(105,36,31)
(184,107,102)
(168,96,80)
(127,157,138)
WhenHematite
training the SVM model and
RF model, the features
of raw images were extracted
by the
Molybdenite
(121,125,126)
(93,96,98)
HOG method. Then all features were resized to 256 × 256. The parameters of SVM and RF were
Calcite
(193,199,193)
(175,177,167)
determined
by the grid search algorithm.
Finally, we found that the
penalty coefficient and -gamma of
Cassiterite
(44,47,53)
SVM are 0.9
and 0.09. The number of(36,33,31)
trees of RF is 180. The parameters
of RF are the default.
Magnetite
Malachite
(88,168,129)
(67,144,104)
(50,108,75)
(0,19,151)
(44,77,201)
Aquamarine
(0,95,56)
(35,159,112)
First, two
Inception-v3 models were
retrained from both raw
images and texture feature-extraction
Augite
(82,80,82)
(122,120,124)
Stibnite
(92,104,115)
- with the
images. Then,
the retrained model from
the texture feature extraction
images was combined
Gyp
(161,161,163)
(230,220,192)
color model to create a comprehensive model. The comparison was made among the three -models to
Azurite
4.2.2. Model
Test and Results
find one with the highest accuracy. To ensure the accuracy was more convincing, 440 images were
eliminate
the influence
of different
condition,
we from
trained
images
used toTotest
these models,
and the testing
images lighting
were randomly
selected
thenumerous
image dataset
of thein
different
lighting
Due are
to the
difference
in the
reflection
of rock
some mineral’s
rock
mineral.
Theseconditions.
testing images
going
to be used
as the
input data
for minerals,
a comprehensive
model
RGB
values
changed
significantly,
while
some
do
not,
therefore,
the
number
of
RGB
value
kinds
and Inception-v3 model trained with or without feature extraction images. Meanwhile, the
images
areof
these
minerals
in
the
color
model
is
different.
going to be automatically processed when testing the comprehensive model and Inception-v3 model
the SVM model
RF model,
theThe
features
of raw
were
extracted
by the
trainedWhen
with training
feature extraction
imagesand
by the
program.
test result
is images
shown in
Table
4.
HOG method. Then all features were resized to 256 × 256. The parameters of SVM and RF were
determined by the grid search algorithm. Finally, we found that the penalty coefficient and gamma
of SVM are 0.9 and 0.09. The number of trees of RF is 180. The parameters of RF are the default.
4.2.2. Model Test and Results
Minerals 2019, 9, 516
Minerals 2019, 9, x FOR PEER REVIEW
12 of 17
12 of 16
Table
five models.
models.
Table4.
4. Results
Results of
of five
Validation
TestTest
Accuracy
Accuracy
Validation Accuracy
Accuracy
Top-1 Top-1 Top-3
Top-3
Models
Models
Model
onimages
raw images
Model
basedbased
on raw
73.1%
Model
based
on texture
extraction
imagesimages 77.4%
Model
based
on texture
extraction
Comprehensive
modelmodel
77.4%
Comprehensive
SVM-HOG
SVM-HOG
RF-HOG
73.1% 64.1%
77.4% 67.5%
77.4% 74.2%
64.1% 96.0%
96.0%
98.3%
67.5% 98.3%
99.0%
74.2% 99.0%
33.6%
33.6%
30.4%
30.4%
RF-HOG
ToToensure
the precision
precisionrate,
rate,recall
recallrate
rateand
and
ensurethe
themodels
modelsininTable
Table 44 are
are statistically
statistically feasible,
feasible, the
F1-measure
value
used
to validate
models.
In Table
the retrained
inception-v3
model
F1-measure
value
areare
used
to validate
the the
models.
In Table
4, the4,retrained
Inception-v3
model
based
on rawwas
images
was
able
to reach aaccuracy
validation
accuracy
ofthe
73.1%,
top-1
and top-3
onbased
raw images
able to
reach
a validation
of 73.1%,
and
top-1and
andthe
top-3
accuracies
were
accuracies
were respectively.
64.1% and 96.0%,
respectively.
test accuracies
rate)
SVM-HOG
64.1%
and 96.0%,
The test
accuraciesThe
(precision
rate) of(precision
SVM-HOG
andofRF-HOG
are and
32.8%
RF-HOG
are 32.8% andThe
31.2%,
respectively.
The
accuracy
is toothat
low,
which
thatnot
theeffective.
HOG
and
31.2%, respectively.
accuracy
is too low,
which
indicates
the
HOGindicates
features are
features
are
not
effective.
On
the
other
side,
the
comprehensive
model
has
a
significant
performance,
On the other side, the comprehensive model has a significant performance, which proves it can extract
which
proves
it can The
extract
texture features.
The test accuracies
of each
category
(recall
rate) ofin
the
texture
features.
testthe
accuracies
of each category
(recall rate)
of these
models
are shown
these8,
models
areF1-measure
shown in Figure
F1-measure
value
of the5.models
shown
5. Asof
Figure
and the
value8,ofand
thethe
models
is shown
in Table
As weiscan
see, in
theTable
accuracy
we
can
see,
the
accuracy
of
each
category
of
the
comprehensive
model
is
much
higher
than
that
of
each category of the comprehensive model is much higher than that of the rest models, which indicates
the
rest
models,
which
indicates
that
the
comprehensive
model
is
suitable
for
rock
mineral
that the comprehensive model is suitable for rock mineral identification. The F1-measure value of the
identification. The F1-measure value of the comprehensive model indicates that the model is
comprehensive model indicates that the model is statistically feasible.
statistically feasible.
Accuracy of each category of test
Gyp
Calcite
70.8%
31.2%
23.0%
Comprehensive model
HOG-SVM
HOG-RF
66.7%
5.0%
7.2%
Cassiterite
87.5%
26.6%
23.8%
Stibnite
55.8%
9.1%
10.7%
Molybdenite
64.2%
46.6%
43.4%
Magnetite
21.7%
Augite
18.6%
21.7%
Aquamarine
59.7%
38.9%
66.7%
23.9%
Azurite
75.0%
32.8%
32.3%
Malachite
91.7%
38.4%
91.7%
39.1%
Hematite
45.8%
43.4%
Cinnabar
76.7%
57.3%
87.5%
43.9%
44.1%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
Figure 8.
8. Test
Test accuracies
accuracies of
Figure
of each
eachcategory.
category.
Table 5. Validation parameters of models.
Models
Precision Rate
Recall Rate
F1-measure Value
Comprehensive model
SVM-HOG
RF-HOG
74.2%
33.6%
30.4%
77.5%
28.5%
31.2%
0.758
0.308
0.308
100%
Table 5. Validation parameters of models.
Models
Comprehensive model
Minerals 2019,SVM-HOG
9, 516
RF-HOG
Precision rate
74.2%
33.6%
30.4%
Recall rate
77.5%
28.5%
31.2%
F1-measure value
0.758
0.308 13 of 17
0.308
ItIt revealed
accuracy
of the
model,
with the
texture
feature feature
extraction
images,
revealedthat
thatthethe
accuracy
of Inception-v3
the inception-v3
model,
with
the texture
extraction
is
higher
than
that
with
the
raw
images
in
Table
4.
The
Inception-v3
model
with
the
texture
images, is higher than that with the raw images in Table 4. The inception-v3 model with the feature
texture
extraction
images
reached
a
higher
accuracy,
which
means
the
texture
is
a
significant
feature
for
rock
feature extraction images reached a higher accuracy, which means the texture is a significant feature
mineral
identification.
The
texture
extraction
makes
it
easier
to
distinguish
different
rock
minerals,
for rock mineral identification. The texture extraction makes it easier to distinguish different rock
which
improves
accuracy
the retrained
minerals,
which the
improves
theofaccuracy
of themodel.
retrained model.
The
The comprehensive
comprehensive model
model achieved
achieved the
the highest
highest accuracy
accuracy in
in the
the three
threemodels.
models. Color
Color properties
properties
could
increase
identification
accuracy
greatly.
From
the
test
accuracy
of
the
comprehensive
could increase identification accuracy greatly. From the test accuracy of the comprehensive model,
model,
we
can
see
that
the
top-1
accuracy
is
74.2%,
and
the
top-3
accuracy
is
99.0%.
By
combining
the
retrained
we can see that the top-1 accuracy is 74.2%, and the top-3 accuracy is 99.0%. By combining the
Inception-v3
model andmodel
the color
a comprehensive
identification
could be made
different
retrained Inception-v3
andmodel,
the color
model, a comprehensive
identification
couldfor
be made
for
rock
minerals.
The
texture
and
color
features
were
used
the
most
during
identification.
different rock minerals. The texture and color features were used the most during identification.
Following
Following are
are some
some recognition
recognition samples
samples in
in Figure
Figure 9.
9. As
As you
you can
can see,
see, these
these models
models sometimes
sometimes
achieved
different
results.
When
the
texture
features
are
similar,
and
the
colors
are
different.
For example,
achieved different results. When the texture features are similar, and the colors are different.
For
magnetite’s
cleavage
is
similar
to
the
azurite’s,
but
their
colors
are
not
similar,
the
color
model
could
example, magnetite’s cleavage is similar to the azurite’s, but their colors are not similar, the color
identify
them successfully.
the minerals’
is similar, color
but has
cleavage
such as
model could
identify themWhen
successfully.
Whencolor
the minerals’
is different
similar, but
has different
calcite
and
gym,
the
Inception-v3
model
trained
with
feature
extraction
images
could
identify
them
cleavage such as calcite and gym, the Inception-v3 model trained with feature extraction images
successfully.
Therefore,
the comprehensive
model
the advantages
of both
model and
the
could identify
them successfully.
Therefore,
the has
comprehensive
model
has the
the color
advantages
of both
Inception-v3
model
trained
with
feature
extraction
images.
Occasionally,
however,
the
comprehensive
the color model and the Inception-v3 model trained with feature extraction images. Occasionally,
model
is going
to be wrong when
theisminerals’
feature
is vague,
or the color
is abnormal.
however,
the comprehensive
model
going totexture
be wrong
when
the minerals’
texture
feature is
This
condition
will
confuse
the
comprehensive
model,
as
shown
in
Figure
10.
It
is
magnetite
vague, or the color is abnormal. This condition will confuse the comprehensive model, as shown in
in
Figure
of the
blue lighting
vague
feature,
these
all received
an
Figure 10,
10. but
It isbecause
magnetite
in Figure
10, butand
because
oftexture
the blue
lighting
andmodels
vague texture
feature,
incorrect
result.
these models
all received an incorrect result.
(a) True
(b) True
(c) True
(d) True
(e) False
(f) True
Figure 9. Cont.
Minerals 2019, 9, 516
14 of 17
Minerals
Minerals2019,
2019,9,9,x xFOR
FORPEER
PEERREVIEW
REVIEW
1414ofof1616
(g)
(g)True
True
(h)
(h)False
False
(j)(j)False
False
Figure
are
from
the
comprehensive
model,
(d–f)
are
from
the
Figure9.9.Samples
Samplesof
ofrecognition
recognitionresults.
(a–c)
are
from
the
comprehensive
model,
(d–f)
are
from
the
Figure
Samples
of
recognition
results.(a–c)
(a–c)
are
from
the
comprehensive
model,
(d–f)
are
from
Inception-v3
model
trained
with
feature
extraction
images,
Inception-v3
model
trained
with
feature
extraction
images,
arefrom
fromthe
theInception-v3
Inception-v3model
model
the
Inception-v3
model
trained
with
feature
extraction
images,(g–j)
(g–j)are
are
from
the
without
feature
extraction.
without
feature
extraction.
without
feature
extraction.
(a)
(a)
(b)
(b)
(c)(c)
Figure
True;
(b)
true;
(c)(c)
true.
Figure10.
10.Identification
Identificationresults
resultsfrom
fromthese
thesethree
threemodels.
models.(a)
(a)True;
True;
(b)
true;
true.
(b)
true;
(c)
true.
5.
Conclusions
5.5.Conclusions
Conclusions
In
established.
InInthis
this
research,
three
models
based
on
Inception-v3
model
were
established.
The
deep
learning
thisresearch,
research,three
threemodels
modelsbased
basedon
onInception-v3
Inception-v3model
modelwere
were
established.The
Thedeep
deeplearning
learning
model,
coupled
with
color
model
reached
top-1
and
top-3
accuracies
of
74.2%
and
99.0%,
respectively.
model,
model, coupled
coupled with
with color
color model
model reached
reached top-1
top-1 and
and top-3
top-3 accuracies
accuracies ofof 74.2%
74.2% and
and 99.0%,
99.0%,
The
retrainedThe
model
usingmodel
raw images
reached
top-1
and top-1
top-3
accuracies
of 64.1%ofof
and
96.0%,
respectively.
retrained
using
images
reached
64.1%
and
respectively.
The
retrained
model
usingraw
raw
images
reached
top-1and
andtop-3
top-3accuracies
accuracies
64.1%
and
respectively.
The
retrained
model
using
texture
extraction
images
reached
top-1
and
top-3
accuracies
96.0%,
96.0%,respectively.
respectively.The
Theretrained
retrainedmodel
modelusing
usingtexture
textureextraction
extractionimages
imagesreached
reachedtop-1
top-1and
andtop-3
top-3
of
67.5% and
98.3%,
respectively.
The
comparison
ofcomparison
the three models
indicates
that the
comprehensive
accuracies
ofof
67.5%
and
The
ofofthe
three
indicates
that
accuracies
67.5%
and98.3%,
98.3%,respectively.
respectively.
The
comparison
the
threemodels
models
indicates
thatthe
the
model
is
the
best
of
all.
The
SVM-HOG
model
achieved
a
validation
accuracy
of
32.8%.
The
RF-HOG
comprehensive
model
is
the
best
of
all.
The
SVM-HOG
model
achieved
a
validation
accuracy
of
comprehensive model is the best of all. The SVM-HOG model achieved a validation accuracy of
model
a validation
of
The accuracy
results
indicate
that The
the
learning
model
32.8%.
The
model
achieved
a a31.2%.
validation
ofof31.2%.
results
indicate
that
the
32.8%.achieved
TheRF-HOG
RF-HOG
modelaccuracy
achieved
validation
accuracy
31.2%.
Thedeep
results
indicate
thatcan
the
extract
effective
image
features.
The
comparison
between
the
traditional
models
and
the
deep
learning
deep
deeplearning
learningmodel
modelcan
canextract
extracteffective
effectiveimage
imagefeatures.
features.The
Thecomparison
comparisonbetween
betweenthe
thetraditional
traditional
model
shows
that
the deep
learning
models
are
much
more
effective
than
theare
traditional
ones.
models
and
learning
model
shows
that
the
much
effective
models
andthe
thedeep
deep
learning
model
shows
that
thedeep
deeplearning
learningmodels
models
are
muchmore
more
effective
The
deep
learning
algorithm
provides
a
new
idea
for
rock
mineral
identification.
The
clustering
than
thanthe
thetraditional
traditionalones.
ones.
algorithm
could
also improve
the performance
of
the
deep
learning
model.
Therefore, the
combination
The
algorithm
provides
idea
for
rock
identification.
The
clustering
Thedeep
deeplearning
learning
algorithm
providesa anew
new
idea
for
rockmineral
mineral
identification.
The
clustering
of
a
deep
learning
algorithm
and
clustering
algorithm
is
an
effective
way
for
rock
mineral
identification.
algorithm
algorithm could
could also
also improve
improve the
the performance
performance ofof the
the deep
deep learning
learning model.
model. Therefore,
Therefore, the
the
Meanwhile,
the
features
and
texture features
of rock algorithm
minerals
significant
for
combination
ofofacolor
learning
algorithm
and
isisan
way
combination
adeep
deep
learning
algorithm
andclustering
clustering
algorithmare
aneffective
effectiveproperties
wayfor
forrock
rock
identification.
The identification
accuracy
ofcolor
Inception-v3
model
could features
befeatures
improved
greatly
by texture
mineral
Meanwhile,
the
features
ofofrock
are
mineralidentification.
identification.
Meanwhile,
thecolor
featuresand
andtexture
texture
rockminerals
minerals
are
and
color
extraction.
significant
significantproperties
propertiesfor
foridentification.
identification.The
Theidentification
identificationaccuracy
accuracyofofInception-v3
Inception-v3model
modelcould
couldbebe
Furthermore,
the
elaborate
mineral
images, with clear mineral features, are adopted. If the
improved
greatly
and
improved
greatlyby
bytexture
texture
andcolor
colorextraction.
extraction.
features,
such as cleavage
and luster,
can beimages,
extracted,
theclear
comprehensive
model can
be
further tested
Furthermore,
the
mineral
are
adopted.
IfIfthe
Furthermore,
theelaborate
elaborate
mineral
images,with
with
clearmineral
mineralfeatures,
features,
are
adopted.
the
for
mineral
specimen
identification,
even
for
field
survey.
features,
features,such
suchasascleavage
cleavageand
andluster,
luster,can
canbebeextracted,
extracted,the
thecomprehensive
comprehensivemodel
modelcan
canbebefurther
further
tested
testedfor
formineral
mineralspecimen
specimenidentification,
identification,even
evenfor
forfield
fieldsurvey.
survey.
Author
AuthorContributions:
Contributions:Modules
Modulesdesign
designand
anddevelopment,
development,C.L.;
C.L.;Overall
Overallframework
frameworkdesign,
design,M.L.;
M.L.;Data
Data
collection
collectionand
andapplication
applicationtest,
test,Y.Z.
Y.Z.and
andY.Z.;
Y.Z.;Algorithm
Algorithmand
andmodel
modelimprovement,
improvement,S.H.
S.H.
Minerals 2019, 9, 516
15 of 17
Author Contributions: Modules design and development, C.L.; Overall framework design, M.L.; Data collection
and application test, Y.Z. and Y.Z.; Algorithm and model improvement, S.H.
Funding: This research was funded by the National Natural Science Foundation for Excellent Young Scientists of
China (Grant no. 51622904), the Tianjin Science Foundation for Distinguished Young Scientists of China (Grant no.
17JCJQJC44000) and the National Natural Science Foundation for Innovative Research Groups of China (Grant
no. 51621092).
Conflicts of Interest: We declare that we have no financial and personal relationships with other people or
organizations that can inappropriately influence our work, there is no professional or other personal interest
of any nature or kind in any product, service or company that could be construed as influencing the position
presented in, or the review of, the manuscript entitled.
References
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
Yeshi, K.; Wangdi, T.; Qusar, N.; Nettles, J.; Craig, S.R.; Schrempf, M.; Wangchuk, P. Geopharmaceuticals of
Himalayan Sowa Rigpa medicine: Ethnopharmacological uses, mineral diversity, chemical identification
and current utilization in Bhutan. J. Ethnopharmacol. 2018, 223, 99–112. [CrossRef] [PubMed]
Rustom, L.E.; Poellmann, M.J.; Johnson, A.J.W. Mineralization in micropores of calcium phosphate scaffolds.
Acta Biomater. 2019, 83, 435–455. [CrossRef] [PubMed]
Laura, B.; Nina, E.; Mona, W.M.; Udo, Z.; Andò Sergio Merete, M.; Reidar, I.K. Quick, easy, and economic
mineralogical studies of flooded chalk for eor experiments using raman spectroscopy. Minerals 2018, 8, 221.
Ramil, A.; López, A.J.; Pozo-Antonio, J.S.; Rivas, T. A computer vision system for identification of
granite-forming minerals based on RGB data and artificial neural networks. Measurement 2017, 117, 90–95.
[CrossRef]
Sadeghi, B.; Madani, N.; Carranza, E.J.M. Combination of geostatistical simulation and fractal modeling for
mineral resource classification. J. Geochem. Explor. 2015, 149, 59–73. [CrossRef]
Li, R.; Albert, N.N.; Yun, M.; Meng, Y.S.; Du, H. Geological and Geochemical Characteristics of the Archean
Basement-Hosted Gold Deposit in Pinglidian, Jiaodong Peninsula, Eastern China: Constraints on Auriferous
Quartz-Vein Exploration. Minerals 2019, 9, 62. [CrossRef]
Rajendran, S.; Nasir, S. ASTER capability in mapping of mineral resources of arid region: A review on
mapping of mineral resources of the Sultanate of Oman. Ore Geol. Rev. 2019, 108, 33–53. [CrossRef]
Shi, B.; Liu, J. Nonlinear metric learning for kNN and SVMs through geometric transformations.
Neurocomputing 2018, 318, 18–29. [CrossRef]
Sachindra, D.; Ahmed, K.; Rashid, M.M.; Shahid, S.; Perera, B. Statistical downscaling of precipitation using
machine learning techniques. Atmos. Res. 2018, 212, 240–258. [CrossRef]
Vassilev, S.V.; Vassileva, C.G. A new approach for the combined chemical and mineral classification of the
inorganic matter in coal. 1. Chemical and mineral classification systems. Fuel 2009, 88, 235–245. [CrossRef]
Vassilev, S.V.; Vassileva, C.G. A new approach for the classification of coal fly ashes based on their origin,
composition, properties, and behaviour. Fuel 2007, 86, 1490–1512. [CrossRef]
Zaini, N.; Van Der Meer, F.; Van Der Werff, H. Determination of Carbonate Rock Chemistry Using
Laboratory-Based Hyperspectral Imagery. Remote. Sens. 2014, 6, 4149–4172. [CrossRef]
Adep, R.N.; Shetty, A.; Ramesh, H. EXhype: A tool for mineral classification using hyperspectral data.
ISPRS J. Photogramm. Remote. Sens. 2017, 124, 106–118. [CrossRef]
Othman, A.A.; Gloaguen, R. Integration of spectral spatial and morphometric data into lithological. J. Asian
Earth Sci. 2017, 146, 90–102. [CrossRef]
Li, Y.; Chen, C.; Fang, R.; Yi, L. Accuracy enhancement of high-rate GNSS positions using a complete
ensemble empirical mode decomposition-based multiscale multiway PCA. J. Asian Earth Sci. 2018, 169, 67–78.
[CrossRef]
Wang, W.; Chen, L. Flotation Bubble Delineation Based on Harris Corner Detection and Local Gray Value
Minima. Minerals 2015, 5, 142–163. [CrossRef]
Chen, J.; Chen, Y.; Wang, Q. Synthetic Informational Mineral Resource Prediction: Case Study in Chifeng
Region, Inner Mongolia, China. Earth Sci. Front. 2008, 15, 18–26. [CrossRef]
Shardt, Y.A.; Brooks, K. Automated System Identification in Mineral Processing Industries: A Case Study
using the Zinc Flotation Cell. IFAC-Papers OnLine 2018, 51, 132–137. [CrossRef]
Minerals 2019, 9, 516
19.
20.
21.
22.
23.
24.
25.
26.
27.
28.
29.
30.
31.
32.
33.
34.
35.
36.
37.
38.
39.
40.
41.
16 of 17
Ślipek, B.; Młynarczuk, M. Application of pattern recognition methods to automatic identification
of microscopic images of rocks registered under different polarization and lighting conditions.
Geol. Geophys. Environ. 2013, 39, 373. [CrossRef]
Młynarczuk, M.; Górszczyk, A.; Ślipek, B. The application of pattern recognition in the automatic classification
of microscopic rock images. Comput. Geosci. 2013, 60, 126–133. [CrossRef]
Shu, L.; McIsaac, K.; Osinski, G.R.; Francis, R. Unsupervised feature learning for autonomous rock image
classification. Comput. Geosci. 2017, 106, 10–17. [CrossRef]
Coates, A.; Ng, A.Y.; Lee, H. An analysis of single-layer networks in unsupervised feature learning.
In Proceedings of the 14th International Conference on Artificial Intelligence and Statistics, Fort Lauderdale,
FL, USA, 11–13 April 2011; pp. 215–223.
Raina, R.; Battle, A.; Lee, H.; Packer, B.; Ng, A.Y. Self-taught learning: Transfer learning from unlabeled
data. In Proceedings of the 24th International Conference on Machine Learning, Corvallis, OR, USA,
20–24 June 2007; pp. 759–766.
Aligholi, S.; Lashkaripour, G.R.; Khajavi, R.; Razmara, M. Automatic mineral identification using color
tracking. Pattern Recognit. 2016, 65, 164–174. [CrossRef]
Li, N.; Hao, H.; Gu, Q.; Wang, D.; Hu, X. A transfer learning method for automatic identification of sandstone
microscopic images. Comput. Geosci. 2017, 103, 111–121. [CrossRef]
Google Deep Learning. Available online: http://deeplearning.net/tag/google/ (accessed on 21 March 2019).
Zhang, Y.; Li, M.C.; Han, S. Automatic identification and classification in lithology based on deep learning in
rock images. Acta Petrol. Sin. 2018, 34, 333–342.
Kitzig, M.C.; Kepic, A.; Grant, A. Near Real-Time Classification of Iron Ore Lithology by Applying Fuzzy
Inference Systems to Petrophysical Downhole Data. Minerals 2018, 8, 276. [CrossRef]
Xiong, Y.; Zuo, R.; Carranza, E.J.M. Mapping mineral prospectivity through big data analytics and a deep
learning algorithm. Ore Geol. Rev. 2018, 102, 811–817. [CrossRef]
Zuo, R.; Xiong, Y.; Wang, J.; Carranza, E.J.M. Deep learning and its application in geochemical mapping.
Earth-Sci. Rev. 2019, 192, 1–14. [CrossRef]
Iglesias, J.C.; Álvarez; Santos, R.B.M.; Paciornik, S. Deep learning discrimination of quartz and resin in
optical microscopy images of minerals. Miner. Eng. 2019, 138, 79–85. [CrossRef]
Hinton, G.E.; Salakhutdinov, R.R. Reducing the Dimensionality of Data with Neural Networks. Science 2006,
313, 504–507. [CrossRef]
Hinton, G.; Mohamed, A.-R.; Jaitly, N.; Vanhoucke, V.; Kingsbury, B.; Deng, L.; Yu, D.; Dahl, G.; Senior, A.;
Nguyen, P.; et al. Deep Neural Networks for Acoustic Modeling in Speech Recognition: The Shared Views of
Four Research Groups. IEEE Signal Process. Mag. 2012, 29, 82–97. [CrossRef]
Gong, M.; Yang, H.; Zhang, P. Feature learning and change feature classification based on deep learning for
ternary change detection in SAR images. ISPRS J. Photogramm. Remote Sens. 2017, 129, 212–225. [CrossRef]
Pham, C.C.; Jeon, J.W. Robust object proposals re-ranking for object detection in autonomous driving using
convolutional neural networks. Signal Process. Image Commun. 2017, 53, 110–122. [CrossRef]
Esteva, A.; Kuprel, B.; Novoa, R.A.; Ko, J.; Swetter, S.M.; Blau, H.M.; Thrun, S. Dermatologist-level
classification of skin cancer with deep neural networks. Nature 2017, 542, 115–118. [CrossRef]
Szegedy, C.; Vanhoucke, V.; Ioffe, S.; Shlens, J.; Wojna, Z. Rethinking the Inception Architecture for Computer
Vision. In Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR),
Las Vegas, NV, USA, 26 June–1 July 2016; pp. 2818–2826.
Qureshi, A.S.; Khan, A.; Zameer, A.; Usman, A. Wind power prediction using deep neural network based
meta regression and transfer learning. Appl. Soft Comput. 2017, 58, 742–755. [CrossRef]
Xu, G.; Zhu, X.; Fu, D.; Dong, J.; Xiao, X. Automatic land cover classification of geo-tagged field photos by
deep learning. Environ. Model. Softw. 2017, 91, 127–134. [CrossRef]
Xu, B.; Shen, S.; Shen, F.; Zhao, J. Locally linear SVMs based on boundary anchor points encoding. Neural Netw.
2019, 117, 274–284. [CrossRef]
Pellant, C. Rocks and Minerals—Smithsonian Handbook; DK Publisher: New York, NY, USA, 2002.
Minerals 2019, 9, 516
42.
43.
17 of 17
The Geological Museum of China. Available online: http://www.gmc.org.cn (accessed on 28 March 2019).
International Telecommunication Union. Available online: https://www.itu.int/rec/R-REC-BT.601-7-201103I/en (accessed on 12 March 2019).
© 2019 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access
article distributed under the terms and conditions of the Creative Commons Attribution
(CC BY) license (http://creativecommons.org/licenses/by/4.0/).