Applying Improved Optical Recognition with Machine Learning on Sorting Cu Impurities in Steel Scrap
Zhiijang Gao1, *, S. Sridhar1, D. Erik Spiller1, Patrick R. Taylor1
1
Affiliation: Kroll Institute for Extractive Metallurgy, Mining Engineering Department, Colorado School of Mines
Address: 1500 Illinois St., Golden, Colorado
*Corresponding Author: Zhijiang Gao
E-mail: zhijianggao@mymail.mines.edu
Telephone: 720-369-4587
Graphical abstract
1. Abstract
Cu impurities in scrap, originating from motors and wires, limit the efficiency of recycling steel scrap, due to the
occurrence of surface hot shortness during hot working resulting from high Cu content. Considering the distinct
difference of color between metal Cu and Fe and the potential differences between shapes of shreds depending on Cu
content, optical recognition, was explored as a method for detecting and separating Cu rich shreds. In order to optimize
detection and minimize effects of surface inhomogeneity etc. Convolutional Neural Network (CNNs), was adopted to
improve the optical recognition of shredded scrap obtained from industrial sources. The results show that the proposed
neural network achieves significantly better recognition on Cu impurities and results in a reduction of Cu content. An
optimized accuracy of 90.6% could be obtained for recognizing Cu impurities through applied CNN architecture with
dataset of cropped photographs. This results in an overall 0.185wt% reduction of Cu impurities in steel scrap, if the
identified Cu rich parts were removed.
Key words: Steel scrap, Cu impurities, Optical recognition, Convolutional Neural Network
2. Introduction
2.1 Steel Scrap Recycling
Over 70% of steel is recycled today by re-melting scrap in electric-arc furnaces (EAF) [1] and 100% steel scrap with
high quality could be preferred for the furnace charging. However, many EAF operating plants adopt pig iron or other
iron-containing materials to blend with scrap [2]. This is because impurities such as Cu and Sn, accumulated in the
scrap through wires and motors in cars and tin-plate, cause surface cracking when the produced steel is processed in
1
hot and oxidizing conditions, during secondary cooling from the continuous caster or reheating for thermo-mechanical
processing. The cracking phenomenon is referred to as surface hot shortness and occurs at 0.1wt% Cu content or
greater [3]. The mechanism for this detrimental effect is the enrichment of Cu leading to a separate liquid Cu-enriched
phase [4] which penetrates the austenite grain boundaries. Worldwide speaking, about 75% of metal part in the charge
of EAF is covered by steel scrap, while the remaining is covered by pig iron, DRI/HBI and other hot metal [5].
The presenting of Cu impurities as motors and wires in steel scrap charged to EAF could be attributed to the
upstreaming liberation. Regular hammer shredder followed by magnetic separator has been commonly implemented
in most of automobile recycling plants. Dismantling of Cu motors and wires would be just confined to units with
larger sizes considering the costs and benefits. During hammer shredding, smaller copper motors and wires become
enmeshed and entangled within Fe shreds, resulting in a decreased separation efficiency of the following magnetic
separation, approximate 80% or less [6].
2.2 Optical Recognition
The advent of sensor-based sorting technology exhibits an excellent potential to remove Cu impurities during physical
separation, before being processed in the EAF plant. Color sensors [7], as optical recognition to distinguish different
color values, has been widely applied to many fields, including the mining industry for sorting impurities or certain
minerals [8], agricultural industry for sorting grains [9], and recycling for sorting plastic bottles or flakes [10].
Considering the distinct color difference between metal Cu and Fe, it is particularly desirable to achieve sortation of
Cu impurities with optical recognition in combination with shape detection, referring to the regular shape of Cu motors
and linear shape of Cu wires. Other types of sensor, such as LIBS (Laser Induced Breakdown Spectroscopy) [11],
XRT&XRF [12, 13], PGNAA (Prompt Gamma Neutron Activation Analysis) [14], have also been discussed for their
feasibility to sort Cu impurities from steel scrap. As a matter of fact, no additional automatic sorting methods have
been adopted successfully with industrial scale, considering the costs and sorting efficiency. Only manual sorting has
been partly selected to pick up evident pieces of Cu motor after magnetic separation.
Compared to other types of sensor, low cost requirement could be satisfied by optical recognition. This approach could
however be subject to the inappropriate liberation and surface cleanliness of steel scrap.
2.3 Image Classification with Machine Learning
With further understanding, the essence of optical recognition is the designed program for analyzing images and
distinguishing different color values. It shares the same mechanism of image classification [15], which has become a
common application for machine learning. The purpose of this study is to investigate and determine the feasibility of
eliminating errors related to surface heterogeneities through applying machine learning to improve optical recognition
of steel scrap for a better sorting efficiency of Cu impurities.
Convolutional Neural Network (CNNs) has been commonly used as a deep learning algorithm for image
differentiation and classification [16]. Like the regular neural network, they both have multiple hidden layers besides
of the input and output layer. However, for the purpose of simplification and clarity, the neurons in one layer of CNNs
will only be connected to a particular region of previous layer, involving 3 dimensions: width, height and depth,
comparing to the regular neuron network, which could induce wasteful overfitting because of the full connectivity of
each hidden layers and neurons [17]. Referring to the input images, the width and height would be the pixel dimensions
of the image, and the depth would be 3 matching with the RGB channels. The simplest architecture of CNNs consists
of input layer, convolutional layer for abstracting the input image into a feature map, pooling layer for streamlining
the underlying computation and full-connected layer for computing the class score of further classification [18].
Therefore, image classification can be treated as the process of inputting image data and outputting the probability of
a specific class. To solve the problem of image classification in different fields, various complicated architectures of
CNNs, including AlexNet, VGGNet, GoogleNet, ResNet, etc., have been developed [19]. These architectures play the
key role of understanding and memorizing features of input dataset during the training stage and recognizing and
classifying new images during the test stage. In our initial experiments, VGGNet architecture is applied for training
with the image dataset from shredded pieces of steel scrap before carrying out the testing. More details will be given
in the following part.
2
3. Data set collection
3.1 Preparation of Photographs
Shredded automobile scrap samples were collected directly from a local metal recycling company, which processed
the obsolete automobiles through shredding and magnetic separating without additional handpicking and sorting. By
visual inspection in the laboratory, Cu impurities were identified and picked out, including Cu motors and wires. With
sufficient illumination, a smartphone was used to acquire the photographs of identified Cu sources, as well as Fe
shreds, placed on a black conveyor belt, which was adopted to simulate the real working condition for sorting system,
as shown in Figure 1. No particular lighting was applied for acquiring the photographs. Considering the small amount
of identified Cu sources, random rotations were manually operated to each piece of identified Cu sources during
acquiring the photographs as a way of data augmentation, in order to maintain a balance with the number of acquired
photographs for Fe shreds. An average of 2000 photographs for identified Cu sources and Fe shreds were collected in
our work.
Fig. 1 Acquired Photographs for shredded steel scrap: (a) Shredded steel scrap in container; (b) Cu wire; (c) Cu
motor rotor; (d) Cu motor with wire; (e) and (f) Fe shreds
3.2 Methodology
As shown in Figure 2, the whole process of experiments was spilt into 3 major parts: inputting dataset, training and
testing. The first part was to input the prepared photographs of Fe shreds and identified Cu sources as two datasets,
which were further labeled with Fe and Cu respectively from the prospect of sorting Cu impurities, for the following
training with CNNs architecture. The next part was the training process using a CNNs architecture. In the initial trial,
architecture VGGNet was used to enable deep learning to be achieved with a relatively simple and linear structure
[20]. When all the labeled photographs had been analyzed through the VGGNet architecture, a model file was
generated, based on learned useful feature information related to two known labels, to classify new photographs. Then
in the third part, the testing dataset, which consisted of new photographs for Fe shreds and identified Cu sources that
had not been included in the training dataset, was tested and classified by the generated model. The detailed
distribution of photographs to the training and testing parts as the dataset is shown in Figure 3.
3
①
Photographs
of Identified
Cu Sources
Datasets
Labeled as Cu
Labeled as Fe
②
③
Testing New
Photographs
Model File
Generated for
Testing
Training with
CNNs
Architecture
Result: Percentage
of Being Label Fe or
Label Cu
Photographs
of Fe Shreds
Fig. 2 Implementation of experiment
Fig. 3 Dataset distribution for training and testing process
3.3 VGGNet Architecture
Normal VGGNet architecture contains convolutional layers, max pooling layers, activation layers and full-connected
layers. Inspired by Adrian Rosebrock [21], who successfully applied a smaller VGGNet architecture for image
classification, modification based on his code was carried out to build a full VGGNet architecture for detailed
configuration training, as shown in Figure 4. Considering the training time and computing performance of a personal
computer, only one full-connected (FC) layer was built in this configuration.
Fig. 4 Configuration of VGGNet architecture
3.3 Training Accuracy and Loss
4
For the training process, 80% of the dataset was chosen for training, while the rest 20% was applied for validation,
which could provide an unbiased evaluation of the model fit on the training dataset for each training epoch. As shown
in Figure 5, the accuracy of training and validating could be approximate 99% and the corresponding loss could be as
low as 0.01 for the final epoch.
Fig. 5 Training with VGGNet and original dataset: (a) Accuracy of training and validating; (b) Loss of training and
validating
3.4 Testing Results
During the testing stage, the generated model tested 36 new photographs of Fe shreds and 32 new photographs of
identified Cu sources, as shown in Figure 3. In our work, the testing results were demonstrated with the possibility of
being classified as label Fe and label Cu. For example, as shown in Figure 6, the obtained testing result for the upper
right photograph from an identified Cu source was about 99.97% of being Label Cu and 0.03% of being Label Fe,
indicating that this piece was recognized correctly. However, for the lower right photograph from the Fe shreds group,
the obtained result was 99.81% of being label Cu, indicating that this piece was recognized incorrectly.
Fig. 6 Demonstration of testing results
For the whole testing dataset, shredded Fe parts in 27 of 36 photographs were recognized correctly, while 17 pieces
among 32 photographs of identified Cu sources were recognized correctly. Thus the accuracy for recognizing Cu
impurities could be estimated as 53.1%. Compared with the high training and validating accuracy, overfitting could
be the major concern for further optimization.
4. Optimization
5
4.1 Dataset: Shape Feature
One possible reason for the low Cu recognizing accuracy and overfitting could be the interference from black
background, which was dominant in most of the photographs compared to the shredded objects. During the training,
this dominated area might be analyzed as the feature information, weakening the weight of other distinct colors, such
as the reddish-brown of Cu metal in the motor rotor, the bright white of the motor shell, and the red or blue insulation
of Cu wire. As mentioned before, the regular shape geometry could be also treated as the feature information for
recognizing identified Cu sources. In order to optimize the Cu recognizing accuracy, shape feature, as shown in Figure
7, was extracted individually from the prepared photographs as a new training dataset.
Fig. 7 Extracted shape feature: (a) Cu wire; (b) Cu motor rotor; (c) Cu motor with wire; (d) and (e) Fe shreds
The same process utilizing VGGNet architecture was applied to the new dataset of shape features. After training, a
new model file was generated for testing. As a result, 25 pieces among 32 photographs of identified Cu sources were
recognized correctly, while 20 pieces among 36 photographs of Fe shreds were recognized correctly. So the accuracy
for recognizing Cu impurities was improved to 78.1%, though the Fe recognizing accuracy decreased, which may be
attributed to the light reflection. During the preparation of photographs, obvious light reflection could be observed for
Fe shreds due to the irregular presentation and normal illumination, leading to the existence of redundant information
that could disturb the recognition, as demonstrated by (d) and (e) in Figure 7.
Fig. 8 Training with VGGNet and shape feature: (a) Accuracy of training and validating; (b) Loss of training and
validating
However, as shown in Figure 8, overfitting problem could be still observed for training with this new dataset,
signifying that it is highly possible that, for the prepared dataset, the background could be learned and detected as a
feature.
4.2 Dataset: Cropped Photographs
6
With this concern, photographs had been cropped by programming to remove the background as much as possible and
applied as new dataset, as shown in Figure 9.
Fig. 9 Cropped dataset: (a) Cu wire; (b) Cu motor rotor; (c) Cu motor with wire; (d) and (e) Fe shreds
Deformation could be observed and acceptable to maintain the inputting size of photographs as 224×224. The same
process utilizing VGGNet architecture was applied to the new cropped dataset. As a result, 20 pieces among 32
photographs of identified Cu sources were recognized correctly, while 22 pieces among 36 photographs of Fe shreds
were recognized correctly. Thus the accuracy for recognizing Cu impurities could be estimated as 62.5%, while the
accuracy for Fe shreds could be 61.1%. As shown in Figure 10, overfitting could be still a problem, nonetheless,
dataset with cropped photographs is worth pursuing.
Fig. 10 Training with VGGNet and cropped photographs: (a) Accuracy of training and validating; (b) Loss of
training and validating
4.3 Architecture: Xception
Based on above investigation and further analysis, it seems that initial VGGNet architecture might not be the suitable
architecture, referring to the specific dataset. Another architecture, Xception, has been tried with cropped photographs
as dataset. Xception [22] was first proposed by Google to improve the performance of deep learning with a modified
depthwise separable convolution. The testing result of this optimization is shown in Table 1.
7
Table 1 Detailed testing results for different trials
Trials
Testing Results
Fe: 27/36
Cu: 17/32
Fe: 20/36
Cu: 25/32
Fe: 22/36
Cu: 20/32
Fe: 28/36
Cu: 29/32
Initial Trial with VGGNet and Original Photographs
Optimization
Trial with VGGNet and
Shape Feature
Trial with VGGNet and
Cropped Photographs
Trial with Xception and
Cropped Photographs
Recognizing Accuracy
75.0%
53.1%
55.6%
78.1%
61.1%
62.5%
77.8%
90.6%
The testing accuracy for Cu impurities and Fe shreds both got improved compared to other trials. Meantime, to a
certain extent, overfitting problem could be regarded as mitigation, though the accuracy of training and validating
remained high, as shown in Figure 11.
Fig. 11 Training with Xception and cropped photographs: (a) Accuracy of training and validating; (b) Loss of
training and validating
5. Discussion
The purpose of our work is to improve the optical recognition through applying machine learning such that Cu
impurities could be sorted and removed during the physical separation. Fundamentally, the system for optical
recognition [23] contains an array of digital cameras positioned above the belt to acquire photographs for individual
pieces of shredded steel scrap, and a control computer for receiving and analyzing each photograph based on a
designed program to identify the Cu impurities, as shown in Figure 12.
8
Fig. 12 Schematic view for optical recognition system
By focusing on the essence of optical recognition, the testing process based on the generated model in our work could
be incorporated into the control computer as the designed program, assuming the same recognizing accuracy could be
achieved for the system. Once a photograph is tested and classified as label Cu, the corresponding ejecting decision is
transmitted to the ejecting system, which activates selected air nozzles in the array mounted at the discharge end of
conveyor to blow the target out of the natural trajectory, i.e. into the reject bin.
Referring to the visual inspection and chemical analysis in our laboratory for shredded steel scrap, the detailed
compositional information is shown in Table 2.
Table 2 Detailed compositional information for shredded steel scrap
Total Weight
Weight of Fe Shreds (WFe Shreds)
Background (Alloyed) Cu Content (wt%) (CBackground Cu)
Identified Cu Sources
Weight of Identified Cu Sources (WCu Sources)
1,337.96 kg
1,335.129 kg
0.061%
Cu Motors and Wires
2.831 kg
Weight Percent of Identified Cu Sources (wt%)
0.212%
Total Cu Content (wt%)
0.272%
Therefore the Cu content (wt%) after optical recognition could be calculated with the equation (1):
100
𝑊
𝑊
Cu Recognizing Accuracy
𝑊
100
100 Cu Recognizing Accuracy
100
Fe Recognizing Accuracy /100
𝑊
𝐶
Fe Recognizing Accuracy /100
and the results are shown in Table 3.
9
/100
100%
1
Table 3 Estimated Cu removal for improved optical recognition with machine learning
Trials
Testing
Results
Recognizing
Accuracy
Initial Trial with VGGNet and Original
Photographs
Trial with VGGNet and
Shape Feature
Trial with VGGNet and
Optimization
Cropped Photographs
Trial with Xception and
Cropped Photographs
Fe: 27/36
Cu: 17/32
Fe: 20/36
Cu: 25/32
Fe: 22/36
Cu: 20/32
Fe: 28/36
Cu: 29/32
75.0%
53.1%
55.6%
78.1%
61.1%
62.5%
77.8%
90.6%
Estimated Cu wt%
before Optical
Recognition
Estimated Cu wt%
after Optical
Recognition
0.193%
0.144%
0.272%
0.191%
0.087%
It is evident that applying this improved optical recognition could be used to reduce the Cu content of shredded steel
scrap, achieving the 0.1wt% limitation for Cu content. At this stage, our work has mainly focused on investigating the
effect of the prepared and optimized dataset on the final recognizing accuracy, in order to eliminate the impact of
overfitting. In the future, other key factors, such as training approach, batch size, and design of different architectures,
can also be researched and optimized to improve the recognizing accuracy. In this manner, the 0.1wt% limitation of
Cu content could be targeted, indicating that commercial steel and structural steel products could be produced by the
direct smelting of sorted steel scrap [24]. Considering that scrap is currently blended with virgin iron, to dilute the
impurity concentration, when manufacturing value-added steel products, such as coated strips, the optimized
improvement shown in Table 3, would reduce the needed level of virgin iron by 2.222t per tonne of sorted steel scrap,
corresponding to the energy saving, when accounting for the embedded energy in coal used to produce virgin iron, of
28.22GJ [25].
6. Conclusion
In this work, we proposed an improved optical recognition system for sorting the Cu impurities from shredded steel
scrap. This was realized by building the designed program for analysis and classification in accordance with the testing
process of machine learning with Convolutional Neuron Network (CNNs). The proposed VGGNet architecture based
CNNs was initially adopted for the training process with real photographs of shredded steel scrap, including identified
Cu sources and Fe shreds. Then the testing process was conducted based on the generated model file after training.
From the experimental results, a Cu recognizing accuracy of 53.1% was achieved for the training with normal
photographs, resulting in overall 0.079wt% reduction in Cu content. Further optimization has been attempted to delve
into the overfitting phenomena through modifying the dataset and applying new architecture. In terms of better
performance, optimized Cu recognizing accuracy of 90.6% was achieved for the training with Xception architecture
and cropped photographs, resulting in overall 0.185wt% reduction in Cu content. We believe that the proposed
technique has the potential to further reduce the Cu content through future research.. Also considering the requirement
of large dataset for training, which would be a challenge for our work, transfer learning with pre-trained CNNs
architecture should also be investigated [26].
Acknowledgment: “This material is based upon work supported by the U.S. Department of Energy’s Office of Energy
Efficiency and Renewable Energy (EERE) under the Advanced Manufacturing Office Award Number DEEE0007897.”
Disclaimer: “This report was prepared as an account of work sponsored by an agency of the United States Government.
Neither the United States Government nor any agency thereof, nor any of their employees, makes any warranty,
express or implied, or assumes any legal liability or responsibility for the accuracy, completeness, or usefulness of
any information, apparatus, product, or process disclosed, or represents that its use would not infringe privately owned
rights. Reference herein to any specific commercial product, process, or service by trade name, trademark,
manufacturer, or otherwise does not necessarily constitute or imply its endorsement, recommendation, or favoring by
10
the United States Government or any agency thereof. The views and opinions of authors expressed herein do not
necessarily state or reflect those of the United States Government or any agency thereof.”
Conflict of interest: The authors declare that there is no conflict of interest.
References
[1] Graedel, T. et al. (2011) What do we know about metal recycling rates? J. Ind. Ecol. 15, 355–366.
[2] Kerimov, R. I., and S. I. Shakhov. "Use of Metallized Raw Materials in Electric Furnace Steelmaking."
Metallurgist (2020): 1-8.
[3] American Society for Metals (1970) “Metals Handbook,” Vol. 5, Forging and Casting, ASM.
[4] Iain Le May, L. McDonald Schetky (1982) “Copper in Iron and Steel - Chapter 3,” John Wiley & Sons, pp. 45.
[5] Madias, Jorge. "Electric furnace steelmaking." Treatise on process metallurgy. Elsevier, 2014. 271-300.
[6] K.E. Daehn, A. C. Serrenho, J. M. Allwood, 2017, “How will copper contamination constrain future global steel
recycling,” Environ. Sci. Technol., Vol. 51, pp. 6599-6606.
[7] Latad, S. K., et al. (2019) "Automatic Object Sorting Machine."
[8] Knapp, Henning, et al. (2014) "Viable Applications of Sensor‐Based Sorting for the Processing of Mineral
Resources." ChemBioEng Reviews 1.3: 86-95.
[9] Pearson, Thomas (2010) "High-speed sorting of grains by color and surface texture." Applied engineering in
agriculture 26.3: 499-505.
[10] U. M. Ibrahim, S. A. Adeshina, S. Thomas, A. N. Obadiah, S. Hussein and O. E. Aina (2019) "Design and
Implementation of a Plastic Waste Sorting System," 2019 15th International Conference on Electronics, Computer
and Computation (ICECCO), Abuja, Nigeria, pp. 1-4.
[11] S. Kashiwakura, K. Wagatsuma, 2013, “Characteristics of the calibration curves of copper for the rapid sorting
of steel scrap by means of laser-induces breakdown spectroscopy under air atmosphers,” Analytical Sciences, Vol. 29,
pp. 1159-1164.
[12] Mesina, M. B., T. P. R. De Jong, and W. L. Dalmijn. "Automatic sorting of scrap metals with a combined
electromagnetic and dual energy X-ray transmission sensor." International Journal of Mineral Processing 82.4 (2007):
222-232.
[13] Günter Buzanich, 2016, “A newly developed XRF-Sensor with high sensitivity for increasing sorting efficiency,”
7th Sensor-Based Sorting & Control 2016, pp. 221-232.
[14] Alvin D. Shulman, 2008, “Method for Bulk Sorting Shredded Scrap Metal,” United States Patent, No. 7886915
B2, 24 pp.
11
[15] A. Vailaya, M. A. T. Figueiredo, A. K. Jain and Hong-Jiang Zhang (2001) "Image classification for contentbased indexing," in IEEE Transactions on Image Processing, vol. 10, no. 1, pp. 117-130.
[16] Guo, Yanming, et al. (2016) "Deep learning for visual understanding: A review." Neurocomputing 187: 27-48.
[17] Hassan, M. (2019) "VGG16: Convolutional Network for Classification and Detection."
[18] Wang, Danshi, et al. (2017) "Modulation format recognition and OSNR estimation using CNN-based deep
learning." IEEE Photonics Technology Letters 29.19: 1667-1670.
[19] Rangarajan Aravind, Krishnaswamy, and Purushothaman Raja (2020) "Automated disease classification in
(Selected) agricultural crops using transfer learning." Automatika 61.2: 260-272.
[20] Rosebrock, Adrian (2017) "Imagenet: Vggnet, resnet, inception, and xception with keras."
[21] Rosebrock, Adrian (2018) "Keras and Convolutional Neural Networks (CNNs)."
[22] Chollet, François. "Xception: Deep learning with depthwise separable convolutions." Proceedings of the IEEE
conference on computer vision and pattern recognition. 2017.
[23] Kumar, Pradeep (2003) "Metal scrap sorting system." U.S. Patent No. 6,545,240.
[24] Huellen, Markus, et al. (2006) "EAF-Based Flat-Steel Production Applying Secondary Metallurgical
Processes." Ironmaking Steelmaking Conference, Linz, Austria.
[25] Fruehan, R. J., et al. (2000) Theoretical minimum energies to produce steel for selected conditions. Carnegie
Mellon University, Pittsburgh, PA (US); Energetics, Inc., Columbia, MD (US).
[26] Wei, Tan Chiang, U. U. Sheikh, and Ab Al-Hadi Ab Rahman (2018) "Improved optical character recognition
with deep neural network." 2018 IEEE 14th International Colloquium on Signal Processing & Its Applications (CSPA).
IEEE.
12