Available online at www.sciencedirect.com Available online at www.sciencedirect.com ScienceDirect ScienceDirect Available online atonline www.sciencedirect.com Procedia CIRP 00 (2019) 000–000 Available at www.sciencedirect.com ScienceDirect ScienceDirect Procedia CIRP 00 (2019) 000–000 www.elsevier.com/locate/procedia www.elsevier.com/locate/procedia Procedia CIRP 00 (2017) 000–000 Procedia CIRP 81 (2019) 771–774 52nd CIRP Conference on Manufacturing Systems 52nd CIRP Conference on Manufacturing Systems www.elsevier.com/locate/procedia A visual inspection system for KTL coatings CIRP inspection Design Conference, May for 2018,KTL Nantes,coatings France A28th visual system a* b Drago Bračun and Igor Lekše a* b a Dragoof Bračun and Igor Lekše methodology analyze the functional and physical architecture University ofto Ljubljana, Faculty Mechanical Engineering, Aškerčeva cesta 6, SI-1000 Ljubljana A new of TPV d.o.o., Kandijska cesta 60, SI-8000 Novo mesto University of Ljubljana, Faculty of Mechanical Engineering, Aškerčeva cesta 6,family SI-1000 Ljubljana existing products for an assembly oriented product identification * Corresponding author. Tel.: +386-1-4771 747; fax: + TPV 386-1-2518 567. E-mail address: drago.bracun@fs.uni-lj.si d.o.o., Kandijska cesta 60, SI-8000 Novo mesto a b b * Corresponding author. Tel.: +386-1-4771 747; fax: + 386-1-2518 567. E-mail address: drago.bracun@fs.uni-lj.si Abstract Abstract Paul Stief *, Jean-Yves Dantan, Alain Etienne, Ali Siadat École Nationale Supérieure d’Arts et Métiers, Arts et Métiers ParisTech, LCFC EA 4495, 4 Rue Augustin Fresnel, Metz 57078, France Many metal components in automotive industry are surface-protected using the electrophoretic cathode metal coating (KTL). As defects can Tel.: occur the components need to be 100% inspected, especially ifcathode the manufacturers are (KTL). committed *surface Corresponding author. +33in 87 37process, 54 30; E-mail address: Many metal components in3 this automotive industry arepaul.stief@ensam.eu surface-protected using the electrophoretic metal coating As to delivering zero-defect Due to the complicated 3D need curved andinspected, black shiny colour ofifthe inspection typically surface defects can occurproducts. in this process, components to shapes be 100% especially thecoating, manufacturers are is committed carried out manually. To avoid labour-intensive manual inspection in serial production, an automated visual inspection system is to delivering zero-defect products. Due to complicated 3D curved shapes and black shiny colour of the coating, inspection is typically developed. The paper presents the implementation of a control device for mass inspection of the products, typical KLT coating carried out manually. To avoid labour-intensive manual inspection in serial production, an automated visual inspection system is Abstract defects, andThe the visual method for the detection of coating defects. A difference image acquisition method implemented developed. paper inspection presents the implementation of a control device for mass inspection of the products, typicalis KLT coating in order to reveal the defects on the black shiny colored parts. An image processing algorithm for the recognition of light patterns defects, and the visual inspection method for the detection of coating defects. A difference image acquisition method is implemented Inand today’s business environment, the trend towardsofmore product variety and customization is unbroken. Due to this development, the need of a clustering for the recognition are presented. in order to revealapproach the defects on systems the black shinydefects colored parts. An image processing algorithm recognition of light patterns agile and reconfigurable production emerged to cope with various products and product families.for Tothe design and optimize production and a clustering recognition of defectsproduct are presented. systems as well as approach to choose for the the optimal product matches, analysis methods are needed. Indeed, most of the known methods aim to © 2019 The Authors. Published by Elsevier Ltd. This is an open access article under the CC BY-NC-ND license analyze product or one product family on the physical level. Different product families, however, may differ largely in terms of the number and © 2019aThe Authors. Published by Elsevier Ltd. (http://creativecommons.org/licenses/by-nc-nd/3.0/) © 2019 The Authors. Published by Elsevier Ltd. This iscomparison an open(http://creativecommons.org/licenses/by-nc-nd/3.0/) access under the CC BY-NC-ND license nature of components. This fact impedes an efficient and article choice of appropriate product family combinations for the production This is an open access article under the CC BY-NC-ND license Peer-review under responsibility of the scientific committee of the 52nd CIRP Conference on Manufacturing Systems. (http://creativecommons.org/licenses/by-nc-nd/3.0/) system. A newunder methodology is proposed to analyze existing products in view their functional and physicalSystems. architecture. The aim is to cluster Peer-review responsibility of the scientific committee of the 52nd CIRPof Conference on Manufacturing Peer-review under responsibility of the scientific committee of the 52nd CIRPofConference on Manufacturing Systems. these products in surface new assembly oriented product families for the optimization Keywords: KTL; defect; inspection; imaging system; pattern recognition; cluster existing assembly lines and the creation of future reconfigurable assembly systems. Based on Datum Flow Chain, the physical structure of the products is analyzed. Functional subassemblies are identified, and surface inspection; imagingasystem; recognition; cluster architecture graph (HyFPAG) is the output which depicts the a Keywords: functionalKTL; analysis is defect; performed. Moreover, hybridpattern functional and physical similarity between product families by providing design support to both, production system planners and product designers. An illustrative coating,caseinspection typically carried outcolumns manually. example of a nail-clipper is used to explain the proposed methodology. An industrial study on twoisproduct families of steering of 1. Introduction Regardless of the staff experience, manual inspection is thyssenkrupp Presta France is then carried out to give a first industrial evaluation of the proposed approach. coating, inspection is typically carried out manually. © 2017 The Authors. Published by Elsevier B.V. subjective and time-consuming in large-series production. 1. Introduction Regardless of the staff experience, manual inspection is Peer-review under responsibility the scientific committee of the 28th 2018. shows several specific solutions for Electrophoretic cathode of metal coating (KTL or E-coat) [1] isCIRP Design The Conference literature review subjective and time-consuming in large-series production. surface defect detection. system described [2] acquires The literature review The shows several specificinsolutions for multiple images under different lighting conditions, then surface defect detection. The system described in [2] and acquires these images are processed separately and merged into and one. then The multiple images under different lighting conditions, features extracted from this fused image and the former these images are processed separately and merged into one. The processing steps the feature space for supervised of features the product rangeestablish and manufactured and/or extracted fromcharacteristics this fused image and a the former learning classifier based on artificial neural networks. The assembled in this system. In this main in processing steps establish thecontext, feature the space forchallenge a supervised system presented in [3] detects defects catalogued as dings and modelling analysis is now only to copenetworks. with single learning and classifier based on not artificial neural The dents on car body surfaces. It sweeps the object with the light products, limited product range defects or existing productasfamilies, system apresented in [3] detects catalogued dings and patterns, and uses fusion on flow butdents also to to image analyze and tobased compare products to to define onbe carable body surfaces. It sweeps theoptical object with theobtain light a resulting fused image which holds the information of all new productand families. It canfusion be observed that classical patterns, uses image based on optical flowexisting to obtain suffered by the product families are regrouped in projected function clients or during features. avariations resulting fused image which holds of thepatterns information of the all sweeping process, indicating the presence of anomalies. A However, assembly oriented product familiespatterns are hardly to find.the variations suffered by the projected during deflectometry method [4] features a high resolution camera and On the product family level, products differ of mainly in two A sweeping process, indicating the presence anomalies. a dedicated illumination system of on displaying main characteristics: (i) the components and (ii) fringe theand deflectometry method [4] number features abased high resolution camera in a monitor, the based detection irregularities in type of components (e.g.allowing mechanical, electrical, electronical). apatterns dedicated illumination system on of displaying fringe surfaces. An overview of three different image acquisition patterns a monitor, allowing the detection irregularities Classicalinmethodologies considering mainly of single products in Because of complicated shapes and black shiny color [1]. of the or surfaces. overview of three different image acquisition product varieties (high-volume to low-volume production) solitary, An already existing product families analyze the 2212-8271 © 2019 The Authors. Published by Elsevier Ltd. This is an open access article under the CC BY-NC-ND license To cope with this augmenting variety as well as to be able to product structure on a physical level (components level) which (http://creativecommons.org/licenses/by-nc-nd/3.0/) 2212-8271 ©under 2019responsibility The optimization Authors. of Published by Elsevier Ltd. an open access article under the CC BY-NC-ND license an efficient definition and identify potentials in This existing causes regarding Peer-reviewpossible the scientific committee ofthe theis52nd CIRP Conference on difficulties Manufacturing Systems. (http://creativecommons.org/licenses/by-nc-nd/3.0/) production system, it is important to have a precise knowledge comparison of different product families. Addressing this an Electrophoretic important technological process for painting protecting cathode metal coating (KTL orand E-coat) [1] is Keywords: Assembly; Design method; Family identification the parts from environmental impacts. The quality of coating is an important technological process for painting and protecting achieved by managing the coating process and byofinspecting the parts from environmental impacts. The quality coating is parts after coating. There are a number of different coating achieved by managing the coating process and by inspecting defects, including: craters (bowl-shaped depressions), 1. Introduction parts after coating. There are a number of different coating roughness including: (patches on acraters cured film that exhibits an alternately defects, (bowl-shaped depressions), non-uniform and smooth appearance), redissolution (part ofofthe Due to (patches the faston development in exhibits the domain roughness a cured film that an alternately coat film washes off or dissolves), dirt (three sources: process, communication andsmooth an ongoing trend redissolution of digitization non-uniform and appearance), (partand of the environmental and oven), streaking (duefacing to pretreatment, digitalization, manufacturing enterprises important coat film washes off or dissolves), dirt are (three sources: process, rinsing and pinholing (a pattern relatively small, challenges in racking), today’s market environments: a continuing environmental and oven), streaking (dueof to pretreatment, random volcano-like holes), air entrapment, gloss variations, tendency reduction of product development times and rinsingtowards and racking), pinholing (a pattern of relatively small, color variations, thinholes), coating, corrosion and peel. shortened product lifecycles. In addition, there is an orange increasing random volcano-like air entrapment, gloss variations, Example of defects are shown in Fig. 1. Coating defects on demand customization, being at corrosion the same time a global color ofvariations, thin coating, and in orange peel. products should be identified immediately after the coating competition with competitors all over the 1.world. Thisdefects trend, on Example of defects are shown in Fig. Coating process. For that be purpose, coated parts aremacro visually inspected. which is inducing the identified development from micro products should immediately after to the coating Because of complicated shapes black shiny color of the process. For that coated parts are inspected. markets, results in purpose, diminished lot and sizes due visually to augmenting Peer-review under responsibility of the scientific committee of the 52nd CIRP Conference on Manufacturing Systems. 2212-8271 © 2019 The Authors. Published by Elsevier Ltd. This is an©open article Published under theby CC BY-NC-ND 2212-8271 2017access The Authors. Elsevier B.V. license (http://creativecommons.org/licenses/by-nc-nd/3.0/) Peer-review under responsibility of scientific the scientific committee theCIRP 52ndDesign CIRPConference Conference2018. on Manufacturing Systems. Peer-review under responsibility of the committee of the of 28th 10.1016/j.procir.2019.03.192 772 2 Drago Bračun et al. / Procedia CIRP 81 (2019) 771–774 Drago Bračun, Igor Lekše / Procedia CIRP 00 (2019) 000–000 approaches for automatic visual inspection of metallic surfaces is presented in [5]. The first method is gray-level intensity imaging with bright and dark field lighting techniques. Subsequently, two range imaging techniques are introduced which may succeed in contrast to intensity imaging if the reflection property across the intact surface changes. However, range imaging for surface inspection is restricted to surface defects with three-dimensional characteristics, e.g. cavities. A surface defect detection system for mobile phone screen glass [6] corrects the image misalignment and then the combination of subtraction and projection is used to identify defects. To segment the defects, an improved fuzzy c-means cluster algorithm is developed. The vision system for on-line surface inspection in an aluminum casting process [7] implements similarity-based algorithms as well as texture algorithms for defect detection in a high-textured surface where, for most defects, segmentation cannot be achieved solely by means of threshold techniques. The system was applied to continuous cast aluminum inspection, where up to fifteen different kinds of defects must be detected and classified. Fig. 2. Automated visual inspection system [8]. Fig. 3. Part transportation system with AGV-s. 3. Image processing Fig. 1. Examples of coating defects: (a) coagulated paint; (b) air entrapment; (c) roughness; (d) corrosion; (e) redissolution; (f) dirt. 2. Automated visual inspection system The automated visual inspection system presented in Fig. 2 is integrated into the part transportation system from the KTL process to the automated warehouse. The part transportation system is carried out with AGV’s and industrial robots, as is demonstrated in Fig. 3. The inspection is carried out employing 16 cameras and carefully designed illumination for each camera. The inspection device has four nests, of which the first and the last are used for robotic part handling, and the middle two are intended for visual inspection. Part transportation between the nests is carried out with two servo-drive based part transportation systems. The control of the device is carried out with the PLC and i7 based industrial PC for image processing. The challenges which were encountered and are presented in the continuation of this paper are detection of defects on complicated 3D curved shapes with black shiny coating (see Fig. 4) and the distinction between critical defects and visual flaws. Coating defects are usually visible as blobs of different image brightness, as is demonstrated in Fig. 4. They are typically brighter than their surroundings, varying in size from a few pixels to larger parts of the image. They can also take the form of a cluster of small blobs. In the image processing stage, these light blobs are initially extracted from the image and then classified either as actual defects or as visual flaws. Their extraction is carried out first by employing threshold values to the image to obtain a mask in form of binary image, with pixel values 0 or 1. The value is 1 if the pixel brightness in the image of interest is brighter than the threshold value, and 0 if it is below. Blobs are identified as areas where the value of adjacent pixels is 1. After being identified, their area i.e the number of adjacent pixels with value 1, is calculated. The initial criterion for the defect identification is the area of the blob as well as other geometric properties, e.g. location and shape. Fig. 4. Image A. A detail of a typical KTL-coated part with 3D form and black shiny color. Arrows point to the coating damage, dust and fingerprint. Drago Bračun et al. / Procedia CIRP 81 (2019) 771–774 Drago Bračun, Igor Lekše / Procedia CIRP 00 (2019) 000–000 However, blob extraction is not so straightforward since the coated objects are curved in all directions. Feature recognition in imaging systems typically starts with the illumination of the object, which should be adjusted so as to reveal the features of interest, i.e. bright or dark field illumination. At curved objects, it is not possible to adjust the illumination to all object features, and from some parts of the object a bit of light reflection will occur directly in the camera while at others the light will reflect away. These reflections resemble the defects in image, which interferes with image processing. One solution to the problem is to create a background image B (see Fig. 5) that is subtracted from the image of interest A (see Fig. 4). The background image represents the average of a large number of images under the same lighting conditions and the same image acquisition setup. These images are acquired by inspecting the same type of product in many products without defects (>20). 773 3 second image. There are three parameters (Tx , Ty) for translation and angle for rotation. The estimation of all motion parameters can be time-consuming in case of many large images. A simplification based on the knowledge of the product clamping inside the inspection device can be carried out. It was found that variability is most significant in positioning the part inside the inspection device and that the main motion is rotation, while the translation can be neglected. In addition, since the point of rotation is known, it is only necessary to identify the value of rotation. The search algorithm is straightforward: (i) rotate the image A with respect to the image B in sequence of small steps, (ii) in each step calculate the difference image D and (iii) calculate the sum off all pixels on the image D. The rotation angle with the minimum sum is the best alignment angle. Variability in dimensions among parts still remains and is most noticeable in the image D at the edges of the product where there is no overlap between the parts due to difference in shape (see Fig. 6). These bright edges are removed employing an additional mask that specifies the region of interest. Further image processing continues as already described at the beginning of this chapter, by thresholding, blob search, calculation of blob properties and defect identification. 4. Defect identification Fig. 5. The background image B. It is the average of a large number of images (>20) under the same lighting conditions and the same image acquisition setup. In many cases, defects do not take the form of big blobs but rather appear as concentrated groups of small blobs. Fig.7 shows such an example. The left white circle marks a group of large area blobs that are actually a surface defect, while the right circle marks the group of small blobs that show dust on the surface and are not a defect. Defect identification that searches for a big blob does not recognize any of these blobs as a defect. To solve this problem, the blobs are grouped into clusters, and for each cluster the parameter quantifying the likelihood of a defect is calculated. Fig. 6. The difference image D is the image A with removed background B. By subtracting the background image B from the image of interest A and calculating the absolute value of the differences, we eliminate brightness variations caused by the object form. If we assume that images A, B and D are arrays of same size and data type, with number of rows U and columns V, where and indices i,j index pixel values from i = 0 to U, and j = 0 to V. The difference image is calculated pixel wise as Di,j=abs(Ai,j - Bi,j), where abs stands for absolute value (see Fig. 6). If there are visible defects in image A, they are also present in difference image D, which can be observed by comparing Fig. 4 and Fig. 6. The weak point of this method is in the alignment of images. The object of interest must be at the same position on both images. This cannotbe perfectly achieved, because there are two main sources of misalignment or variability: the first is variability among parts, i.e. how repeatable their dimensions are, and the second is in positioning of products inside the inspection device. Both can be minimized to a certain extent, but never completely eliminated. To deal with them, a software alignment of images is carried out based on Euclidean motion, where the first image is rotated and shifted with respect to the Fig. 7. Defects are visible as groups of small blobs. For grouping the blobs, a k-means clustering algorithm is employed. The basic idea behind this clustering is to define clusters such that the total intra-cluster variation is minimized. The standard k-means clustering algorithm defines the total within-cluster variation as the sum of squared Euclidean distances (total within-cluster sum of squares TSS) between items (blob centers) and the corresponding cluster centroid. Let us assume that we arrive at k clusters. For each cluster from k = 1 to K a total area of all blobs can be calculated as N Ak = al , l =1 (1) Drago Bračun et al. / Procedia CIRP 81 (2019) 771–774 Drago Bračun, Igor Lekše / Procedia CIRP 00 (2019) 000–000 774 4 where 𝑎𝑎𝑙𝑙 is the blob area and N is the number of blobs in cluster k. Similarly, a cluster mean radius Rk is calculated as N Rk = (r a ) l =1 l Al l (2) , where rl is the distance from the cluster centroid to the center of l-th blob. The parameter E which we use to quantify the likelihood of a defect can be now defined as Ek = Ak . Rk (3) The likelihood of a defect is high when the total area Ak of blobs in cluster is high (big blobs), and when they are concentrated within small radius Rk. This is demonstrated on the left cluster in Fig.7, where E amounts to 1.12. On the right cluster, with dust on the surface, this parameter amounts to only 0.27. Testing on many other images revealed that this approach reliably separates actual defects, e.g. scratches and coagulated paint, from dust and other small irregularities that are spread over larger surfaces. An important issue with the k-means algorithm is that it requires the user to specify the number of clusters k to be generated. A simple and popular solution is to inspect the dendrogram produced using hierarchical clustering to see if it suggests a particular number of clusters. For example, the Elbow method looks at the total TSS as a function of the number of clusters and proposes to choose the number of clusters so that adding another cluster does not improve the total TSS considerably. In our test we did not observe TSS but the maximum Ek as the function of the number of clusters k. Table 1. shows the results of such an analysis for a single image. could be a problem without controlled environment. An automated inspection device is therefore carefully designed to meet all these requirements. In many cases surface defects are visible as concentrated groups of small blobs. For that purpose, a special parameter E, which defines the likelihood of a defect in a cluster, is developed. Serial production testing has shown that apart from the parameter E, the maximum blob size should still be considered for selecting defective parts. The presented inspection device eliminates defective parts from further processes. Furthermore, we are interested in providing feedback on the KTL coating process, by providing the type, location and frequency of defects. The most demanding task here is to identify the type of defect, because various defects look similar in the image. We did several initial tests with neural networks, but very ambiguous results were achieved because of similarities. Our future work is therefore aimed at the improvement of the measuring systems, as well at the identification of additional parameters that will contribute to a clearer definition of the defect type. Acknowledgements This work was supported by the Ministry of Higher Education, Science and Technology of the Republic of Slovenia, research program P2-0270, and by the TPV Group. The authors would also like to thank the staff of the TPV for their assistance and support in this research work. References [1] [2] Table 1. The parameter E with respect to the number of clusters k. k max(Ek) 2 0.338 3 0.579 4 0.753 5 0.757 6 0.850 7 0.853 [3] [4] The cluster with the maximum Ek was always in the same location when k > 2. If k is 4 and 5, Ek is almost the same, indicating that the elbow is around 4. By repeating this test on other images, similar results are achieved. For this reason, we use k = 4 at k-means clustering. [5] 5. Conclusion [7] The difference image acquisition and processing method reliably reveals surface defects on black shiny KTL-coated parts. For its successful operation, a reference image of the background is required. It must be aligned with the examined image and acquired within same lightening conditions. This [8] [6] “E-Coat (KTL).” [Online]. Available: http://www.elameta.lt/e-coatktl.html. [Accessed: 10-Feb-2019]. S. Satorres Martínez, C. Ortega Vázquez, J. Gámez García, and J. Gómez Ortega, “Quality inspection of machined metal parts using an image fusion technique,” Measurement, vol. 111, pp. 374–383, 2017. L. Arnal, J. E. Solanes, J. Molina, and J. Tornero, “Detecting dings and dents on specular car body surfaces based on optical flow,” J. Manuf. Syst., vol. 45, pp. 306–321, 2017. A. Isasi-Andrieu, E. Garrote-Contreras, P. Iriondo-Bengoa, D. Aldama-Gant, and A. Galdran, “Deflectometry setup definition for automatic chrome surface inspection,” in IEEE International Conference on Emerging Technologies and Factory Automation, ETFA, 2018. F. Pernkopf and P. O’Leary, “Image acquisition techniques for automatic visual inspection of metallic surfaces,” NDT and E International. 2003. C. Jian, J. Gao, and Y. Ao, “Automatic surface defect detection for mobile phone screen glass based on machine vision,” Appl. Soft Comput., vol. 52, pp. 348–358, 2017. C. Fernandez, C. Platero, P. Campoy, and R. Aracil, “Vision system for on-line surface inspection in aluminum casting process,” in Proceedings of IECON ’93 - 19th Annual Conference of IEEE Industrial Electronics, 1993, pp. 1854–1859 vol.3. “TPV Group.” [Online]. Available: https://www.tpv.si/en/. [Accessed: 10-Feb-2019].