2010 1st International Conference on Applied Robotics for the Power Industry Delta Centre-Ville Montréal, Canada, October 5-7, 2010 1 Robotized Inspection of Power Lines with Infrared Vision Jonathan Henrique Efigênio de Oliveira, and Walter Fetter Lages, Member, IEEE Abstract—This paper presents a method to automatically detect faults in transmission lines by using thermographic images. It involves the acquisition of infrared images over a network and its processing. Since the method is intented to be used embedded in a robot, low demands on processing power are required, in order to enable inspection on large extensions of transmission lines. The automatic inspection of power lines, allows for inspections over a longer period of time and with a decreased risk for the professionals involved. Experimental results show that the proposed method is able to detect cables with problems. Index Terms—Filed robotics, Automation, Sensor, Vision, Implementation, Inspection, Maintenance. I. I NTRODUCTION T HE methods for inspection of power transmission lines are expensive, dangerous and inaccurate. This article presents a method for making preventive inspections of transmission power lines. The preventive inspections are important as they will be able to detect faults before they occur. The proposed method is based on thermographic analysis of the line by using a infrared camera mounted on a robot that moves on the line. This way, there is a decrease in the need for maintenance personnel to stay close to energized lines and therefore an increase in safety, besides enabling a more accurate process since operator fatigue is avoided. Typically, problems on cables, connections and even insulators have a direct consequence on their electrical resistance and therefore on the electrical current through the device, increasing the temperature around the point with problem. Note that although the usual behavior of conductors (cables and connections) and insulators are very different, under failure both present an increase in temperature. In conductors, the increase in the temperature is due to an increase in their electrical resistance, while in insulators, the increase in the temperature is due to an increase in the leak current through them. Thus an abnormal increasing in temperature can be associated to a failing device. Thermographic inspection has been growing in various industry segments, since it is a nondestructive method [1], is fast and does not require contact with the body under inspection. For electrical plants, it is even more convenient, since in general it is easier to perform the inspection without the interruption of system operation. The visual inspection of power lines, even if just searching for hotspots in infrared images, is a very monotonous and Authors are with the Department of Electrical Engineering, Federal University of Rio Grande do Sul, Porto Alegre, RS, 90035-190 BRAZIL e-mails: jonathan@ece.ufrgs.br, fetter@ece.ufrgs.br. 978-1-4244-6634-4/10/$26.00 ©2010 IEEE tedious task and therefore if done by human beings it is very prone to fatigue of the operator. The method proposed in this paper overcomes this problem by performing the analysis of the infrared images automatically. Also, since the robot is equipped with a differential GPS receiver, detected problems are georeferenced so that they can be easily located by maintenance teams. Furthermore, since the camera is mounted on a robot moving on the line (aerial or underground), the image is obtained very close to the line, thus avoiding the wellknown problem of averaging of thermal pixels. The averaging of pixels can hide hotspots in images taken far from the object under inspection. This paper describes the procedures to acquire images from the camera and the processing of the images to detect potential problems on conductors and insulators. Details on the robot can be found on [2]–[5]. The thermographic camera is an IP camera working as a RTSP (Real-Time Stream Protocol) [6] server, thus the images are provided as data streams. In order to have individual images to be automatically processed, the images should be captured from the received data stream. The location of each object under risk of failure, obtained by the robot GPS, is stored along with the image and details of the analysis for further validation and schedule of maintenance teams. This article is organized as follows: Section II presents some aspects that should be taken into account when processing thermographic images. Section III addresses the main aspects of the method developed in this work. Sections V and VI present the experimental results and conclusions. II. T HERMOGRAPHIC I MAGE A thermographic image is a digital image formed by electromagnetic waves in the infrared portion of the spectrum, rather than in the range corresponding to the visible light, like images generated by usual cameras. Each pixel represents the temperature of a point in the objects in image or the energy emitted by its material. In this paper a thermographic camera from FLIR, model A320 is used (see Fig. 1). Its operation is based on microbolometers and it generates images with resolutions up to 320×240 pixels. It has two temperature ranges: from -20 o C to 120 o C or 0 o C to 250 o C, providing images in different formats like standard composite video (NTSC or PAL) or RAW and MPEG4 over an Ethernet connection. The camera provides several different streams of the same thermal image: 2 Fig. 1. Thermographic camera FLIR A320. Fig. 2. 1) MPEG4 compressed video in three resolutions (640×480, 320×240, 160×128): Useful only for presenting an operator view, since the MPEG4 compression imposes some loss on the information contained on the captured image, usually observed as pixellization of the image. 2) FCAM FLIR: This stream is in a proprietary format of the manufacturer. Since details of the format are not available, it is not useful for the general user. 3) RAW IR signal: This stream provides the raw data from the infrared sensors. Since the data has not been calibrated for the sensors characteristics, and a calibration pattern for the sensor is not usually available for the user, this stream is not very useful too. 4) linear IR temperature image in two resolutions (320×240, 160×120): These streams are thermal images calibrated in degrees Kelvin with resolutions of 0.1 K or 0.01 K. The stream with resolution of 320×240 pixels and 0.01 K is the one selected to be automatically processed, since it can be easily interpreted and has not distorted the captured data. In this paper the linear IR temperature image with a resolution of 320×240 is used. In this format, each pixel is a 16-bit value representing the temperature in Kelvin of that point. The precision can be selected to be either 0.1 K or 0.01 K. However, since the camera accuracy is ±2 o C, the precision can selected much more as a function of the desired temperature range. III. I MAGE C APTURE The camera behaves as a RTSP server providing a stream with thermographic images. Stream is a continuous flow of images wrapped in the Real Time Protocol, RTP [7]. In order to capture the stream, the processing software should talk to the camera by using the stack of protocols shown in Fig. 2. To receive streams, The client should in the first place establish an RTSP session. This session configures some parameters to be used for the streaming between the server and the client. The RTSP protocol refers to the protocol used to control the operation of the streaming server and provides commands to select the stream, obtain a description the stream properties, start, stop, pause and restart the data stream. Table I Protocol stack. TABLE I RTSP C OMMANDS S UPPORTED BY THE C AMERA Command OPTIONS DESCRIBE SETUP GETPARAMETER PLAY PAUSE TEARDOWN Description list the supported optional commands list the streams available by the camera set one RTP session using a specific stream get parameters like frame rate and file format begin send the streams pause the sending close the RTP session shows the RTSP commands supported by the camera used in this work. The connection between the server on the camera and the client processing software on the robot to transfer an image stream is called a multimedia session. Before starting a multimedia session its parameters should be configured. The Session Description Protocol, SDP [8] is used to describe the desired session parameters such as such as session name, media name, address and connection information. The SDP protocol is used in the setup phase. In this phase, the server describes the available streams to the client, which then configures the parameters for the desired stream. Figure 3 shows the reply sent by the camera to a RTSP DESCRIBE command. The SDP session description returned by the camera begins at the line which starts with v=0. Notice the lines starting with a=rtpmap: which starts the description of each stream available from the camera. The stream used in this work is the one described by a=rtpmap:103 raw/90000, a=framesize:103 320-240 and a=fmtp:103 sampling=mono; width=320; height=240; depth=16. The stream itself is sent though the RTP (Real Time Protocol) [7]. This protocol defines how the data is packed to be sent. This protocol enables the client to interpret the information that it receives and decode the stream in images. Nowadays, the Ethernet, IP, UDP and TCP protocols are standard features of most operating systems, but RTSP, RTP and SDP should be provided by extra software. This work uses the implementations of the RTSP, SDP and RTP protocols available through the Live 555 library [9]. That library has 3 RTSP/1.0 200 OK CSeq: 2 Date: 24 Jun 2010 20:57:18 GMT Content-Type: application/sdp Content-Length: 1028 Content-Base: rtsp://10.1.32.1/ v=0 o=- 0 0 IN IP4 10.1.32.1 s=IR stream i=Live infrared t=nowc=IN IP4 10.1.32.1 m=video 13124 RTP/AVP 96 97 98 100 101 103 104 a=control:rtsp://10.1.32.1/sid=96 a=framerate:30 a=rtpmap:96 MP4V-ES/90000 a=framesize:96 640-480 a=fmtp:96 profile-level-id=5;config=000001B005000001B509000001010000012002045D4C28A021E0A4C7 a=rtpmap:97 MP4V-ES/90000 a=framesize:97 320-240 a=fmtp:97 profile-level-id=5;config=000001B005000001B509000001010000012002045D4C285020F0A4C7 a=rtpmap:98 MP4V-ES/90000 a=framesize:98 160-128 a=fmtp:98 profile-level-id=5;config=000001B005000001B509000001010000012002045D4C28282080A4C7 a=rtpmap:100 FCAM/90000 a=framesize:100 320-240 a=fmtp:100 sampling=mono; width=320; height=240; depth=16 a=rtpmap:101 FCAM/90000 a=framesize:101 160-120 a=fmtp:101 sampling=mono; width=160; height=120; depth=16 a=rtpmap:103 raw/90000 a=framesize:103 320-240 a=fmtp:103 sampling=mono; width=320; height=240; depth=16 a=rtpmap:104 raw/90000 a=framesize:104 160-120 a=fmtp:104 sampling=mono; width=160; height=120; depth=16 Fig. 3. Fig. 4. Original radiometric image. Fig. 5. Segmented image. SDP session description returned by the camera. C++ classes to start a session and send RTSP commands to control the server (the camera). In order to receive an thermographic image the client should send the commands to choose the stream (in this case RAW stream with a resolution of 320×240) and then send the command for the server start sending the RTP packets. On reception the packets are extracted from the stream. The payload of each RTP packet is according to the RFC4715 (RTP Payload Format for Uncompressed Video) [10]. Each of these arrays is a image that will be analyzed to detect flaws. The analysis of the infrared images is based on the detection of meaningful hotspots and an assessment of the degree of risk of failure that they represent. Depending on the temperature of hotspots and its morphology it will be considered mediumrisk, serious-risk or imminent-risk of failure. IV. FAILURE D ETECTION The purpose of this algorithm is to process the images to detect hotspots and decide weather it represents a problem on the transmission line or not. Each received image is processed for inspection, searching for the existence and severity of failures. The proposed method is based on the Infrared Thermography Anomaly Detection Algorithm (ITADA) proposed in [11]. The original ITADA was proposed for inspection of electrical equipment in operation and is based on detection of high temperatures and/or extreme variations in temperature. Furthermore, it uses a visual 8 bit palette-based image, while in this paper a 16 bit radiometric image is used (see Fig. 4). As in many computer vision methods, the first step is the segmentation of the image. To segment an image is to separate the target object(s) from the background. In this case, the background are the areas with low temperatures. Therefore the result of the segmentation step is an image where only the interesting high temperatures areas have a temperature value different from the background. Furthermore, the background itself is adjusted to have a single value, which can be easily distinguished from the high temperatures areas, usually 0. The segmentation is done by thresholding the image with a value determined in a similar way to [12]. Therefore, all pixels with temperature below the threshold value is considered background and has its temperature set to 0 (see Fig. 5). Let α be the original image(see, for example 9(b)) and β the binarized version of α, T is the threshold value from the Otsu method, W and H the width and height of he image, x and y the coordinates of a pixel. Then, the segmented image is: ( α(x, y), if α(x, y) ≥ T γ(x, y) = 0, if α(x, y) < T ∀0 ≤ x ≤ W, 0 ≤ y ≤ H The hotspots are then detected on the segmented image. 4 Fig. 6. Hottest pixels. Fig. 7. First the pixels with highest temperature value are located (see Fig. 6). Thot = max 0≤W, 0≤H γ(x, y) (1) These pixels with highest temperature value are used to all connected hot ares. Each Thot is used as a seed for a dilation, denoted by Ω0 . ( 1 if α(x, y) = Thot Ω0 (x, y) = ∀0 ≤ x ≤ W, 0 ≤ y ≤ H 0 otherwise (2) Then, a dilation operation is used to find connected hot areas around the hotspots, resulting in a set of hot areas as shown in Fig. 7: Ωk = (Ωk − 1 ⊕ B) ∩ C k = 1, 2, 3, .. (3) where B is an 8-neighbors mask, C is a constraint representing the experimental limit on the gradient between neighbors in the foreground image γ (a value of 16 was adopted). The algorithm converges when Ωk = Ωk−1 , resulting in the converged image Ω∗ , which represents the all hotspots in the thermographic image. The very small areas with less than 5 pixels are regarded as noise and are discarded for the purposes of subsequent analysis. The largest hotspot is used to characterize the situation of the cable. The temperature of the largest hotspot is defined by the mean value of its pixels: Thot = 1 D W −1 H−1 X X x=0 y=0 ! γ(x, y), ∀ (x, y) ∈ A (4) where A represents all pixels of the largest hotspot and D the number of pixels in the largest hotspot. The criticality of each detected hotspot is determined by a look-up table, with values based on standards such from IEC Hotspots. (International Electrical Commission) or ABNT (Brazilian Association of Technical Standards). The criticality of a hotspot can be defined by a “quantitative” or a “qualitative” analysis of its temperature. The quantitative analysis considers the exact measured temperature of the hotspot. This method is generally not as important, as the accuracy of these values are often affected by environmental factors such as the current ambient temperature, humidity and emissivity, etc.. The “qualitative” analysis considers the temperature values for a hotspot in relation to other parts of the equipment with similar conditions, based on the delta of temperature defined as ∆T = Thot − Tref (5) where Tref is the temperature of the component if it were operating normally. It is computed from the temperature of pixels not present in any hotspot: Tref = M N (6) where M= W −1 H−1 X X γ(x, y), if Ω∗ (x, y) 6= 1 (7) γ(x, y) , ifΩ∗ (x, y) 6= 1 γ(x, y) (8) x=0 y=0 N= W −1 H−1 X X x=0 y=0 Table II presents absolute temperature values and Table III presents relative temperature values used to decide on the criticality of each hotspot. The reference temperature, Tref , takes into account the voltage levels temperature of the background. It is generally a value around 70 o C [13]. This value can vary due to cable manufacturer and climatic conditions. Hence, the actual temperature of the coldest parts of the image is used as reference, which represents just a portion of cable that is free of flaws. 5 TABLE II Q UANTITATIVE A NALYSIS Condition Normal Not Serious Medium Serious Emergency Temperature Limits (o C) Thot ≤ 68.3 68.3 < Thot ≤ 76.7 76.7 < Thot ≤ 85.0 85.0 < Thot ≤ 98.9 98.9 < Thot TABLE III Q UALITATIVE A NALYSIS Condition Normal Not Serious Medium Serious Emergency Temperature Limits (o C) ∆T ≤ 10 10 < ∆T ≤ 20 20 < ∆T ≤ 30 30 < ∆T ≤ 40 40 < ∆T (a) Visual image. V. E XPERIMENTS An aluminum cable with 25mm2 section (see Fig. 8(a)) is used in this experiment. It was submitted to a current of 500 A. The thermographic image is shown in Fig. 8(b). In this image it is possible to see a warm region, detected as the hotspot shown in Fig. 8(c), but this region have an acceptable temperature. It is just about 5 o C warmer than the cold areas. The qualitative and quantitative analysis results in a no fail decision. Then the very same cable, has been damaged, lowering its conductivity, as shown in Fig. 9(a), and a new thermographic image was obtained as shown in Fig. 9(b). Note that at the failure point there is an increase in the temperature relative to the previous image. The temperature on the cable in the first image is around 27 o C, the temperature at the point where the failure occurs is around 42 o C. The here proposed method detected the region where the damage occurred as a hotspot (see Fig. 9(c)). This temperature does not pass the quantitative limits test. However, the qualitative analysis results in a damage. (b) Thermographic image. VI. C ONCLUSION In this paper a method for detection of failures from thermographic images was presented. The next step is to submit aluminum cables, like those used in transmission lines, to currents on the order of hundreds of Amperes, simulate a damage in it and automatically detect this fault. ACKNOWLEDGMENT The authors would like to thank to Companhia Estadual de Energia Elétrica (CEEE), Coordenação de Aperfeiçoamento de Pessoal de Nı́vel Superior (CAPES) and Conselho Nacional de Pesquisa (CNPq) for the financial support. R EFERENCES [1] F. Chunli, S. Fengrui, and Y. Li, “Investigation on nondestructive evaluation of pipes using infrared thermography,” in Proceedings of the IEEE International Conference on Terahertz Electronics, vol. 2, Sept. 2005, pp. 339–340. (c) Hotspot. Fig. 8. Aluminum cable without problems. [2] V. M. de Oliveira and W. F. Lages, “Linear predictive control of a brachiation robot,” in IEEE Canadian Conference on Electrical and Computer Engineering. Ottawa, Canada: IEEE, May 2006, pp. 1517– 1520. [3] ——, “Predictive control of an underactuated brachiation robot,” in 6 Elsevier, 2006. [4] ——, “MPC applied to motion control of an underactuated brachiation robot,” in Proceedings of the 11th IEEE International Conference on Emerging Technologies and Factory Automation. Prague: IEEE Press, 2006. [5] ——, “Comparison between two actuation schemes for underctuated brachiation robots,” in Proceedings of the 2007 IEEE/ASME International Conference on Advanced Intelligent Mechatronics, Zürich, 2007. [6] H. Schulzrinne, A. Rao, and R. Lanphier, “Real time streaming protocol (RTSP),” Network Working Group, RFC 2326, April 1998, available at hftp://ftp.ietf.org/rfc/rfc2326.txti. [7] H. Schulzrinne, S. Casner, R. Frederick, and V. Jacobson, “RTP: A transport protocol for real-time applications,” Network Working Group, RFC 1889, January 1996, available at hftp://ftp.ietf.org/rfc/rfc1889.txti. [8] M. Handley, V. Jacobson, and C. Perkins, “SDP: Session description protocol,” Network Working Group, RFC 4566, July 2006, available at hftp://ftp.ietf.org/rfc/rfc4566.txti. [9] LIVE555 Streaming Media, Live Networks, Inc., Mointain View, CA, available at hhttp://www.live555.com/LiveMediai. [10] L. Gharai and C. Perkins, “RTP payload format for uncompressed video,” Network Working Group, RFC 4566, September 2005, available at hftp://ftp.ietf.org/rfc/rfc4175.txti. [11] Y.-C. Chou and L. Yao, “Automatic diagnosis system of electrical equipment using infrared thermography,” in Proceedings of the 2009 International Conference on Soft Computing and Pattern Recognition. IEEE Computer Society, 2009, pp. 155–160. [12] N. Otsu, “A threshold selection method from gray-level histograms,” Systems, Man and Cybernetics, IEEE Transactions on, vol. 9, no. 1, pp. 62–66, jan. 1979. [13] I. E. C. (IEC), “IEC 60826: Design criteria of overhead transmissionlines,” 2006. (a) Visual image. Jonathan Henrique Efignio de Oliveira received the B.Sc. in Electrical Engineering from the Universidade Federal do Rio Grande do Sul (UFRGS), Porto Alegre, Brazil, in 2007 and is working towards his M.Sc. degree in Electrical Engineering at the same University. (b) Thermographic image. Walter Fetter Lages (S’91, M’99) received the B.Sc. in Electrical Engineering from Pontifı́cia Universidade Católica do Rio Grande do Sul (PUCRS), Porto Alegre, Brazil in 1989 and the M.Sc. and D.Sc. degrees in Electronic and Computer Engineering from Instituto Tecnológico de Aeronáutica (ITA), São José dos Campos, Brazil in 1993 and 1998, respectively. From 1997 to 1999 he was an Adjoint Professor in the Physics Department of the Fundação Universidade Federal do Rio Grande (FURG), Rio Grande Brazil. Currently he is an Associate Professor in the Electrical Engineering Department of the Federal University of Rio Grande do Sul (UFRGS), Porto Alegre, Brazil. Dr. Lages is a member of IEEE, ACM, Brazilian Automation Society and Brazilian Computer Society. (c) Hotspot in the damaged aluminum cable. Fig. 9. Damaged aluminum cable. Proceedings of the 8th IFAC Symposium on Robot Control. Bologna: