DATA LEVEL FUSION OF IMAGES FROM DISPARATE SENSORS Dr. Claire L. McCullough* 326 Grote Hall, Dept. 2302 University of Tennessee at Chattanooga 615 McCallie Ave. Chattanooga, TN 37403-2598 ABSTRACT In both military and civilian applications, increasing interest is being shown in fusing outputs from disparate sensors for increased situational awareness. This interest is based on the observation that by combining information from different types of sensors, object identification (or pattern recognition) can be improved in robustness and reliability, due to the complementary nature of the sensor data included. The work discussed here successfully demonstrates fusion of both data and features of vision and infrared images. This work can be useful to such disparate applications as search and rescue, military operations, firefighting, and improved night vision for the automotive industry. 1.0 INTRODUCTION In previous work, several fusion methods have been demonstrated for combining thermal and vision images into a single image emphasizing the most salient features of the surrounding environment. The methods demonstrated have used chromatic image display, with the relative portions of these two data signals controlling the color of the image and place it somewhere between red and violet in the light spectrum , fusion based on differences between the two images, and fusion based on pixel information content . By using different fusion functions and by comparing the results with a known set of images, one can select a function that can produce optimal fusion for a given application. Both fuzzy and crisp fusion functions have been used, with each proving useful for different fusion tasks. This paper contains a continuation of this work, first in the comparison of additional data level fusion techniques, including a fuzzy “interest” based approach. This technique examines the information content of the two images by measuring the signal variations over segments of two images, generating a fuzzy function based on those measurements, and selectively combining the segments that contain the most information, in both crisp and proportional fashions. In addition, feature level techniques are compared for efficacy with the data level techniques, by extracting salient features (e.g., shapes, lines, and edges) from the fused images, and comparing this with the same features extracted from the individual images and then fused using the previously discussed techniques. The fuzzy/”interest” technique which is discussed in the paper has been demonstrated to successfully combine visual and infrared images, images with different types of noise applied, “edge” images, and Synthetic Aperture Radar, infrared, and visual images (fusing three images into one). While much work remains to be done, the new method appears to compare favorably with existing fusion techniques, and to be applicable to fusion of up to three pixel registered intensity images from a variety of disparate sensor types. 2.0 DATA FUSION Multi-sensor fusion can be approached in many different ways, which fall into three basic *Formerly with the U.S. Army Space and Missile Defense Command, PO Box 1500, Huntsville, AL, 35807-3801, for whom this work was performed. categories. In the first approach, data from all sensors is fused and then feature extraction and/or identification is carried out with fused data from all sensors (data level fusion). In the second, the relevant features are first abstracted from the data and then fused, to produce either a fused feature set, which is then used in classification, or a detection/decision based on the fused feature set (feature level fusion). In the third category, decisions/detections are based on the outputs of individual sensors, and the decisions are then fused (decision level fusion). The fusion methods discussed here are data level fusion, although these techniques can also be used with the second approach if shapes and edges are extracted prior to fusion . 2.1 “INTEREST” OF A PIXEL In fusing two images from disparate sensors, it is obviously desirable to include, or more heavily weight, pixels which are more “interesting” on some level than those which are not emphasized. “Interest” is defined in this context as relevance or importance to the scene in the images to be fused. While the definition of “interest” could vary widely for different applications, two measures which have been hypothesized for “interest” are between a pixel and each of the eight pixels which surrounds it, as shown in Figure 1, and those differences are summed to give a single value ini for each pixel. As the pixel of concern is in the center of the square, in order to deal with the edges of the image, while producing a fused image of the same size as the originals, the edge how different the pixel is from the pixels around it the relative intensity of the pixel’s color, or gray level, value. Both of these are measured for every pixel of each image to be fused, and the results incorporated into a rule-based fusion system. Fusion is performed in both a “crisp” fashion, in which the more “interesting” pixel of the pixels in the same positions of the images to be fused is inserted into the fused image, and a “proportional” fashion, where values for all the the pixels in the same positions are combined, with the weighting of each being proportional to the interest measures. 2.2 “INTEREST” BASED FUSION For the first measure of “interest,” an absolute intensity difference is calculated Figure 1. The pixel of concern is in the center of the square, with absolute intensity differences calculated to each of the surrounding eight pixels rows and columns are duplicated. That is, the “interest” ini is calculated for each original pixel on a matrix two rows and two columns larger than the original, in which the edge pixels are duplicates of those in the edge rows of the original matrix, for each sensor type. This gives a matrix of values for ini with the same dimensions as the original images. The equation for the fuzzy function of “interest” is defined as fuzini exp in mean i i spread i poweri where ini is the “interest” of a given pixel relative to sensor i, meani is the center of the fuzzy function for sensor i, spreadi is a parameter governing the width of the function, and poweri is the power to which the value is raised. Choosing this function for the fuzzy interest value gives a great deal of flexibility to the user, as by changing values for power and spread, new families of functions may be generated with very little effort by the user. This is illustrated by the graph in Figure 2, which shows the different shapes which may be generated by changing the power in the function. Upon execution of the program, the user is allowed to either accept default values for meani, spreadi, and poweri, or to enter his own for each image to be fused. The crisp version of combination is given by the rule set If fuzin1 (a, b) fuzin2 (a, b) ntens1 (a, b) ntens2 (a, b) then fuz _ crsp (a, b) 1 0.9 (value1 (a, b) value2 (a, b)) 2 0.8 0.7 (ntens1 (a, b) fuzin1 (a, b)) (ntens2 (a, b) fuzin2 (a, b)) Else, if 0.6 then fuz _ crsp (a, b) value1 (a, b) 0.5 0.4 0.3 Power=.5 fuz _ crsp(a, b) value2 (a, b) 0.2 0.1 0 Power=5 0 2 4 Power=2 6 8 Else 10 Figure 2. An illustration of the fuzzy function with differing values for Power The “intensity” measure for each pixel is simply ntensi 2 pixelvalue .5 which gives a value of “1” for pixels which are either pure white or pure black, and a value between 0 and 1 for shades of gray. Other crisp rules of combination were also evaluated for image combination, but this was one of the more visually appealing versions. 2.3 EXPERIMENTAL RESULTS All work described here was done on a 300 Mhz Pentium II with 98M of RAM, equipped with MATLAB. The software accepts input images in the .gif format, although it can be easily adapted to accept other image types as well. The images on which the algorithms will be illustrated are found in Figure 3, a vision image, and Figure 4, an infrared image. The image formed by fusion using the proportional technique is found Once these measures are calculated, the proportional version of the fused image is generated for each pixel using the rule of combination fuz _ prop (a, b) 2 fuzin n (a, b) valuen (a, b) ntensn (a, b) valuen (a, b) n 1 2 fuzin m (a, b) ntensm (a, b) m 1 where valuei is the intensity of pixel (a,b) in image i. Figure 3. Vision Image Figure 4. Infrared Image Figure 7. Edges extracted from vision image Figure 5. Image fused using the proportional technique Figure 6. Image generated using crisp fusion technique Figure 8. Edges extracted from infrared image Figure 9. Edges fused using proportional method in Figure 5, and the crisp image in Figure 6. Note that both the truck, which is clearly visible in the vision image, and the helicopter, which is visible primarily in the infrared image, are visible in the fused images. 3.0 FEATURE LEVEL FUSION In many defense applications, either before or after fusion with other information, features are extracted from an image, which can then be used in pattern recognition, automatic target classification, etc. For a scene such as those illustrated in this paper, shapes, lines, and edges would be among the features ordinarily extracted. As part of the work described here, edges were extracted from both IR and vision images, with the “edge” images then fused. The resulting image was compared to the “edge” image obtained from first fusing the images using the information-based fusion methods described in Section 2, and then extracting edges from the fused image. The reasons for this were twofold: Figure 10. Edges fused using the crisp method to see if the "interest" fusion approach was successful as a feature-level fuser, as well as a data level fusion algorithm; to see which approach, in this example, seemed to retain more of the relevant shape information from the thermal and vision images. The “edge” images from the thermal and vision images are shown in Figures 7 and 8, respectively. All edges shown here were extracted using the Adobe Photoshop image processing package. The images generated by fusing these edges is found in Figure 9 for the proportional fusion, and Figure 10 for crisp fusion. The crisp image has greater contrast, but the proportional method seems to retain more detail in the image. Figure 11. Edges of image fused with proportional method The feature images generated by extracting edges from the fused images (i.e., the images in Figures 5 and 6) are shown in Figures 11 and 12. For the examples used here, it appears that in the images in which edges were extracted from previously fused images, additional detail is retained, as compared to the image in which Figure 12. Edges of crisp fused image edges were extracted and then fused. However, the image also retains additional artifacts which might be mistakenly categorized as objects of interest. While no general conclusions can be drawn from this, it would suggest that in some cases, fusing before shapes are extracted may be beneficial. Obviously, many more images need to be examined in order to develop a general fusion/extraction strategy. 4.0 CONCLUSIONS An “interest” based fusion method for combining the thermal and vision images into a single image is described. Fusion is performed by evaluating fuzzy functions of differences in intensity among adjoining pixels, as well as absolute pixel intensity. The algorithms described here were demonstrated to be efficacious for fusion of both raw imagery, and extracted shapes and edges. Future work will include examination of additional images, as well as additional fusion approaches. 5.0 ACKNOWLEDGEMENTS The unprocessed vision and infrared images used in this paper were provided by the TNO Physics and Electronics Lab, and were produced by Defence Research Establishment Valcartier. The help of both organizations and permission to include the images is gratefully acknowledged. REFERENCES  M. E. Ulug and C. L. McCullough, "Fusion of Thermal and Vision Images," presented at the 1998 Conference on Artificial Neural Networks in Engineering (ANNIE98), St. Louis, MO, November 1998. Also appeared in Intelligent Engineering Systems Through Artificial Neural Networks, Vol. 8, ASME Press, New York, 1998.  M. E. Ulug and C. L. McCullough, “Feature and Data Level Fusion of Infrared and Visual Images,” presented at the SPIE Aerosense Conference, Orlando, FL, April 1999.