PROCEEDINGS OF SPIE
SPIEDigitalLibrary.org/conference-proceedings-of-spie
A novel method for camera
calibration and image alignment of a
thermal/visible image fusion system
P. Bamrungthai, P. Wongkamchang
P. Bamrungthai, P. Wongkamchang, "A novel method for camera calibration
and image alignment of a thermal/visible image fusion system," Proc. SPIE
11331, Fourth International Conference on Photonics Solutions (ICPS2019),
113310H (11 March 2020); doi: 10.1117/12.2553042
Event: Fourth International Conference on Photonic Solutions (ICPS 2019),
2019, Chiang Mai, Thailand
Downloaded From: https://www.spiedigitallibrary.org/conference-proceedings-of-spie on 23 Mar 2023 Terms of Use: https://www.spiedigitallibrary.org/terms-of-use
A Novel Method for Camera Calibration and Image Alignment
of a Thermal/Visible Image Fusion System
P. Bamrungthai*a, and P. Wongkamchangb
a
Department of Mechanical Engineering, Faculty of Engineering at Sriracha,
Kasetsart University, Chonburi, Thailand
b
Department of Mechanical Engineering, Navaminda Kasatriyadhiraj Royal
Air Force Academy, Bangkok, Thailand
ABSTRACT
This paper presents a novel method for camera calibration and image alignment of an image fusion system that
consists of a thermal and a color camera. The calibration board that consists of a heated metal and an acrylic plate with
asymmetric circle grid pattern was developed. The board was used to calibrate intrinsic parameters of the thermal camera
and also used to find the homography to align images of the two cameras. The aligned images have been fine adjusted
for their misalignment due to difference in projection centers by region of interest (ROI) setting. The visible camera was
calibrated separately by using a typical calibration board with an asymmetric circle grid printed paper. The evaluation
was performed in two scenarios. The first one is an indoor static scene. And the second one is an outdoor dynamic scene
that is a vehicle tracking application by using CAMShift algorithm. The results validate the proposed method with high
accuracy.
Keywords: Thermal camera calibration, calibration object, image alignment, thermal/visible image fusion
1. INTRODUCTION
There is an increasing demand for using image fusion because of its ability to provide more information of a scene
from multiple sources of acquisition. The resulting image can be applied for many applications such as situation
awareness [1], surveillance [2], and object detection and tracking [3-5]. The infrared thermal and visible (color) camera
system is one of the most common platform because each of them has its individual advantages. The thermal camera is
used when we need thermal radiation information of target objects. When its image was combined with image obtained
from visible camera, the situation can easily be understood in the context of color image environment.
There are many techniques to fuse thermal and visible image. A recent review can be found in [6]. The pixel-level
fusion is used when spatial information is needed. Before starting the fusion process, image alignment is necessary to
spatially align the input images into the same geometric base [7]. Calibration-based alignment is better than using
similarity matching between the features in thermal and visible images because it can provide high accuracy. So, many
research works focused on calibration technique development for image fusion system. Calibration methods based on
passive calibration object were developed such as in [8-10]. These calibration objects do not use external energy sources
for being detectable by the sensors during the calibration process. Active calibration object was also developed by using
small light bulbs such as in [11] to enhance image contrast and get robust calibration results.
This paper is the progress of our work on a thermal/visible image fusion system [1]. It was the system development
for situation awareness application. However, there are two limitations of the system. Firstly, the fused image cannot be
perfectly aligned at larger distances from the image center due to lens distortion of both cameras. Secondly, the image
alignment process had to be done manually by the user to find homography between the two cameras. Therefore, we
developed the novel calibration object for the thermal camera calibration and automatic alignment process. The details
will be described in the following sections.
Fourth International Conference on Photonics Solutions (ICPS2019), edited by Tetsuya Kawanishi,
Surachet Kanprachar, Waranont Anukool, Ukrit Mankong, Proc. of SPIE Vol. 11331, 113310H
© 2020 SPIE · CCC code: 0277-786X/20/$21 · doi: 10.1117/12.2553042
Proc. of SPIE Vol. 11331 113310H-1
Downloaded From: https://www.spiedigitallibrary.org/conference-proceedings-of-spie on 23 Mar 2023
Terms of Use: https://www.spiedigitallibrary.org/terms-of-use
2. CAMERA CALIBRATION
2.1 The camera model and camera calibration
In essence, a camera is a mapping between the 3-D world and a 2-D image. A camera model is a mathematical
model of image formation process of the camera. The simplest one is the pinhole camera model as shown in Fig. 1 that a
light ray from object at point P projects onto the image plane at point pu . But due to lens distortion, the point was
distorted, so that the real image point will be projected at pd . These relationships can be shown in equation (1) to (4)
where x is the image plane coordinates  xu , yu ,1 , X is the object world coordinates  X w , Yw , Z w  , K is the intrinsic
parameters of the camera with some additional parameters for CCD camera system, R is the 3x3 rotation matrix
representing the orientation of the camera coordinate frame relatives to the world coordinate frame, t is the 3x1
translation matrix from the camera center to the world coordinate frame, and 1 ,  2 , and  3 are the coefficients for
radial correction. The details of camera model can be found in the literature [12].
O
X
oi
Y
x
pu  xu , yu 
pd  xd , yd 
y
Zw
z, Z
Ow
Yw
P  X w , Yw , Z w 
Xw
Figure 1. The camera model with lens distortion.
and
x  K  R t  X
(1)
xd  xu 1  1r 2   2 r 4   3r 6 
(2)
yd  yu 1  1r 2   2 r 4   3r 6 
(3)
where
r  xu 2  yu 2
(4)
Camera calibration is the process to determine the value of intrinsic parameters, elements in the matrix K and the
distortion coefficients 1 ,  2 , and  3 , and extrinsic parameters, the matrix R and t . There are many techniques to find
these parameters. One of the most popular method is Tsai’s camera calibration [13] that required a precise 3-D
calibration object. The other popular method is Zhang’s camera calibration method [14] that used a planar object with
chessboard pattern. The Zhang’s method was chosen in this paper because planar calibration object can be produced
easily and the method provides high accuracy. The OpenCV implementation of the Zhang’s method has been applied in
this research with asymmetric circle grid pattern of the calibration object.
2.2 The calibration board
The calibration object developed in this paper has been inspired from the work of Sun Xiaoming et al [10]. It was a
passive calibration board that used to calibrate a binocular vision system of the thermal cameras by using heat source
from the human operator. But as we first implement the calibration object with circular hole pattern, we found that by
Proc. of SPIE Vol. 11331 113310H-2
Downloaded From: https://www.spiedigitallibrary.org/conference-proceedings-of-spie on 23 Mar 2023
Terms of Use: https://www.spiedigitallibrary.org/terms-of-use
using human body as the heat source was not practical when we need to simultaneously detect it with the color camera.
So, we attached the aluminum plate to the back of the acrylic board with asymmetric circle pattern as shown in Fig. 2.
The usage of the board is first preheat the aluminum plate to make the thermal camera can see the different temperature
through the hole pattern of the acrylic plate. The visible camera was calibrated separately by using a typical calibration
board with an asymmetric circle grid printed paper.
(a)
(b)
(c)
Figure 2. The thermal camera calibration board components (a) the acrylic board with asymmetric circle pattern
(b) the aluminum plate and (c) the integrated calibration board.
3. IMAGE ALIGNMENT AND FUSION
3.1 The automatic image alignment
The thermal and the visible camera have to be aligned before the image fusion can be started. The relationship
between them can be found from the homography matrix as shown in Fig. 3. The homography H is the 3x3 matrix that
relates image coordinates of the two cameras system with a planar coordinates as shown in equation (5) or xv  Hxt ,
where xv is the image coordinates of the visible camera and xt is the image coordinates of the thermal camera.
xπ
π
xv
xt
H
Ct
Cv
Figure 3. The homography between the two cameras.
Figure 4. The automatic homography estimation process by using the thermal camerea calibration board.
Proc. of SPIE Vol. 11331 113310H-3
Downloaded From: https://www.spiedigitallibrary.org/conference-proceedings-of-spie on 23 Mar 2023
Terms of Use: https://www.spiedigitallibrary.org/terms-of-use
 xv   h11 h12
 y   h
 v   21 h22
 1   h31 h32
h13   xt 
h23   yt 
h33   1 
(5)
The automatic alignment process was shown in Fig. 4. It showed the detected reference points of the calibration
board that were used for homography estimation. Region of interest (ROI) of the two images is used to fine tune the
alignment due to difference in projection centers of the two cameras. The user has to select points in each image to
define the left, right, top, and bottom of the ROIs. The ROIs are then applied to the result images of the homography to
create perfectly aligned pixel positions.
3.2 The image fusion
After the image alignment has been performed, the image fusion can be started by first thresholding the thermal
image at a specified intensity level T. Then, the pixel values in the color image will be replaced by a predefined color if
the corresponding pixels in the thermal image exceed the threshold value to create the fused image. This scheme is very
simple but operates with high efficiency and low computational requirement that is suitable for real-time applications.
4. EXPERIMENTAL RESULTS
4.1 The camera system
The camera system consists of two cameras that are a thermal and a visible (RGB color) camera. They are mounted
in the same platform in parallel optical axes configuration as shown in Fig. 5. If the camera system has been calibrated
and aligned by using the method described in the previous two sections, the system is ready to operate. The calibration
and the alignment process will be necessary in case of the relative camera position and orientation was changed or the
lens were adjusted. A laptop computer is used as a processing unit and it connects to the two cameras via USB ports. The
computer contains the software that implements the camera calibration, image alignment, and image fusion algorithm.
Thermal
Camera
Visible
Camera
Figure 5. The thermal and the visible camera were setup in the same platform.
4.2 Indoor static scene
The first test of the system is an indoor static scene. The objective is to validate the camera calibration and the image
alignment process. The scene is the robotics laboratory at the 8th floor of the 23 Building at Kasetsart University,
Sriracha campus. Because of hot weather in Thailand, the result in Fig. 6 shows many hot regions of concrete structure
when the test was performed in the afternoon. However, it also shows the perfectly aligned between the thermal and the
visible images with some limitation on the black border region of the transformed thermal image. The threshold value of
the thermal image in this test is about 135.
Proc. of SPIE Vol. 11331 113310H-4
Downloaded From: https://www.spiedigitallibrary.org/conference-proceedings-of-spie on 23 Mar 2023
Terms of Use: https://www.spiedigitallibrary.org/terms-of-use
(a)
(b)
(c)
Figure 6. Images of the indoor testing (a) aligned thermal image (b) color image and (c) fused image.
4.3 Outdoor dynamic scene
The second test of the system is an outdoor dynamic scene. It shows potential of the system to develop as an object
tracking system. The target was selected by the user and the CAMShift algorithm [15] was implemented in the system to
track the selected target. The images shown in Fig. 7 are extracted from the sequence of image stream of the test. In the
test scene, the target car was tracked when it comes to the parking lot to u-turn. The CAMShift algorithm was efficient
enough for tracking the target car even there are two poles in the middle of screen that occluded some parts of the car
during the test. The results show successful tracking of the target car from entering the parking lot to leaving it on the
same road.
5. CONCLUSIONS
The two contributions of this paper are the thermal camera calibration object that was used to find the intrinsic
parameters of the thermal camera and the automatic homography estimation by using the developed object. The
experimental results show that the images can be fused with high accuracy both in static and dynamic scenes. The black
region in the transformed thermal images results from image transformation in the image alignment process. To improve
this, image cropping must be added to the processing pipeline to make the same visible region of the two cameras. There
are some limitations in dynamic scene testing that used the CAMShift algorithm. Robust tracking algorithm will be
developed in the future to apply the system for target object tracking.
Proc. of SPIE Vol. 11331 113310H-5
Downloaded From: https://www.spiedigitallibrary.org/conference-proceedings-of-spie on 23 Mar 2023
Terms of Use: https://www.spiedigitallibrary.org/terms-of-use
(a)
(b)
(c)
(d)
Figure 7. Images of the outdoor testing (a) aligned thermal images (b) color images and (c) fused images (d) tracking images.
Proc. of SPIE Vol. 11331 113310H-6
Downloaded From: https://www.spiedigitallibrary.org/conference-proceedings-of-spie on 23 Mar 2023
Terms of Use: https://www.spiedigitallibrary.org/terms-of-use
REFERENCES
[1] Bamrungthai, P. and Wongkamchang, P., "Development of a Thermal/Visible Image Fusion System for
Situation Awareness," In 2018 5th International Conference on Advanced Informatics: Concept Theory and
Applications (ICAICTA), IEEE, 96-100 (2018).
[2] Zin, T. T., Takahashi, H., Toriu, T. and Hama, H., "Fusion of Infrared and Visible Images for Robust Person
Detection," Image Fusion, 239-264 (2011).
[3] Spremolla, I. R., Antunes, M., Aouada, D. and Ottersten, B. E., "RGB-D and thermal sensor fusion-application
in person tracking," Proc. Int. Joint Conf. Comput. Vis. Imag. Comput. Graphics Theory Appl., 3, 612-619
(2016).
[4] Yu, Z., Yan, L., Han, N. and Liu, J., "Image Fusion Algorithm Based on Contourlet Transform and PCNN for
Detecting Obstacles in Forests," Cybern. Inf. Technol., 15, 116–125 (2015).
[5] Gaszczak, A., Breckon, T. P. and Han, J., "Real-time people and vehicle detection from UAV imagery," Proc.
SPIE vol. 7878 (2011).
[6] Ma, J., Ma, Y. and Li, C., "Infrared and visible image fusion methods and applications: A survey," Information
Fusion, 45, 153-178 (2019).
[7] Mitchell, H. B., [Image Fusion: Theories Techniques and Applications], Springer-Verlag, New York, (2010).
[8] Yang, M. D., Su, T. C. and Lin, H. Y. "Fusion of Infrared Thermal Image and Visible Image for 3D Thermal
Model Reconstruction Using Smartphone Sensors," Sensors, 18(7), 2003, (2018).
[9] St-Laurent, L., Mikhnevich, M., Bubel A. and Prévost, D., "Passive calibration board for alignment of VISNIR, SWIR and LWIR images," Quantitative InfraRed Thermography Journal, 14(2), 193-205 (2017).
[10] Xiaoming, S., Haibin, W., Wei, W., Liubo. and Guoguang, C., "Infrared camera calibration in the 3D
temperature field reconstruction," International Journal of Multimedia and Ubiquitous Engineering, 11(6), 2734 (2016).
[11] Zoetgnandé, Y. W. K., Fougères, A.-J., Cormier, G. and Dillenseger, J. L., "Robust low resolution thermal
stereo camera calibration," 11th International Conference on Machine Vision (ICMV 2018), proc. SPIE 11041,
11041-1D (2019).
[12] Hartley, R. and Zisserman, A., [Multiple View Geometry in Computer Vision], 2nd edition, Cambridge
University Press, Cambridge, UK, (2003).
[13] Tsai, R., "A Versatile Camera Calibration Technique for High Accuracy 3D Machine Vision Metrology using
Off-The-Shelf TV Cameras and Lenses," IEEE Journal of Robotics and Automation, 3(4), 323-344 (1987).
[14] Zhang, Z. Y., "A flexible new technique for camera calibration," IEEE transaction pattern analysis and machine
intelligence, 22(11), 1330-1334 (2000).
[15] Bradski, G. R., "Computer Vision Face Tracking for Use in a Perceptual User Interface," Intel Technology
Journal (1998).
Proc. of SPIE Vol. 11331 113310H-7
Downloaded From: https://www.spiedigitallibrary.org/conference-proceedings-of-spie on 23 Mar 2023
Terms of Use: https://www.spiedigitallibrary.org/terms-of-use