Proceedings of 2007 CACS International Automatic Control Conference
National Chung Hsing University, Taichung, Taiwan, Nov. 9-11, 2007
Dual Cameras for 360-Degree Visual Surveillance
Ming-Shyan Wang
Yu-Chun Huang
Tsung-Ching Yang
Department of Electrical Engineering,
Department of Electrical Engineering,
Southern Taiwan University
Tainan, Taiwan, ROC
e-mail: mswang@mail.stut.edu.tw
Southern Taiwan University
Tainan, Taiwan, ROC
m9520205@webmail.stut.edu.tw
Department of Electrical
Engineering
Southern Taiwan University
Tainan, Taiwan, ROC
Abstract—The work aims at providing a video surveillance
system, which includes an omnidirectional sensor to get 360
degrees of view angle of vision by analyzing only single
image for a wider working area, a general CCD to track
moving objects for clear view, and a permanent magnet
synchronous motor (PMSM) and its drive to rotate the
camera. The CCD mounted with the omnidirectional sensor
is first calibrated to obtain one-to-one correspondence between the image pixels through a catadioptric mirror and
the locations on image plane. Its image pre-processing based
on an algorithm of normalized grayscale correlation is employed to process images for motion detection. The position
data of the detected object are converted to be input command of the rotating mechanism. The general CCD then
executes the image tracking within the range of omnideirectional view. Instead of using normalized grayscale correlation, Sobel edge detection is adopted to determine the
geometric model of the captured target and compare it with
preset sample model. Finally, the experimental results validate the effectiveness of the proposed system.
Camera
Omnidirectional
CCD
Position moving
object
Image
difference
Capture image
Adaptive
binarization
Video recording
Median
filtering
Template
matching
Morphological
processing
Update position
data
YES
Keywords—Omnidirectional sensor, normalized grayscale
correlation, Sobel edge detection.
Moving obj. ?
NO
I. INTRODUCTION
NO
Target missing ?
Omnidirectional vision, or panoramically viewing
and imaging an environment is useful in a number of
applications, such as surveillance, elderly care, and robot
navigation and tracking. In order to make an imaging
system possible to cover a 360-degree field of vision,
there are two obvious solutions to obtain a panoramic
view of the scene. One is to rotate the entire imaging
system about its center of projection then combine the
sequence of acquired images [1-4]. This system just
needs a general CCD and may capture large and clear
view. However, the main disadvantages of it are that it
requires the use of moving parts and precise positioning
and more total time to obtain an image with enhanced
field of view. The other employs a so-called omnidirectional CCD and analyzes only one image [1-8]. But, an
omnidirectional CCD is more expensive that a PTZ
(Pan/Tilt/Zoom) like. Additionally, a poor image resolution of the omnidirectional CCD makes wide-angle imaging inadequate for monitoring objects around the predefined center [1-3].
In this paper, in order to have a clear image view of
moving objects in a large working area, we propose a
360-degree visual surveillance system with dual cameras
by using a low-resolution omnidirectional CCD and a
YES
End of recording
Fig. 1. Flowchart of image processing.
general CCD to provide both merits of the rotating imaging system and an omnidirectional CCD. As part of the
surveillance system, the omnidirectional CCD is not only
firstly responsible for real-time image acquisition for
follow-up visual tracking by the general CCD. It also
always monitors all covered working range so that the
general CCD can track, miss, and re-track a fast moving
target. The image pre-processing, including image unwarping, YUV transformation, median filtering, binarization, and image dilation and erosion, is employed to
detect an invasion first captured by the omnidirectional
CCD. Afterward the rotating mechanism and the general
CCD execute the image tracking. In stead of using
grayscale-based thresholding, geometric pattern matching is adopted to track moving objects by comparing the
determined geometric model with a preset model. The
flowchart of image processing is shown in Fig. 1, which
includes video recording for further consideration.
752
Proceedings of 2007 CACS International Automatic Control Conference
National Chung Hsing University, Taichung, Taiwan, Nov. 9-11, 2007
Y1
The paper is organized as follows for further discussion: Section I, introduction; Section II, moving object
detection; section III, video tracking; section IV, experimental results, and; conclusions.
X1
II. MOVING OBJECT DETECTION
Camera calibration of the omnidirectional vision system
includes two parts: one is to derive the equivalent focal
length and image center; the other is to find the
one-to-one correspondence between the ground positions
and the image pixels. The captured concentric omnidirectional image can be unwarped to create a panoramic
image provided that the equivalent focal length is available. The video is transferred to PC then converted by the
following equation [9],
Y2  G (Y1 )cos( X 1 )
X 2  G (Y1 )sin( X 1 )
(1)
where X 1 and Y1 are the pixel values of input image,
X 2 and Y2 are the pixel values of output image, and G is a
function of the radius of the catadioptric sensor.
Fig. 2 (a) shows the input image where green lines
stand for axes of omnidirectional image plane ( X 1 and Y1 )
and red segment is its radius. The unwarped image (or
panoramic image) is displayed in Fig. 2(b) where green
lines stand for axes of image plane ( X 2 and Y2 ) and red
segment is the distance of image. The range of X 2 is
from -180° to 180°. The generating procedure of unwarped image begins the transformation from the left
upper position and ends at the right lower one. However,
due to the errors of omnidirectional sensor and transformation, we employ equation (2) and choose a modified
value to calibrate the output image,
w
X 2  X 1  ( Y1  D)
2
h
Y2  ( X 1  D)  Y1
2
(2)
-180
X2
0
Y2
180
(b)
Fig. 2. Omnidrectional Image.
fˆ ( x, y )  mediang ( s, t )
( s ,t )S x y
(4)
where fˆ ( x, y ) is the output and g(s,t) is output of bubble
sorting for captured image. Thresholding is the transformation of an input image to its output binary image
defined as follows:
0
T( f )  
255
f  T0
f  T0
(5)
where T0 is the global-threshold value dependent on the
lightness of environment.
There generally exists incompleteness in an image
after binarization. The functions of dilation and erosion
may compensate it. Assuming that A and B are two sets
of Z 2 , A dilated by B denoted by A B is defined as [10]

 
A  B  z ( Bˆ ) z  A  A ,
(6)
Similarly, set of A eroded by B denoted by AB is defined as [10]
(7)
AB  z ( B) z  A
Opening an image is defined as
where D is the modified value and w and h are the respective sum of pixel on X 1 and Y1 axes. The rotating
direction decides the increment or decrement of D.
For the image pre-processing in the flowchart shown
in Fig. 1, we first convert the colors of the captured image
into grey levels in order to save about 2/3 data processing
time and speed up the image tracking. The grey scale Y of
YUV form will be obtained from RGB (red, green, and
blue) form of captured color image by the following
equation,
Y  0.299 R  0.587 G  0.114 B
(a)
(3)
The background is deleted and the moving block is
detected by subtracting two consecutive images pixel by
pixel. We get rid of the noise in the image via median
filtering by the following equation,
A  B  ( AB)  B
(8)
Closing an image is defined as
(9)
A  B  ( A  B)B ,
After we have finished closing or opening an image, we
know if there exists a moving object.
III. ADAPTIVE FUZZY LOGIC CONTROL
As soon as the position data of the captured object
have been identified, they will be transformed into the
input command of the rotating mechanism that includes a
permanent magnet synchronous motor (PMSM) and its
drive. The command is transmitted between the PC and
motor drive only via RC-232 cable without using any
interface circuit. The servo motor steers the general CCD
for more precisely visual tracking.
753
Proceedings of 2007 CACS International Automatic Control Conference
National Chung Hsing University, Taichung, Taiwan, Nov. 9-11, 2007
There are some algorithms on boundary mask, such as
Sobel, Prewitt, Roberts, and Laplacian gradients. However, Sobel mask is the most popular because it has the
merits of easy computation, better noise suppression, and
good performance. The gradient G of an image f ( x, y) at
position ( x, y) by Sobel operator is defined as [10]
G
f ( x, y )
f ( x, y )
i
j  Gx i  G y j (10)
x
y
where G x and G y denote the x and y components of G.
The magnitude of G is calculated and approximated as
G  Gx  G y  Gx  G y
2
2
(11)
and the direction of G is
G  tan 1
Gy
Gx
.
(12)
For a 3 3 mask shown in Table I, the gradient G of an
image by Sobel operator will be simplified as
Gx  ( x7  2 x8  x9 )  ( x1  2 x2  x3 )
G y  ( x3  2 x6  x9 )  ( x1  2 x4  x7 )
(13)
IV. EXPERIMENTAL RESULTS
The experimental setup shown in Fig. 4 includes one
low-resolution omnidirectional sensor from Vstone [11].
It is composed of a NTSC color camera and a hyperbolic
mirror with a field of view extended from 10 o to
55o below the horizon. The 1/3 in color CCD has the
image resolution of 768*508 pixels. In this work, it is
mounted on the ceiling of the laboratory to provide global
surveillance of the environment. The specifications of
PMSM are listed in Table II. In addition, the characteristics of the interlaced CCD are shown in Table III.
An output image of the omnidirectional CCD is
shown in Fig. 5. We unwrap it and display it in Fig. 6. Fig.
7 presents one moving object. The grey values of each
pixel in Figs. 6 and 7 are shown in Figs. 8 and 9 as two
consecutive images. Fig. 10 displays their difference
pixel by pixel. Median filtering is performed to get rid of
the noise in Fig. 10, shown in Fig. 11. Thresholding is
operated to transform the image in Fig. 11 to its output
binary image by using equation (5) for T0 =10, shown in
Fig. 12. Figs. 13 and 14 present the dilation and erosion
operations on the binary image, respectively. Fig. 15 is
the output of closing the binary image in Fig. 12. The
moving object has been marked by a red rectangle in Fig.
16.
(14)
Based on a preset threshold, the score of G will determine
if the image pixel is an edge point. Fig. 3 shows a picture
and its output by Sobel operator. If the preset geometric
edges locate on the central half of the image, the camera
will stand still. Additionally, if the edges locate the
one-fourth left (right) of the image, the camera will be
shifted left (right) with a constant speed by the PMSM.
(a)
Fig. 4. Experimental setup.
(b)
Fig. 3 A picture (a) and its output (b) by Sobel operator.
754
Proceedings of 2007 CACS International Automatic Control Conference
National Chung Hsing University, Taichung, Taiwan, Nov. 9-11, 2007
Fig. 13. Dilation.
Fig. 5. Captured image.
Fig. 14. Erosion.
Fig. 6. Unwarped image.
Fig. 15. Image closing .
Fig.7. An image with invasion.
Fig. 16. Moving object located.
Fig. 8. Image histogram.
Fig. 9. Moving object histogram.
PC estimates the center position of marked object in
Fig. 16, determines the rotating angle of the rotating
mechanism, and transmits the command data via RS-232
to the PMSM drive. The captured object enters the laboratory and walks from left to right then returns back.
The general CCD initially positions at the center, shown
in Fig. 17(a), and then shifts left after the omnidirectional
CCD captures the moving object and PC sends the rotating command to it, shown in Fig. 17(b). Figs. 17 (b)-(i)
present a sequence of tracking images when the captured
object moves back and forth. The rectangle in Fig. 17
stands for the preset geometric edges, and the cross
symbol is its center. The threshold for the score by Sobel
operator is 25%. However, the influence of the object size
on this system must be considered.
Fig. 10. Image difference.
V. CONCLUSIONS
We have proposed a video surveillance system that
can cope with different applications with 360-degree and
clear visual field by using an omnidirectional CCD and a
general CCD. Our main focus has been the omnidirectional image detection and tracking for invading objects.
Our system takes a grayscale-based approach where an
adaptive thresholding is employed during image
pre-processing, which includes YUV transformation,
median filtering, binarization, and image dilation and
erosion. The post-processing for the general CCD is
made within a pattern-matching approach to track the
captured moving objects by the omnidirectional CCD.
The communication between PC and the PMSM drive of
the rotating mechanism only needs a RS-232 cable. Finally, based on the experimental results, we have validated the effectiveness of the proposed system.
Fig. 11. Median filtering.
Fig. 12. Binarization.
755
Proceedings of 2007 CACS International Automatic Control Conference
National Chung Hsing University, Taichung, Taiwan, Nov. 9-11, 2007
(g)
(a)
(h)
(b)
(i)
Fig 17. A sequence of tracking images when the captured object moves
back and forth.
(c)
x1
3 3 mask.
x2
x3
x4
x5
x6
x7
x8
x9
Table I.
Output
power
Torque
(d)
Torque
constant
Rated
velocity
Table II. Parameters of PMSM 7CB30.
PR 300W
Back EMF
KE 54.9V/Krpm
constant
TR 0.95Nm
Inertia
JM 0.224Kgcm2
KT 0.524Nm/A
NR 3000RPM
Stator resistance
Stator inductance
Rs 2.79 Ω
Ls 5.80 mH
Table III Specifications of interlaced CCD
Scanning system
525 lines 30 frames/sec. 2:1
interlaced
CCD sensor
Color 1/2” IT EXview HAD
CCDTM
Sensing area
6.6(h)x4.8(v) mm
Effective pixels
768(h)x494(v)
Shutter flickerless
1/100 sec.
(e)
ACKNOWLEDGMENT
The authors would like to express their appreciation to
Ministry of Education for financial supporting.
(f)
756
Proceedings of 2007 CACS International Automatic Control Conference
National Chung Hsing University, Taichung, Taiwan, Nov. 9-11, 2007
REFERENCES
Y. Yagi, S. Kawato, and S. Tsuji, “Real-time omnidirectional
image sensor (COPIS) for vision-guided navigation,” IEEE
Trans. on Robotics and Automation, Vol. 10, No. 1, pp. 11-22,
1994.
[2] A. Bonarini, P. Aliverti, and M. Lucioni, “An omnidirectional
vision sensor for fast tracking for mobile robots,” IEEE Trans. on
Instrumentation and Measurement, Vol. 49, No. 3, pp. 509-512,
2000.
[3] S. K. Nayar, “Catadioptric Omnidirectional Camera,” Proceedings IEEE Conference on Computer Vision and Pattern Recognition, Puerto Rico, pp. 482-488, June 1997.
[4] H.Y. Lin, M.L. Wang, C.C. Huang, and B.W. Tsai, “Intelligent
surveillance using an omnidirectional CCD camera,” 2005 Automatic Control Conference, CACS2005, Tainan, Nov. 18-19.
[5] S. K. Nayar and S. Baker, “Catadioptric image formation,” Proc.
of DARPA Image Understanding Workshop, May 1997.
[6] Y. Yagi and S. Kawato, “Panoramic scene analysis with conic
projection,” Proc. of International Conference on Robots and
Systems (IROS), 1990.
[7] J. Hong, “Image based homing,” Proc. of IEEE International
Conference on Robotics and Automation, May 1991.
[8] K. Yamazawa, Y. Yagi, and M. Yachida, “Obstacle avoidance
with omnidirectional image sensor hyperomni vision.” Proc. of
IEEE International Conference on Robotics and Automation,
pp. 1062-1067, May 1995.
[9]
G. Scotti,; L. Marcenaro,; C. Coelho,; F. Selvaggi,; C.S. Re
gazzoni, , “Dual camera intelligent sensor for high definition
360 degrees surveillance,” IEE Conf. on Vision, Image and
Signal Processing, Vol. 152, No. 2, 8 April, pp. 250-257, 2005.
[10] Gonzalez and Woods, Digital Image Processing, 2nd Edition,
Prentice Hall, 2002.
[11] V. Corporation. http://www.vstone.co.jp.
[1]
757