Tennis Touching Point Detection based on High Speed
Camera and Kalman Filter
Weixin Wu
Dept. of Electrical and Computer Engineering
Clemson University
Abstract
This paper introduces the tracking of tennis ball and touching point detecting using high speed camera
and Kalman filter. Due to the motion blur of the target, position measurement is corrupted with noise.
The segmentation algorithm also fails when the tennis when the ball is passing the line and especially
when bouncing on the line. To address the problem, principal components analysis and Kalman filter are
used for the measurement and estimation respectively. The position of tennis ball and line is
reconstructed for on-line/outside detection. Experimental results demonstrate the effectiveness of the
proposed method.
1. Introduction
The goal of the project is to automatically track the tennis ball and measure if the bouncing point is on
the line or outside the line, which is also known as good shot or bad shot. To achieve the goal we need
to detect the line and the tennis ball’s dimension, position, bouncing point and reconstruct the relative
position of the line and the ball. Three steps are required. The first step is segmentation, which includes
line segmentation and tennis ball segmentation. The second step is tennis ball position measuring,
tracking, diameter calculating and bouncing point detection. The third step is tennis ball reconstruction
using above parameters and on-line/outside calculating. However, two main problems exist in the
process. The first problem is blurring along the motion direction. Movement of the tennis ball is
recorded using high speed camera with certain frames per second (fps). The camera we have provides
maximum of 1000 fps recording rate, however, due to the “exposure time” that CMOS needs to be
charged, the video recorded with 1000 fps is not bright enough for further image processing. To balance
the video brightness and minimum speed requirement to get accurate measurement, we selected 50 fps
as the suitable speed. As the shutter speed is not ideally fast, blurring exists in the moving direction[1].
To address the problem the blurring caused to tennis ball position and diameter measuring, principal
components analysis (PCA) is used to calculate the center point and diameter. However the
measurement of position in the motion direction is still corrupted with measurement noise. The second
problem is segmentation failure when tennis ball crossed the line or bouncing on the line. Due to the
grayscale image, the tennis ball and line tend to merge into each other. It brings difficulty for segment
and for direct measurement of position, which is shown in Figure 5 and Figure 6. To address the above
two main problems, Kalman filter[2] is used to track center point at each moment, especially at the
bouncing point. Experimental results demonstrate the effective tracking results based on PCA and
Kalman filter. The organization of the rest of the paper is as following: The analysis of the problem ,
Kalman filter and algorithm are introduced in section2. In section 3 some experimental results are
proposed and conclusion is given in section 4.
2. Method
Three steps of the project are introduced in this section. The first step is line segmentation using double
threshold and tennis ball segmentation using frame differentiating. The second step is tennis ball center
point measurement, diameter calculating based on PCA, and bouncing point tracking based on Kalman
filter. The last step is tennis ball reconstruction and on-line/outside measurement.
2.1.
Line and tennis ball segmentation, motion blurring analysis, center point
detection and diameter calculating based on PCA.
The line is segmented using double threshold in order to suppress noise and acquire sharp edge of
line for further processing. Due to the specific lighting condition and recording rate, the high and low
threshold value are chosen as 200 and 50 respectively.
Tennis ball is segmented based on frame differentiating, since the shutter speed is not ideally fast,
motion blur exists and the segmented target is not round but a jaggy blob. We assume the blurring only
happens in the motion direction but not the perpendicular motion direction, shown in Figure 1. We use
PCA to find the Eigenvector and Eigenvalue. The maximum Eigenvalue and corresponding Eigen vector
represent the motion direction; the minimum one represents the perpendicular direction. So the
maximum distance of the blob in the perpendicular motion direction is tennis ball’s diameter. The
diameter is measured every time, the maximum value is used as the true value of diameter.
Figure 1. Motion blur and PCA
Figure 2. Position reconstruction
The center point of the blob is taken as the center point of the tennis ball. Obviously this
measurement is corrupted with noise, known as measurement noise.
However, above method fails when the ball crosses the line and bouncing on the line because in
grayscale image, the intensity of tennis ball and line is almost saturated. It means they “merge” into
each other at these moments. The segmentation results are more than 100 pixels away from the true
position. To address this problem, Kalman filter is used to predict the center point.
2.2.
Kalman filter and bouncing point reconstruction and on-line/outside
detection
In this project, the tracking is based on only current frame and previous frame, which can be taken
as Markov process. The movement of tennis ball can be decomposed as free dropping movement in
vertical direction and movement with constant speed in horizontal direction. Even though the direction
between gravity and velocity in vertical direction changes instantly after the bouncing and friction may
apply to tennis ball, what we really interested is the movement before the bouncing, so the system can
still be regarded as linear before that moment. The noise is presumed as Gaussian-distributed, so
Kalman filter is a good option for this problem. The System transition functions are expressed as
following[3, 4]:
in which T is sampling time 1/50 sec, g is gravity constant 9.8 m2/s.
The reconstruction of tennis ball bounce uses center point and diameter acquired previously, which
is shown in Figure 2. The center touching point between tennis ball and ground is selected as the lowest
point of the ball, and touching area is select as an eclipse.
2.3.
Algorithm
The algorithm consists of 6 parts shown as flow chart in Figure 7.
3. Experimental results
Two sets of experimental results are presented as following. Figure 3 and 4 is for the situation that the
ball is bouncing on the line; Figure 5 and 6 is outside the line.
Figure 3. (From left to right) Segmented image before and after the bounce for on-line situation.
Figure 4. (From left to right) Kalman filter output of center point in vertical direction, horizontal direction,
reconstructed image and original image for on-line situation.
Figure 5. (From left to right) Segmented image before and after the bounce for outside situation.
Figure 6. (From left to right) Kalman filter output of center point in vertical direction, horizontal direction,
reconstructed image and original image for outside situation.
4. Conclusions
From Figure 3 we can find the segmentation when the ball is crossing the line is almost failure.
However, from Figure 4 and Figure 6 we can see the Kalman filter prediction weights more
prediction than measurement. And in last two pictures in Figure 4 and Figure 6 we can see the
prediction of Kalman filter is quite close to the real situation, which demonstrate the effectiveness
of the algorithm. However, this work is limited as the results may not tell how credible it is in
probability distribution. To improve this, particle filter can be used in future, so it can be used to test
the situation that the tennis ball has a little overlap on the line. What’s more, some other method
of deblurring algorithm can be used to directly measure the tennis ball position, which may produce
better result.
5. References
1. Image Processing: Analysis and Machine Vision, M Sonka, V Hlavac, R Boyle - Brooks/Cole,
Thomson Asia Pte Led.
2. An Introduction to the Kalman Filter, Greg Welch and Gary Bishop,
http://www.cs.unc.edu/~welch/media/pdf/kalman_intro.pdf.
3. Kalman Filtering, Dan Simon, http://www.innovatia.com/software/papers/kalman.htm.
4. Kalman Filter from Wikipedia, http://en.wikipedia.org/wiki/Kalman_filter.
5. J. Canny, A computational approach to edge detection, PAMI, 8(6):679-698, 1986.
Start
The last
frame?
Line segmentation
Ball segmentation
Evector and Evalue calculation
using PCA
Center point calculation and
diameter calculation
Kalman filtering
Bounce reconstruction and online/outside detection
End
Figure 7. Algorithm flow chart