Decomposed Contour Prior for Shape Recognition
Zhi Yang, Yu Kong and Yun Fu
Department of CSE, SUNY at Buffalo, NY 14260, USA
{zhiyang, yukong7, yunfu}@buffalo.edu
Abstract
GolfSwing
Embedding Force
In this paper, we address the problem of representing
objects using contours for the purpose of recognition.
We propose a novel segmentation method for the integration of a novel contour matching energy into level
set based segmentation schemes. The contour matching
energy is represented by major components of Elliptic
Fourier shape descriptors and serves as a shape prior
to guide the curve evolution. The contours in training
dataset serve as templates and are utilized to match an
unknown image to infer its category. Our method is
evaluated on the UCF sports dataset and Caltech 101
dataset. Experiments show that our method achieves
promising recognition accuracy and is robust to noisy
low-level features and background clutter.
1. Introduction
Image segmentation and shape recognition in computer vision are usually driven both by low-level cues
such as intensity and texture, and by prior knowledge
about objects. Currently, modeling the interaction between data-driven and model-based processes has become the focus in image segmentation. In this paper, we
consider prior knowledge given by the shape of human
or object and focus on how to exploit such knowledge
for shape recognition.
Level set based contour representations have become
a popular framework for image segmentation [14, 5].
They allow us to elegantly model topological changes
of the implicitly represented boundary. Level set segmentation schemes can be formulated to utilize various
low-level image features such as intensity, color, texture, etc. However, as shown in [5], low-level features
can be misleading due to noise or background clutter.
To overcome this problem, some studies have investigated the integration of shape prior knowledge into
level set method. Leventon et al.[10] utilized shape
and pose information derived by principle component
Original
Picture
Region
Forces
Decomposed
Curvature
SKS
Converged
Contour
SwingBench
SKS
Contour
Converging
Skate
Boarding
SKS
Figure 1. Flowchart of our method.
analysis(PCA) on a given contour. Etyngier et al.[6]
proposed a method to learn shape using a non-linear
method. Unlike [10], the algorithm in [6] extracts shape
prior using manifold learning, and in particular, a diffusion map is constructed using Nyström extension. Cremers [3] proposed another non-linear shape prior using
autoregressive (AR) model to approximate linear combination. Jiang et al.[7] presented a TPS-RPM based
method to measure similarity between two points sets.
In this paper, we propose novel shape constraints to
guide contour evolution. The flowchart of our method
is shown in Fig.1. An object or a human is represented
by a set of Fourier coefficients. We introduce a new
curve energy function based on curvature to measure the
shape similarity between the target curve and an evolving curve. The curve energy is smoothly integrated with
region energy to guide segmentation. This makes the
segmentation process robust to noisy low-level features.
We use the obtained contours as templates for matching
and recognize human actions or objects from videos or
images.
2
2.1
Our Method
Contour Representation
The shape of an object can be effectively represented
by Fourier series on its contour [9]. In our work, we
to be segmented. We define the shape similarity of two
objects by the curvatures on corresponding segments.
Let C(s) be a curve parameterized by s, and Cψ≡0 (s)
be the target curve, and Cevolve (s) be the evolving curve.
Then the curvature of the target curve κψ and the evolving curve κC can be given by
κψ = O ·
OCevolve (s)
OCψ≡0 (s)
, κC = O ·
kOCψ≡0 (s)k
kOCevolve (s)k
κ
Figure 2. [Best viewed in color.] Contour
evolution with respect to target contours.
adopt the Elliptic Fourier descriptor (EFD) parameterized by s to represent object contour:
∞
2πns
2πn
2πns
∂x X 2nπ
=
an sin(
)+
bn cos(
)
−
∂s
T
T
T
T
n=1
∞
X
∂y
2nπ
2πns
2πn
2πns
−
=
cn sin(
)+
dn cos(
)
∂s n=1
T
T
T
T
where x and y are 2D coordinates on an image.
A contour in our work is described by the first N order Fourier coefficients Fig.2. These coefficients characterize the shape of an object and are utilized as the
shape prior to guide curve evolution. This type of shape
prior plays an important role in curve evolution and thus
facilitates the recognition task.
Motivated by [11], we use convex keypoints for initialization. The keypoints are defined as multi-scale organization of curvature zero crossing points of a planar
curve
∂2y
∂2x
=
0,
= 0,
∂s2
∂s2
The condition shows that a convex keypoint is an inflection point if the curve is continuous. With alignment of
such a set of convex points, our level-set based method
can start.
2.2
Contour Evolution
Our method is based on a region-based level set
scheme introduced by Chan and Vese [14]. Their
method generates a segmentation of an input image with
two gray value constants. However, their method relies
on low-level features and may fail to converge to the desired segmentation due to unfavorable background clutter. To cope with such degraded low-level information,
we propose a novel curvature matching energy function
Ecurve to guide contour evolution.
Our curvature matching energy function measures
shape similarity between the target object and the object
We use the ratio of curvature κ(s) = κCψ to denote the
shape similarity of a local segment s. Then the shape
matching energy is defined as
I
Ecurve = µ
|OH(ψ(C(s))κ(s)|ds,
(1)
ϕ(s)≡0
where H(z) is the Heaviside function: H(z) = 1 if
z > 0, otherwise H(z) = 0. ψ(C(s)) is used to find
corresponding segments between the target curve and a
evolving curve.
Shape energy Ecurve encodes the shape information
of an object and can provide the shape prior for segmentation. In this work, we incorporate this shape energy into the region energy to guide curve evolution.
Therefore, the total energy is expressed as
I
E = Ecurve + Eregion = µ
|OH(ψ(C(s))κ(s)|ds
ϕ(s)≡0
Z
+λ1
|µ0 (x, y) − c1 |2 H(ψ(x, y))dxdy
Ω1
Z
+λ2
|µ0 (x, y) − c2 |2 (1 − H(ψ(x, y)))dxdy
Ω2
(2)
where Eregion is the inhomogeneity energy [14], µ, λ1
and λ2 are parameters, Ω1 denotes the region enclosed
within Cϕ(s)≡0 (s), Ω2 is the region outside Ω1 .
2.3
Discussion
The advantage of the energy function in Eq.(2) is that
it can remove background noise caused by local shape
irregularities (see Fig.3).
Let Ê be an estimator. Suppose evolving contour
X is estimated by an observed contour vector Y and
Ê[X 2 ] < +∞ . We define X, Y1 ,..., Yn as random
process with finite second moments; all on the same
probability space. A guiding curve Y can be decomposed into harmonics of Fourier Series. Here we assume Ê[Yi ] = 0 for all i >= 1 and Ê[Yi Yj ] = 0 for
i 6= j, then
Êcurve [X|Y1 , Y2 , ..., Yn ] = Ê[X] +
n
X
Ê[X − E[X]|Yi ]
i=1
(3)
Our Method with Contour Curvature Decomposition
Figure 3. [Best viewed in color.] Contour
evolution to the target contours.
D−S
0.50
0.00
0.10
0.10
0.00
0.10
0.00
0.10
0.10
0.00
0.00
0.00
0.00
G−S−B
0.00
0.70
0.00
0.00
0.20
0.00
0.00
0.00
0.00
0.00
0.00
0.10
0.00
G−S−F
0.10
0.10
0.40
0.10
0.00
0.00
0.00
0.10
0.00
0.00
0.00
0.10
0.10
G−S−S
0.00
0.00
0.00
0.80
0.00
0.10
0.10
0.00
0.00
0.00
0.00
0.00
0.00
K−F
0.00
0.10
0.00
0.00
0.60
0.10
0.00
0.00
0.00
0.20
0.00
0.00
0.00
K−S
0.20
0.00
0.00
0.00
0.10
0.70
0.00
0.00
0.00
0.00
0.00
0.00
0.00
L
0.10
0.10
0.10
0.00
0.00
0.20
0.50
0.00
0.00
0.00
0.00
0.00
0.00
R−H
0.10
0.00
0.00
0.10
0.00
0.10
0.00
0.60
0.00
0.10
0.00
0.00
0.00
R−S
0.00
0.00
0.00
0.00
0.00
0.20
0.00
0.00
0.70
0.10
0.00
0.00
0.00
S−B−F
0.00
0.10
0.00
0.00
0.00
0.00
0.10
0.00
0.10
0.70
0.00
0.00
0.00
S−B
0.20
0.00
0.00
0.00
0.00
0.00
0.10
0.00
0.10
0.00
0.50
0.10
0.00
S−SA
0.00
0.10
0.00
0.10
0.10
0.10
0.00
0.00
0.10
0.00
0.00
0.50
0.00
W−F
0.00
0.00
0.10
0.00
0.00
0.00
0.00
0.00
0.00
0.20
0.00
0.00
0.70
K−F
K−S
L
R−H
R−S
S−B−F
S−B
S−SA
W−F
Evolving error is given by:
D−S
êcurve = X − Êcurve [X|Y1 , Y2 , ..., Yn ]
(4)
Recall that local shape irregularities are often represented by short period components in frequency domain. According to Eq.(2) and Eq.(4), these irregularities can be reduced during energy optimization.
2.4
Shape Matching
Many effective methods have been proposed to identify shape, such as Signed Distance Map (SDM) [10],
Curvature Scale Space (CSS) [8], etc. In our work, we
use the SKS algorithm [8] to compare two shapes. SKS
algorithm is a robust 2-d shape recognition algorithm
which uses the evidence accumulation.
Let feature vector v = (κ1 , κ2 , ..., κn ) be a vector of
scaled curvatures with respect to a reference point. The
similarity of two shapes can be computed by
sk = e
||v−vk ||
δ2
.
(5)
We use sk as the shape similarity and find the best
match for an unknown image from training dataset.
Figure 5. Confusion matrix of our method.
The confusion matrix of our method is displayed
in Figure 5. Our method achieves 60.77% recognition accuracy. Comparison results with other methods are shown in Table 1. Results indicate that our
method achieves the best overall accuracy over other
methods in comparison. The underlying reason is that
our method reduces local irregularity which essentially
is high-frequency noise. It decomposes the contour curvature into components and select the top N components. Thus high-frequency noise can be reduced.
To further investigate the noise reduction in our
method, we compare our method with Zhu and Chan’s
method [2]. We show intermediate results (the number of iterations is set to 100) in Figure 4. Our method
can efficiently remove background noise due to heterogeneity of local regions. The proposed method ignores high-frequency components which are sensitive
to background noise and thus achieves robust segmentation results. Therefore, our method achieves better
recognition accuracy.
3.2
3
Results on Caltech 101 Dataset
Experiment
We evaluate the proposed method, namely decomposed contour curvature prior (DCCP) with embedded
contour curvature prior (ECCP) on UCF sports dataset
and Caltech 101 dataset. We conduct two experiments
to evaluate our methods. First, we compare level set
based methods with DCCP and ECCP on UCF databset. Then segmentation results are compared on Caltech
101 object dataset.
3.1
G−S−B G−S−F G−S−S
Results on UCF-Sports Dataset
On this dataset, we compare the recognition accuracy of our method with 6 benchmark level set based
methods [12, 15, 14, 1, 13, 16]. Since these methods do
not have shape prior, to conduct a fair comparison, we
embed contour curvature prior ECCP into these methods using the embedding strategy described in [4].
We compare segmentation results with the method in
[2]. Results are shown in Figure 6. It can be seen that
our method is not sensitive to local heterogeneity of local regions. Our method selects components which are
insensitive to background noise while Zhu and Chan’s
method considers all high-frequency components which
would suffer from local irregularities.
4
Conclusion
We have proposed a decomposed contour curvature
prior for level set method for segmentation and shape
recognition. Our method ignores high-frequency components and can reduce background noise such as heterogeneity of local regions and irregularities. We use
segmentation results to conduct shape recognition task.
Results on two datasets show that our method is insensitive to background noise and achieves promising results
in recognition.
Table 1. Recognition results (%) of our method and methods in [12, 15, 14, 1, 13, 16].
Methods
Bernard’s [12] Casselles’s [15] Chan’s [14] Lankton’s [13] Li’s [1] Shi’s [16]
Ours
Accuracy
44.61
46.92
50.77
46.92
53.85
50.77
60.77
Figure 4. Contour evolution of our method (top row) and the method in [2] (bottom row).
Figure 6. Segmentation results of our
method (top row) and the method in [2]
(bottom row) on Caltech 101 dataset.
References
[1] J. G. Z. D. C. Li, C.Y. Kao. Minimization of region-scalable
fitting energy for image segmentation. IEEE Transactions on
Image Processing, 17(10):1940–1949, 2008.
[2] T. Chan and W. Zhu. Level set based shape prior segmentation.
In CVPR 05, pages 1164–1170, Washington, DC, USA, 2005.
IEEE Computer Society.
[3] D. Cremers. Nonlinear dynamical shape priors for level set
segmentation. In Computer Vision and Pattern Recognition,
2007. IEEE Conference on, pages 1–7, 2007.
[4] D. Cremers, T. Kohlberger, and C. Schnörr. Nonlinear shape
statistics in mumford-shah based segmentation. In Proceedings
of the 7th European Conference on Computer Vision-Part II,
ECCV ’02, pages 93–108, 2002. Springer-Verlag.
[5] D. Cremers, M. Rousson, and R. Deriche. A review of statistical approaches to level set segmentation: Integrating color,
texture, motion and shape. International Journal of Computer
Vision, 72(2):195–215, 2007.
[6] P. Etyngier, F. Sgonne, and R. Keriven. Active-contour-based
image segmentation using machine learning techniques. In
MICCAI (1)’07, pages 891–899, 2007.
[7] T. Jiang, F. Jurie, and C. Schmid. Learning shape prior models for object matching. In Conference on Computer Vision &
Pattern Recognition, jun 2009.
[8] K. Krish, S. B. Heinrich, W. E. Snyder, H. Cakir, and S. Khorram. Global registration of overlapping images using accumulative image features. Pattern Recognition Letters, 31(2):112–
118, 2010.
[9] F. P. Kuhl and C. R. Giardina. Elliptic fourier features of a
closed contour. Computer Graphics and Image Processing,
0
18(3):236–258, 1982.
[10] O. F. M. E. Leventon, W. E. L. Grimson. Statistical shape
influence in geodesic active contours. volume 1, pages 316–
323, 2000.
[11] M. S. D. M.A. Charmi, M.A. Mezghich and F. Ghorbel.
Geometric shape prior to region-based active contours using
fourier-based shape alignment. In 2010 IEEE International
Conference on: Imaging Systems and Techniques (IST), pages
478 – 481, 2010.
[12] P. T. M. U. O. Bernard, D. Friboulet. Variational b-spline
level-set: A linear filtering approach for fast deformable
model evolution. In IEEE Transactions on Image Processing,
18(06):1179–1191, 2009.
[13] A. T. S. Lankton. Localizing region-based active contours.
IEEE Transactions on Image Processing, 17(11):2029–2039,
2008.
[14] T.F.Chan and L.A.Vese. Active contours without edges. IEEE
Transactions on Image Processing, 10(2):266–277, 2001.
[15] R. K. V. Caselles and G. Sapiro. Geodesic active contours.
International Journal of Computer Vision, 22:61–79, 1997.
[16] W. C. K. Yonggang Shi. A real-time algorithm for the approximation of level-set-based curve evolution. In IEEE Transactions on Image Processing, 22(05):645–656, 2008.