corner, interest point, feature,
artificial pattern, homogeneous surface,
detector, matching,3D reconstruction
Paweł POPIELSKI1, Zygmunt WRÓBEL1
THE ASSASMENT OF THE CORNER DETECTION ALGORITHMS FOR
THE INTERST POINTS EXTRACTION FROM THE HOMOGENEOUS
SURFACES
In this article we deal with one of the fundamental problems in the area of the 3D reconstruction
for objects with homogeneous surface as, inter alia, human body or sculptures. The interest point
detection on typical photos of many differing elements and changing intensities is already well-solved
issue. Considerable difficulty and novelty is the interest point detection for homogeneous surfaces. To
reconstruct such surfaces from images we have to artificially produce as many elements on surface as it is
needed to allow proceed with the 3D coordinate’s extraction process with desired density. Four methods
were selected. First tested was definitely the best documented the Harris corner detector. Next was the
Nobel’s version of auto-correlation, then the minimum eigenvalue method known as the Kanade-Tomasi
algorithm and the last the fast radial feature detector known as the Loy-Zelinsky algorithm. Chosen
methods are well-known on the 3D reconstruction theatre and are more efficient in the terms of
computational complexity then others, well implemented and documented. Also some image
enhancements were utilized before feature extraction to improve detection process. As shown the best
choice is the Nobel’s version of auto-correlation function and very interesting candidate for further
research is the Loy-Zelinsky method.
1. INTRODUCTION
From at least two decades automatization of 3D objects reconstruction has been the
centre of attention of computer vision and close range digital photogrammetry. The interests
in 3D object reconstruction are motivated by a wide spectrum of applications, particularly as
visualization and measurement systems in medicine [1][11][12]. It is possible that in not
distant future 3D reconstruction system will be part of complex diagnostic systems. In
medical applications great attention is addressed for quality of the results, especially in the
terms of dimensions, volumes and curve descriptions [14]. To fulfil very rigorous
requirements and achieve high accuracy very dense network of interest points have to be
acquired [10].
At a certain level of abstraction typical 3D reconstruction procedure consists from three
main steps. Firstly very accurate principal point locations for all camera stations have to be
determined. At this point utilized may be widely used resection method based on the
collinearity equation [6]. Secondly conjugate points have to be detected on the sequence of
1
University of Silesia, Faculty of Computer Science and Materials Science, Institute of Computer Science,
Department of Biomedical Computer Systems, ul. Będzińska 39, 41-200 Sosnowiec,
e-mail: ppopielski@us.edu.pl, wrobel@us.edu.pl
pictures. Thirdly in accordance with collinearity equation and with help of intersecting
conjugate rays three-dimensional coordinates are established [16].
2. PROBLEM FORMULATION
Second step, conjugate points detecting, may be automate only with the feature point
extraction methods belong to the computer vision domain [18]. At this stage has to be
explained that "corner", "interest point" and "feature" are used interchangeably in literature
and in this paper. Actually a corner can be defined as the intersection of two edges. A corner
can also be defined as a point for which owns two dominant and different edge directions in a
local neighbourhood of the point.
On the other hand an interest point is a point in an image which has a well-defined
position and can be robustly detected. This means that an interest point can be a corner but it
can also be, for example, an isolated point of local intensity maximum or minimum, line
endings, or a point on a curve where the curvature is locally maximal.
The problem is that the feature detector to work needs some amount of distinct features
on processed object’s surface. It doesn’t work if there is nothing on the surface. It is common
problem in medical, architectural and fashion applications where must to be established the
curvature of the homogeneous surface to establish in next stage dimensions. To keep very
high quality requirements examined object must be in artificial way enriched in dense mesh of
distinct features what can be achieved through projection of artificially designed pattern
(Figure 1) consisting of enough small elements to provide required accuracy. Method is well
described in literature [2][3] and its efficiency also was proofed [15].
b)
a)
Figure 1. Random coded artificial pattern (a) and its projection towards reconstructed object (b).
3. FEATURE DETECTORS
The feature, corner or interest point detection constitutes a crucial phase of conjugate
points identifying process. Though the future detection is well known problem with a lot of
industry implementations still remains a focus of research. Another challenge is to test well
known methods under new conditions in new environments.
Artificial pattern projection involves new different conditions as the projectors disturb
a natural light scene. The noise not detected at first glance severely makes it hard to take
photos. Proper set up of an aperture size and a time of exposure need a lot of tests and an
experienced operator. Finally images are darker than usual with contrast disturbances.
Occurrence of new conditions provides a basis for further research on the well known
methods. Selected methods have documented history in industry applications and confirmed
high effectiveness [17][21] what determined the choice.
3.1. HARRIS DETECTOR
Definitely the most popular interest point detector based on the local auto-correlation
function of a signal [5], what is indeed the sum of squared differences (SSD) similarly to
Moravec algorithm. Moravec SSD given by equation (1) is calculating similarity between an
image patch over the area (u,v) and its shift by (x,y).
𝑆(𝑥, 𝑦) = ∑𝑢 ∑𝑣 𝑤(𝑢, 𝑣)(𝐼(𝑢 + 𝑥, 𝑣 + 𝑦) − 𝐼(𝑢, 𝑣))
2
(1)
In the equation the weighted sum of squared differences between two patches is denoted
as S(x,y). The lower number indicates more similarity in the image and local maximal value of
the number indicates that a feature of interest is present. The advantage of the Harris detector
over the Moravec method is to replace shifted patches directly with the differential of the
corner score with respect to direction what produces the approximation (2).
𝑆(𝑥, 𝑦) ≈ ∑𝑢 ∑𝑣 𝑤(𝑢, 𝑣)(𝐼𝑥 (𝑢, 𝑣)𝑥 − 𝐼𝑦 (𝑢, 𝑣)𝑦)
2
(2)
In the equation (2) the partial derivatives of I are denoted as Ix and Iy. This
improvement hugely reduced the calculation complexity of the algorithm. If we apply a
circular window or circularly weighted window (Gaussian function) the response will be
isotropic, what is another huge advantage over Moravec method.
𝑀𝐶 = 𝜆1 𝜆2 − (𝜆1 + 𝜆2 )2 = 𝑑𝑒𝑡(𝐴) −  𝑡𝑟𝑎𝑐𝑒 2 𝐴
(3)
Finally in (3) eigenvalues of the Harris matrix (A) are replaced with determinant(A) and
trace(A) what makes next computational improvement in terms of time complexity.
3.2. NOBEL’S AUTO-CORRELATION MODIFICATION
To overcome difficulty related to parameter , which have to be every time tuned Noble
[13] proposed a modified version of the Harris detector that does not contain any parameter
(4).
𝑑𝑒𝑡(𝐴)
𝑀𝐶 = 𝑡𝑟𝑎𝑐𝑒(𝐴)−𝜀
(4)
The constant 𝜀 is close to zero. This is to avoid a singular denominator in case of a rank
zero of the Harris matrix A.
3.3. KANADE-TOMASI METHOD
In [19] authors solved problem of feature tracking methods between consecutive frames.
Mathematical basis are very similar to previously discussed methods except Harris matrix A is
now known as autocorrelation matrix. Shi-Tomasi function is the smallest eigenvalue of the
autocorrelation matrix A.
𝑀𝐶 = 𝜆min (𝐴)
(5)
Method was again formulated and presented by Tomasi and Kanade in [20] and that is
why we interchangeably call this method “Kanade-Tomasi” and “Shi-Tomasi”.
3.4. FAST RADIAL ALGORITHM
Method uses image gradient to locate points of high radial symmetry [8]. An algorithm
is characterized by ease of implementation and computational effectiveness. Its computational
order is O(KN) when considering local radial symmetry in NxN neighbourhoods across an
image of K pixels. In this paper the transform was calculated for set of radii N={1,3,5} what
was considered as enough accurate. Radial strictness parameter α was set to value 1, what was
necessary to ensure maximum effectiveness of the algorithm at the expense of a certain
number of errors.
4. TEST METHOD
To examine interest point detectors for suitability for 3D reconstruction process from
images with projected artificial pattern was taken convergent pair of pictures from baseline 90
cm and at distance 2 meters from mannequin. Then images were cropped to show head and
shoulder to size 260 to 300 pixels to reduce computational time. Next images were converted
to 8 bit greyscale.
First were tested algorithms on original images (column 2), then images were enhanced
with help of MATLAB embedded functions. Normalization was performed as first
improvement (column 3) and then histograms were transformed to flat shape (column 4).
To investigate algorithms under real 3D reconstruction conditions extracted points were
processed further with normalized cross-correlation and (NCC) then all points were filtered to
assess the real usefulness.
The normalized cross-correlation (7) is well-known area-based matching method for
assessing the image similarity and that is why it was used to get putative matches. Small
windows composed of grey values are used as matching primitives to perform comparisons
over uniform zones of images until the best correspondence is reached. In our case the size of
template window was set up to 41 pixels. Normalized cross-correlation, given by equation (6)
significantly reduced disadvantages of regular cross-correlation. Lewis proved also
supremacy of NCC over the other matching methods in typical applications.
𝛾(𝑢, 𝑣) =
̅ ][𝑡(𝑥−𝑢,𝑦−𝑣)−𝑡̅]
∑𝑥,𝑦[𝑓(𝑥,𝑦)−𝑓𝑢,𝑣
2
1
(6)
̅ ] ∑𝑥,𝑦[𝑡(𝑥−𝑢,𝑦−𝑣)−𝑡̅]2 }2
{∑𝑥,𝑦[𝑓(𝑥,𝑦)−𝑓𝑢,𝑣
Outliers were eliminated during process of creating fundamental matrix (9) with help of
robust RANSAC estimation algorithm (4). Model was estimated with very rigorous value of
the distance threshold between a data point and the model t = 0.001 pixel. The RANSAC
algorithm belongs to non-deterministic models due to certain assumptions of statistical nature.
Therefore 20 models were determined and then median and standard deviation was calculated.
5. OBTAINED RESULTS
For tested images with rather poor contrast the Harris detector (Table 1) (Figure 2 a,b)
performed very stably. Autocorrelation method is well known from sensitivity to contrast
differences. The grater the contrast the more interest points detected. Probably bad quality of
images in terms of improper exposure time due to interferences in a scene lighting caused by
artificial light from projector, may explain very similar results for original images and then
normalized and flatted images. Differences in points number at a 1% level relieve us of the
need to enhance contrast what significantly improve processing time. Evan number of
putative matches and filtered matches are almost similar and oscillates around 1 % level.
Recommendation is that there is no need to improve images quality prior to interest point
detection with the Harris algorithm.
Table 1. Results for interest point extraction and matching for the Harris detector.
the Harris detector
original image increased contrast flatted histogram
left image
892
904
899
right image
830
841
817
putative matches
257
258
240
filtered matches
1646
1628
1659
In contrast to the previous method there is clear evidence (Table 2) (Figure 2 c,d) that
Noble detector was well responded for images enhancements. The gain for normalized images
is at a 9% level and for images with flatted histogram shape is at a 16%.
Table 2. Results for interest point extraction and matching for the Nobel’s version of autocorrelation.
Nobel’s version of autocorrelation
original image increased contrast flatted histogram
left image
764
832
886
right image
613
760
808
putative matches
226
245
241
filtered matches
15310
1648
1617
Unfortunately there is no difference for filtered matches, what again mean that there is
no need to lose computational time for image improving.
Next method based on the smallest eigenvalue of the auto-correlation matrix (Table 3)
(Figure 2 e, f) clearly shows significant gain of points number between original and a
histogram tuned images at a 15% level. There is also observed gain on filtered matches at a
25% level. This means that even for images with rather bad contrast the Kanade-Tomasi
algorithm still retains sensitive for contrast enhancements. This method is characterized by the
highest computational complexity.
Table 3. Results for interest point extraction and matching for the Kanade-Tomasi detector.
the Kanade-Tomasi algorithm
original image increased contrast flatted histogram
left image
1605
1609
1834
right image
1488
1483
1737
putative matches
300
319
334
filtered matches
1105
11310
1387
The Loy-Zelinsky algorithm (Table 4)(Figure 2 g,h) demonstrates a great ability to
respond to changes in contrast. The point’s gain respectively is at a 7% level and at a 34%.
The result for filtered matches seems be unclear probably disturbed by very strict RANSAC
distance threshold value. However sill can be observed the gain between original images and
the histogram tuned.
Table 4. Results for interest point extraction and matching for the Loy-Zelinsky detector.
the Loy-Zelinsky algorithm
original image increased contrast flatted histogram
left image
755
804
1011
right image
768
876
1208
putative matches
195
197
218
filtered matches
1206
1164
14010
6. CONCLUSIONS
Analysing all results there is huge jump in number of extracted points between the
Kanade-Tomasi algorithm and others. This method based on the smallest eigenvalue of the
auto-correlation matrix almost doubled number of detected points. Its higher computational
complexity as well as the cost in comparing to others tested methods can explain the result.
Next result among other is given by the Harris detector which combined with its
computational simplicity which translates into low cost is indisputable advantage.
The result of filtered matches clearly shows a domination of autocorrelation and the
Nobel’s version of autocorrelation with a predominance of Harris detector. What argues for a
Nobel’s version is a lack of tuneable parameter k which is present in the Harris algorithm. For
purpose of this paper there was used parameter k=0.04 where the values suggested in the
literature are in the range 0.04 - 0.15. There are no recommendations or standards as to the
value of parameter k and for each application the k must be individually tuned, what indeed
exclude this method for real time applications. The cure was given by Nobel in his doctoral
thesis and that is why Nobel’s autocorrelation should be preferred in the future application.
Noteworthy is the Loy-Zelinsky algorithm. Number of detected points on images after
histogram transformation is quite impressive. There can be observed almost doubled point
extraction performance in comparing to the Noble method. Number of filtered matches is
more than 10% less compared to the result for the Harris detector. Although have to be noted
that during process of outliers elimination to get filtered matches was used extremely strict
parameter t specifies the distance threshold between a data point and the model. It is quite
possible that for less rigorous t parameter much better performance may be archived.
In further work author of this paper based on the above evidence is going to extract
conjugate points for purpose of 3D reconstruction with the Nobel’s modification of
autocorrelation. Besides due to the low computational cost the Loy-Zelinsky algorithm is still
in the centre of research for author.
BIBLIOGRAPHY
[1]
[2]
[3]
[4]
[5]
[6]
[7]
[8]
[9]
[10]
[11]
[12]
[13]
[14]
[15]
[16]
[17]
[18]
[19]
[20]
D’APUZZO, N., Automated Photogrammetric Measurement of Human Faces, Int. Archives of
Photogrammetry and Remote Sensing, Hakodate, Japan, Vol. XXXII, Part B5, pp. 402-407, 1998.
D’APUZZO N., Measurement and modelling of human faces from multi images, International Archives
of Photogrammetry and Remote Sensing 34(5), pp 241-246, 2002
CHANG Y., A Photogrammetric System for 3D Reconstruction of a Scoliotic Torso, A Master Thesis,
Department of Geomatics Engineering, University of Calgary, Canada, 2008.
FISCHLER M. A., BOLLES R. C., Random Sample Consensus: A Paradigm for Model Fitting with
Applications to Image Analysis and Automated Cartography, Comm. of the ACM 24: 381–395, 1981.
HARRIS C. & STEPHENS M., "A combined corner and edge detector," Proceedings of Alvey Vision
Conference, 15: 147-151, 1988.
KRAUS K., Photogrammetry, Duemmler Verlag, Bonn, 1: 277-279, 1993.
LEWIS J. P., Fast normalized cross-correlation, Vision Interface, 120–123, 1995.
LOY G., ZELINSKY A., Fast radial symmetry for detecting points of interest. IEEE PAMI, Vol. 25, No.
8, pp 959-973, 2003.
LUONG Q. T., FAUGERAS O. D., The Fundamental Matrix: Theory, Algorithms, and Stability
Analysis, International Journal of Computer Vision 17 (1): 43–75, 1996.
MALIAN A., AZIZI A., VAN DEN HEUVEL F. A., MEDPHOS: A new photogrammetric system for
medical measurement, International Archives of the Photogrammetry, Remote Sensing and Spatial
Information Sciences, 35 (B5): 311-316, 2004.
MITCHELL H. L., Applications of digital photogrammetry to medical investigations, ISPRS Journal of
Photogrammetry and Remote Sensing, 50 (3): 27-36, 1995.
MITCHELL H. L., NEWTON I., Medical photogrammetric measurement: overview and prospects,
ISPRS Journal of Photogrammetry and Remote Sensing, 56 (5-6): 286-294, 2002.
NOBLE A., "Descriptions of Image Surfaces", PhD thesis, Department of Engineering Science, Oxford
University 1989.
PATIAS P., Medical imaging challenges photogrammetry, ISPRS Journal of Photogrammetry and
Remote Sensing, 56 (5-6): 295-310, 2002.
POPIELSKI P., WRÓBEL Z., An Attempt to Optimize the Process of Automatic Point Matching for
Homogeneous Surface Objects, Manuscript submitted for publication.
SCHENK T., Digital photogrammetry, TerraScience, Laurelville, Ohio, 428, 1999.
SCHMID C., MOHR R. & BAUCKHAGE C., Evaluation of interest point detectors, International
Journal of Computer Vision, 37 (2): 151-172, 2000.
SHAPIRO L., STOCKMAN G. C., Computer Vision, p. 257. Prentice Books, Upper Saddle River, 2001.
SHI J., TOMASI C., Good Features to Track, 9th IEEE Conference on Computer Vision and Pattern
Recognition. Springer, 1994.
TOMASI C., KANADE T., Detection and Tracking of Point Features, Pattern Recognition 37: 165–168,
2004
[21]
ZULIANI M., KENNEY C., MANJUNATH B.S., A Mathematical Comparison of Point Detectors,
Second IEEE Image and Video Registration Workshop (IVR), Washington, DC, Jun. 2004.
a)
b)
c)
d)
e)
f)
g)
h)
Figure 2. Image responses and extracted corners for algorithms: Harris (a,b), Nobel (c,d), Kanade-Tomasi (e,f),
Loy-Zelinsky (g,h).