EMBEDDED CIRCLE FITTING
EMBEDDED CIRCLE FITTING
Author
John Daly
Supervisor
Dr. Martin Glavin
COURSE
B.E. In Electronic and Computer Engineering
i
EMBEDDED CIRCLE FITTING
ACKNOWLEDGEMENTS
I would like to thank Dr. Martin Glavin and Eddie Jones for their help and supervision throughout
the year.
I would also like to thank the technicians, Miles Mehan, Martin Burke and, especially, Shaun Porter
for their help and invaluable advice.
Declaration of Originality
I declare that this thesis is my original work except where stated.
Signature
Date
__________________
___________________
ABSTRACT
Many inexpensive camera lenses, especially wide angle lenses, suffer from very clear distortion.
We show that even with such a lens near perfect images can be obtained by using a correction
algorithm based on a simple mathematical model and without any special measuring tools.
ii
EMBEDDED CIRCLE FITTING
Table of Contents
1
2
3
Introduction
1.1
Overview.........................................................................................
1
1.2
Project Aim.....................................................................................
2
1.3
Project Format.................................................................................
3
1.4
Background......................................................................................
4
1.5
Abbreviations..................................................................................
6
ALGORITHMIC DEVELOPMENT
2.1
MATLAB.........................................................................................
7
2.2
Edge Linking....................................................................................
8
2.3
Edge Detection.................................................................................
11
2.4
Circle Fitting....................................................................................
12
2.5
Applying Spatial Transformation.....................................................
13
EMBEDDING THE CODE
3.1
Field Programmable Gate Arrays.....................................................
14
3.2
Simulink……………………………………………………………
18
3.2.1 Implementation of the Algorithm in Simulink……………..
19
3.2.2 Conversion to Fixed Point…………………………………..
24
System Generator…………………………………………………..
25
3.3.1 Making the Model Amicable to Hardware Implementation…
28
Conclusion....................................................................................................
34
References.....................................................................................................
35
3.3
4
iii
EMBEDDED CIRCLE FITTING
TABLE OF FIGURES
Figure 1 Lens Distortion………….........................................................................
4
Figure 2 Pincushion Distortion………...................................................................
5
Figure 3 Original Image…………………………………………………………..
10
Figure 4 Detected Edges………………………………………………………….
10
Figure 5 Original Image…………………………………………………………..
12
Figure 6 Circle Fit………………………………………………………………...
12
Figure 7 Pincushion Effect………………………………………………………..
13
Figure 8 Transformed Image………………………………………………………
14
Figure 9 Evolution of the FPGA………………………………………………….
15
Figure 10 – Simulink Library Browser…………………………………………....
18
Figure 11 - Sobel Edge Detection Algorithm……………………………………...
19
Figure 12 – Sobel Implemented in Simulink……………………………………...
20
Figure 13 – Filter Specifications…………………………………………………..
21
Figure 14 – Complete Simulink System…………………………………………..
22
Figure 15 – Original……………………………………………………………….
23
Figure 16 – Edges………………………………………………………………….
23
Figure 17 – Data type Conversion…………………………………………………
24
Figure 18 – Data type Conversion…………………………………………………
24
Figure 19 – Xilinx Blockset……………………………………………………….
25
Figure 20 -System Generator Incorporation………………………………………
27
Figure 21 – Matrix Size…………………………………………………………...
28
Figure22 – More Elaborate Filter…………………………………………………
28
iv
EMBEDDED CIRCLE FITTING
Figure 23 – Convert to Serial Stream……………………………………………..
29
Figure 24 – More Elaborate Filter………………………………………………..
29
Figure 25 – Recreate Image………………………………………………………
30
Figure 26 – Generating Code……………………………………………………..
32
Figure 27 – Generating Code……………………………………………………..
32
Figure 28 - Generating Code……………………………………………………..
33
Figure 29 – FPGA in the Loop…………………………………………………...
34
v
EMBEDDED CIRCLE FITTING
1 INTRODUCTION
1.1
OVERVIEW
Some members of the Connaught Automotive Research (CAR) group in the Department of
Electronic Engineering are currently working on ways to estimate the curvature of an optical lens
with the aim of digitally compensating for the lens effects. Much of the work is focusing on the
radial distortion associated with wide-angle lenses.
When an image is captured with a very wide-angle lens, straight lines in the target become curved
in the resultant image. The exact curvature of the straight lines is directly linked to the geometry of
the lens, therefore if the curvature of the lines can be calculated accurately, then the lens itself can
be characterized. Once the curvature is known, it is possible to apply a transformation to
compensate for the lens effects.
Since the distortion caused by a wide angle lens is mainly radial, the resultant curves can be
considered as part of a circle. A circle fit algorithm can be used to calculate the size of the circle
which would relate to the amount of distortion.
This problem can be sometimes neglected but becomes critical in many applications of computer
vision where shape recognition, localization and tracking (in case of videos) is essential. Therefore,
a mathematical formulation of the distortion is needed, together with camera calibration methods to
correct the problem
1
EMBEDDED CIRCLE FITTING
1.2
PROJECT AIM
The aim if this project is to program a simple but reliable method to compute the best estimation of
this parameter to correct the radial distortion in an automatic or semiautomatic way. This method
can be applied to single images or to videos and can be adopted as an initial step for numerous
computer vision processes.
This project will involve the implementation of a circle fit algorithm which will help with the
characterization of a camera lens. The project will involve algorithmic development in MATLAB to
apply edge detection, edge linking and circle fit. Once this has been completed, the project will
move to an embedded implementation of some aspects of the circle fit algorithm. A DSP board will
have to be commissioned and the edge detection and edge linking algorithms implemented in
hardware.
2
EMBEDDED CIRCLE FITTING
1.3
PROJECT FORMAT
The project will be programmed in MATLAB initially following the following algorithm-
1. Prepare the image for processing.
2. Apply edge detection to image reducing it to its significant edges only.
3. Apply Edge Linking function.
4. Apply Circle fit function. Calculate the curvature.
5. Apply polynomial transformation to original image
When completed, the project can be converted to C code or simulink to prepare it for embedding.
Finally the code developed in MATLAB will be implemented on a DSP board.
3
EMBEDDED CIRCLE FITTING
1.4
BACKGROUND
When a lens with a field of view greater or smaller than this is used, the entry and exit angles are no
longer the same, and this causes perspective proportions to become distorted.
With a wide field of view the image has to be squeezed into a smaller space, as shown in figure 1.
The result is that scale and distance proportions between foreground and background are increased,
objects appearing smaller and distance greater the further away they are.
Figure 1
4
EMBEDDED CIRCLE FITTING
The squeezing and stretching of images vary radially due to the design of the lenses, making these
distortions visually most prominent at the image corners and sides. These distortions are shown
below in figure 2.
Figure 2 - Pincushion Distortion
These distortions are commonly corrected by first applying a parametric radial distortion model,
then estimating the distortion coefficients, and eventually correcting the distortion. Most of the
existing works on the radial distortion models can be traced back to an early study in
photogrammetry, where the radial distortion is governed by the following polynomial equation (ru:
undistorted radius, rd: distorted radius):
rd = f(ru) = ru + k1 ru3 + k2 ru5 + ….
Several tests have demonstrated that approximating the series with only the lower-order component
corrects more than the 90% of the radial distortion.
5
EMBEDDED CIRCLE FITTING
1.4
ABREVIATIONS
DSP
Digital Signal Processing
FIR
Finite Impulse Response
FPGA
Field Programmable Gate Array
HDL
Hardware Description Language
IOB
Input/Output Block
ISE
Integrated Software Environment
LMS
Least Mean Squares
LTE
Linear Transversal Equalizer
LUT
Look-Up Table
RAM
Random Access Memory
VHDL
Very High Speed Integrated Circuit Hardware Description Language
XST
Xilinx Synthesis Tool
2 ALGORITHMIC DEVELOPMENT
2.1
MATLAB
MATLAB is a high-performance language for technical computing. It integrates computation,
visualization, and programming in an easy-to-use environment where problems and solutions are
expressed in familiar mathematical notation.
The Image Processing Toolbox is a collection of functions that extend the capability of the
MATLAB numeric computing environment. The toolbox supports a wide range of image
processing operations, including Spatial image transformations, Morphological operations,
6
EMBEDDED CIRCLE FITTING
Neighborhood and block operations, Linear filtering and filter design, Transforms Image analysis
and enhancement, Image registration, Deblurring, and Region of interest operations.
Many of the toolbox functions are
MATLAB M-files, a series of
MATLAB statements that
implement specialized image
processing algorithms.2.2
Edge Detection
To prepare the picture for a circle fit algorithm, there are number of steps which we have first to
take. The first is edge detection.
There are number of different edge detection algorithms in MATLAB. Before you can apply an
edge detection algorithm to a picture, the picture must be converted into a binary image, otherwise
known as an intensity image. MATLAB's edge detection function takes a binary image I as its
input, and returns a binary image BW of the same size as I, with 1's where the function finds edges
7
EMBEDDED CIRCLE FITTING
in I and 0's elsewhere.
EDGE supports six different edge-finding methods:
The Sobel method finds edges using the Sobel approximation to the
derivative. It returns edges at those points where the gradient of
I is maximum.
The Prewitt method finds edges using the Prewitt approximation to
the derivative. It returns edges at those points where the gradient
of I is maximum.
The Roberts method finds edges using the Roberts approximation to
the derivative. It returns edges at those points where the gradient
of I is maximum.
The Laplacian of Gaussian method finds edges by looking for zero
crossings after filtering I with a Laplacian of Gaussian filter.
The zero-cross method finds edges by looking for zero crossings
8
EMBEDDED CIRCLE FITTING
after filtering I with a filter you specify.
The Canny method finds edges by looking for local maxima of the
gradient of I. The gradient is calculated using the derivative of a
Gaussian filter. The method uses two thresholds, to detect strong
and weak edges, and includes the weak edges in the output only if
they are connected to strong edges. This method is therefore less
likely than the others to be "fooled" by noise, and more likely to
detect true weak edges.
I chose the Canny method for the reason that it is less likely to fooled by noise and that the
“strength” of the edges can be specifed.
What the edge detection leaves is a often a very broken trace of the edges within the original image,
an edge linking function is now needed to clean up the new image.
9
EMBEDDED CIRCLE FITTING
Figure 3 - Original Image
Figure 4 – Detected Edges
2.3
Edge Linking
Linking together the detected edges, a number of functions must be called to link the edges.
“Edgelink” is a function that links edge points together into chains. Where an edge diverges at a
junction the function simply tracks one of the branches. The other branch is eventually processed as
another edge. It returns an edgelist, a cell array of edge lists in row, column coordinates and an
image of all the edges labeled with an edge number.
The “LineSeg” function takes each array of edgepoints in edgelist, finds the size and position of the
maximum deviation from the line that joins the endpoints, if the maximum deviation exceeds the
10
EMBEDDED CIRCLE FITTING
allowable tolerance the edge is shortened to the point of maximum deviation and the test is
repeated. The LineSeg function allows you to specify the tolerance i.e. Maximum deviation from
straight line before a
segment is broken in two (measured in pixels), the Angle Tolerance
which is used when attempting to merge line segments (radians) and the maximum distance
between end points of line segments for segments to be eligible for linking (pixels).
The “MergeSeg” is then called, it scans through the list of line segments seeking to merge segments
together. Segments are merged if the orientation difference is less than the specified angle tolerance
and if the ends of the segments are within a specified maximum distance of each other. The
comparison is performed by first sorting line segments by angle, then each line segment is only
tested against other line segments that satisfy the orientation constraint.
Finally “DrawSeg” draws the series of line segments stored in the Nx4 array that mergeseg
created.2.4
Circle Fit
The Edge Detection and Edge Linking algorithms have created an image that has stripped down the
original image to its significant edges. A circle fit algorithm can now be applied to these edges to
see what sort of curvature the originally straight edges have suffered.
This in done using the Linear least-squares circle in the plane method - an approximation to the true
least-squares circle.
11
EMBEDDED CIRCLE FITTING
I have found that radius of under a certain threshold should be ignored, as the distortion caused by a
lens affects straight lines which leave large radii.
Figure 6 – Circle Fit
Figure 5 - Original Image
2.3
Applying Spatial Transformation
Where the radial distortion is governed by the following polynomial equation (ru: undistorted
radius, rd: distorted radius):
rd = f(ru) = ru + k1 ru3 + k2 ru5 + ….
The distortion parameter k1 changes with the focal length. But its value does not vary by huge
amount. Having selected k1, it is clear that it’s not necessary to calculate the radial distance and the
correction factor and so on every time.
12
EMBEDDED CIRCLE FITTING
Figure 7 – Pincushion Effect
The curvature in the above figure is characterized by a number, a number which will relate to an
average curvature produced by lens distortion, indicating whether the picture suffered more or less
distortion than the normal code corrects and can adjust the polynomial equation accordingly.
% radial pin cushion distortion
xt = xi(:) - imid;
yt = yi(:) - imid;
[theta,r] = cart2pol(xt,yt);
a = -.0000009; % Try varying the amplitude of the cubic term(a = -.00000037;).
s = r + a*r.^3;
[ut,vt] = pol2cart(theta,s);
u = reshape(ut,size(xi)) + imid;
v = reshape(vt,size(yi)) + imid;
tmap_B = cat(3,u,v);
I_pin = tformarray(I,[],resamp,[2 1],[1 2],[],tmap_B,.3);
13
EMBEDDED CIRCLE FITTING
Figure 8 Transformed Image
3
3.1
EMBEDDING THE CODE
Field Programmable Gate Arrays
A field programmable gate array (FPGA) is a semiconductor device containing programmable logic
components and programmable interconnects. The programmable logic components can be
programmed to duplicate the functionality of basic logic gates such as AND, OR, XOR, NOT or
more complex combinational functions such as decoders or simple math functions. In most FPGAs,
these programmable logic components (or logic blocks, in FPGA parlance) also include memory
elements, which may be simple flip-flops or more complete blocks of memories.
14
EMBEDDED CIRCLE FITTING
FPGAs have evolved from those where DSP structures were built using logic-only cells to those
having dedicated embedded DSP structures.
Figure 9 shows how FPGAs for signal processing have evolved over the years.
For this project, Xilinx was chosen as the FPGA platform for the following reasons:
- Xilinx is a market leader in FGPA technology
- Xilinx produces a blockset for Simulink
Xilinx software tools were available to me in this college.
An FPGA is an integrated circuit that is programmable by the designers rather than by the device
manufacturer. The FPGA may be re-programmed by the designer rather than by the device
manufacturer. The FPGA may be re-programmed unlike an ASIC (Application Specific Integrated
15
EMBEDDED CIRCLE FITTING
Circuit). An FPGA provides a 2-Dimensional array of configurable logic resources that can be used
to implement a wide range of functions. These resources include:-
- Look-up Tables (LUTs)
- Registers
- Memories
- Multipliers
- Multiplexers
- Digital clock managers
An FPGA is programmed by downloading a configuration bitstream into SRAM memory. Because
the configuration is stored in SRAM, FPGAs are volatile devices. FPGAs provide an ability to
construct highly parallel architectures for processing data. The fast clock rates (up to 200MHZ) and
the distributed memory architecture allow the designer to exploit the parellelism of DSP
application.
Advantages of FPGA include:-
- Lower non-recurring engineering costs compared with an ASIC
- Shorter time to market
- Configurability
16
EMBEDDED CIRCLE FITTING
With the continued growth in complexity of FPGA-based designs, the need for a more flexible and
efficient design methodology has arisen. Currently, most designs are accomplished through the use
of HDL-centric flows. However, device densities have increased at a pace that such flows have
become both cumbersome and outdated. The need for a more innovative and higher-level design
flow that directly incorporates model simulation with hardware implementation is needed.
3.2
SIMULINK
Simulink is a well-known tool which allows designers to model a system at a high level and is ideal
for certain classes of applications. Simulink is an extension of MATLAB that allows creating
algorithms in a graphical fashion. It provides an interactive graphical environment and a
17
EMBEDDED CIRCLE FITTING
customizable set of block libraries, and can be extended for specialized applications. It offers tight
integration with the rest of the MATLAB environment and both drive MATLAB or be scripted
from it.
Figure 10 – Simulink Library Browser
3.21 IMPLEMENTATION OF THE ALGORITHM IN SIMULINK
Figure 11 - Sobel Edge Detection Algorithm
A typical block diagram of a sobel edge detection algorithm. The Input image is passed through
two 2-D Filters, the top computing the X gradients, the bottom doing the Y gradients. We square
the results of the filtering, sum them together, we square root the results, pass them through a
thresholding stage. The output will be an image which will represent the edges of the original
image.
This is an example of a block diagram representation that is a very natural representation for
18
EMBEDDED CIRCLE FITTING
algorithms in signal processing. This is the basis of how simulink is coded.
To implement this algorithm in Simulink, it is just a case of finding the equivalent blocks and
dragging them into line. If you look at the model below you will see the resemblance with the
original algorithm above.
Figure 12 – Sobel Implemented in Simulink
19
EMBEDDED CIRCLE FITTING
The filters in our model are coded individually by putting the kernel edge detection for each
gradient into the coefficients input option.
Figure 13 – Filter Specifications
20
EMBEDDED CIRCLE FITTING
This is our algorithm and can now be put into a subsystem and incorporated into a fully functioning
model to view an original image and a seperate image to view the edges.
Figure 14 – Complete Simulink System
21
EMBEDDED CIRCLE FITTING
Figure 15 – Original
Figure 16 - Edges
22
EMBEDDED CIRCLE FITTING
3.2.2 CONVERSION TO FIXED POINT
In preparation for downloading to an FPGA, we need to convert our model to fixed point.
Figure 17 – Data type Conversion
So we do that by converting each block at a time to fixed point numbers, here we are using a nine
bit unsigned number. This can be optimized, however, in order to save space on the FPGA.
Figure 18 – Data type Conversion
23
EMBEDDED CIRCLE FITTING
3.3
SYSTEM GENERATOR
System Generator for DSP is a software platform that uses Simulink to represent a high-level
abstract view of a DSP system. It then automatically maps the system to an efficient hardware
implementation enabling the customization of the Xilinx architecture to suit the DSP algorithm.
Using System generator greatly shortens the path from design concept to working hardware by
bridging the gap between high-level system design and actual implementation on a Xilinx FPGA.
System Generator takes this graphical algorithmic approach and extends it to FPGA development
by using special Simulink Blocks that can easily be translated into FPGA resource and loaded onto
the FPGA. This allows the engineer to visually follow algorithms without getting lost in low level
code.
Figure 19 – Xilinx Blockset
24
EMBEDDED CIRCLE FITTING
When System Generator processes System Generator blocks and imported cores, it indirectly uses
Xilinx ISE Tools.
System Generator has a number of advantages:
-High-Level Abstraction
Algorithmic development no longer consists of coded processes used in conjunction with
cores, signal routing, and clocking concerns. With System Generator, the user outlines
higher level connectivity and algorithmical blocks while being abstracted away from lowlevel hardware concerns. This allows the user to concentrate on more algorithmic
development as opposed to low-level coding concerns.
25
EMBEDDED CIRCLE FITTING
-Flexibility
System Generator can be incorporated as a component into a top-level VHDL project. This
allows flexible algorithmic development while allowing VHDL usage for board specific
FPGA concerns.
Figure 20 - System Generator Incorporation
System Generator also allows integration of VHDL modules and cores into a System
Generator Project. This allows engineers familiar with VHDL to easily incorporate existing
modules with System Generator.3.3.1
MAKING THE MODEL AMICABLE
TO HARDWARE IMPLEMENTATION
26
EMBEDDED CIRCLE FITTING
Figure 21 – Matrix Size
Our filters are accepting matrix size 200x100 and, in practice, we cannot send a matrix directly to
the FPGA. We need to serialize this data and before sending it to the FPGA.
Figure22 – More Elaborate Filter
27
EMBEDDED CIRCLE FITTING
Figure 23 – Convert to Serial Stream
This subsystem converts a matrix to a serial stream of data. Since this simulates a real time system,
the output rate is much higher than the input rate. In this example, the output rate is 20000 times the
input rate (Matrix Size 200x100). This is represented in Simulink by a red signal at the output
versus a green signal at the input.
Figure 24 – More Elaborate Filter
This is an implementation of the 2D filter that is closer to real implementation in the hardware. As
you see, two line buffers are used to get a serial stream of data (that represents an image) and do a
2D-filtering on it.
28
EMBEDDED CIRCLE FITTING
Figure 25 – Recreate Image
This subsystem recreates the image from a stream of data. The output rate in this case is less than
the input rate.
29
EMBEDDED CIRCLE FITTING
3.3.2 IMPLEMENTING ON THE FPGA
The final step is to implement the filter using the blocks provided in the Xilinx Blockset. The
blocks used here are all familiar VHDL functions. Once your filter works using the Xilinx blocks,
system generator will generate working VHDL code automatically.
30
EMBEDDED CIRCLE FITTING
When you double click the system generator Icon, the system generator allows you to select the
particular board for which you wish to generate code and where you wish to copy the files
generated.
Figure 26 – Generating Code
Figure 27 – Generating Code
31
EMBEDDED CIRCLE FITTING
Figure 28 - Generating Code
To take it one step further, now it is possible to download the filter to the FPGA using system
generator and run the simulation with hardware in the loop.
Figure 29 – FPGA in the Loop
32
EMBEDDED CIRCLE FITTING
4
Conclusion
The proposed method is a robust approach to detect and correct radial lens distortion. It can be
applied to different types of images, stored or directly acquired by cameras, and can be considered a
powerful initial step for many computer vision applications. It is particularly valuable if manual
calibration of the camera is not possible since no information about the camera is available. The
method’s precision depends on the number of samples, the Hough space’s resolution, and the
straightness of the cues present in the scene. A human observer often perceives the distortion
phenomenon because straight lines are distorted: our proposal relies on this and thus gives us a
good approximated solution.References:
Harri Ojanen, “Automatic Correction of Lens Distortion by Using Digital Image Processing”, 1999
R. Cucchiara, C. Grana, A. Prati, R. Vezzeni, “A Hough Transgorm-based method for Radial Lens
Distortion Correction”, 2003
Neil O'Keeffe, “Programmable Logic For DSP Applications”, 2004
33