Distortion Correction
ECE 6276 Project Review
Team 5:
Basit Memon
Foti Kacani
Jason Haedt
Jin Joo Lee
Peter Karasev
Outline
• Motivation
• Components
• Component Optimization
• Current Results
• Plans for Catapult C
• Schedule
2
ECE 6276 Final Project Team 5
7/14/2009
Objective
Given a distorted image with known size and
known lens distortion parameter, generate an
undistorted image.
3
ECE 6276 Final Project Team 5
7/14/2009
Motivation – Why?
• The formation of undistorted images can be
described by a series of matrix multiplications
• Distortion makes it very difficult to calibrate a
camera to measure geometry (depth, size,
orientation, etc)
• Many applications in image processing and
computer vision like structure estimation, image
mosaicing, and ultimately vision-based control.
4
ECE 6276 Final Project Team 5
7/14/2009
Motivation contd..
Application: Measure motion and geometry
Problem: Known geometry in the scene is warped,
relationship between 3D and 2D points is nonlinear.
Solution: Undo the distortion, so x2D = A * X3D
5
ECE 6276 Final Project Team 5
7/14/2009
Literature Review (I)
K.T. Gribbon, C.T. Johnston, and D.G. Bailey, “A Real-time FPGA
Implementation of a Barrel Distortion Correction Algorithm with
Bilinear Interpolation”
•Focus on reducing hardware complexity.
•Uses LUTs to store mapping data.
•No quantitative results provided.
•Logic resource utilization on RC-100 is 51%
6
ECE 6276 Final Project Team 5
7/14/2009
Literature Review (II)
Qiang, L.; Allinson, N.M.;” FPGA Implementation of Pipelined
Architecture for Optical Imaging Distortion Correction”, Signal
Processing Systems Design and Implementation, 2006. SIPS '06.
•Same
algorithm
as
previous
one
•Implementation on a Xilinx FPGA XCS3 1000-4
uses
75%
of
the
hardware
multipliers.
•Residual error of the undistorted image was 1.5%
of the distorted image.
7
ECE 6276 Final Project Team 5
7/14/2009
Literature Review (III)
Hany Farid & Alin C. Popescu, “Blind Removal of Lens Distortion”, Journal
of the Optical Society of America 2001
•For removal of distortion in absence of any calibration data.
•Uses polyspectral analysis to detect higher order correlations
in frequency domain which are proportional to the distortion.
•Computationally intensive.
•No quantitative results.
•Accuracy is not comparable to those based on known
distortion parameters.
8
ECE 6276 Final Project Team 5
7/14/2009
Components to Achieve Objective
Component 1: Matlab forward distortion function
– Verifies correctness of undistortion algorithm
Component 2: Data ordering test bench
– Order the C++ input stream from MATLAB generated data
Component 3: Undistortion lookup table or lookup function
– Initial prototype in MATLAB
Component 4: Least squares interpolation lookup function
– Initial prototype in MATLAB
– Compare different techniques such as LUT vs NEAREST NEIGHBOR
Component 5: Verification structure
– Compare original C++ result to undisorted image in MATLAB
MATLAB C++  Catapult C  MATLAB
9
ECE 6276 Final Project Team 5
7/14/2009
Component Optimization Questions
• What size buffers do we need to compare against
previous frames?
• For undistortion function can we compute them
dynamically or do we need a pre-defined LUT?
• For finding fast least squares / linear system solver,
compare speed cost vs. nearest neighbor and effect
on output error
10
ECE 6276 Final Project Team 5
7/14/2009
Matlab Demo of Algorithm - Original
11
ECE 6276 Final Project Team 5
7/14/2009
Matlab Demo of Algorithm - Distorted
12
ECE 6276 Final Project Team 5
7/14/2009
Matlab Demo of Algorithm - Recovered
13
ECE 6276 Final Project Team 5
7/14/2009
Design Goals
• Limited Buffers
• 8 bits per sub-pixel (24 bits total)
• Resolution (up to 640x480)
• Concerned with geometry
• Area, Throughput, Latency
•Target a low-cost implementation that handles
consumer video application pixel clocks of 165
MHz (Apprx 6 ns cycle time).
14
ECE 6276 Final Project Team 5
7/14/2009
Test Vectors
The Line Test
Original Image
Distorted Image
Recovered Image
15
ECE 6276 Final Project Team 5
7/14/2009
Plans for Catapult C Code
• Use C/C++ and Algorithmic C data types to describe
synthesizable hardware
• Architectures (type of hardware interface (streaming
buffers) )
• Constraints (Throughput, area, latency)
•RTL generation and verification
• Optimizations
-Pipelining
-Parallelism
-Loop Unrolling
-Scheduling
-Streaming buffers/ Read & Write
16
ECE 6276 Final Project Team 5
7/14/2009
Project Timeline
Tasks
Algorithm Research
Owner
ALL
Description
Week 1
Research different papers on barrel distortion
Mock up distortion algorithm using MATLAB and use image
MATLAB Implementation
Peter
processing libraries for ease of use
Take MATLAB results and convert them into C++ structure to
MATLAB/C++ Testbench
Jason
be used in C++ testbench
C++ Code Porting
Jin Joo
Port MATLAB implementation of undistortion to C++
Presentation 1
ALL
Initial project report
Catapult C Porting
Foti,Memon
Port to supported CATAPULT datatypes
Catapult C Synthesis
Jason
See results of CATAPULT algorithm
Plans for Optimization
Foti, Memon
Look at ways to optimize CATAPULT code
Completed Catapult Code, Synthesis Results, Steps for
Presentation 2
ALL
Optimization
Catapult C Optimization
Jin Joo
Complete optimization suggestions
Catapult C Synthesis of Optimization
Jason
See results of CATAPULT algorithm
Demo
ALL
Show CATAPULT demo and final HDL
Presentation 3
ALL
Final report out
Week 2
Week 3
17
ECE 6276 Final Project Team 5
7/14/2009
Project Risks
Risk 1:
Slowness of interpolation plan: the key goal of undistortion is “geometrical
accuracy” which hold even if noise is injected. To mitigate this risk we have a
forward mapping algorithm that is fast compared to any interpolation method but at
the cost of missing pixels near the edges.
Risk 2:
Not enough storage space to keep a lookup table of coordinates. To mitigate this
risk we could compute coordinates on the fly at the cost of math operations.
Risk 3:
Cannot find least squares method for FPGA. To mitigate this risk we could do
nearest neighbor interpolation.
18
ECE 6276 Final Project Team 5
7/14/2009
Current Status
•Straight lines are recovered with minimal injection of
missing points or noise through our algorithm.
•Add more quantitative results on the geometrical
accuracy of recovered images than presented in
previous results.
•Generalize the coordinate mapping to make the
implementation more robust to varying distortion
models and hence supporting more cameras “on the
fly.”
19
ECE 6276 Final Project Team 5
7/14/2009
References
• Yi Ma, Stefano Soatto, et al., “An Invitation to 3-D Vision”
•Richard Hartley, Andrew Zisserman, “Multiple View Geometry in
Computer Vision”
•Edward M. Mikhail, James S. Bethel, J. Chris McGlone, “Introduction
to Modern Photogrammetry”
20
ECE 6276 Final Project Team 5
7/14/2009
Questions?
?
21
ECE 6276 Final Project Team 5
7/14/2009