C#, OpenCV, and simple
Camera/Robot Calibration
Sebastian van Delden
USC Upstate
svandelden@uscupstate.edu
OpenCV

Open Source Computer Vision Library

OpenCV is Intel’s Open Source Computer
Vision Library.
It is a collection of C functions and a few C++
classes that implement some popular Image
Processing and Computer Vision algorithms.
EMGU provides wrapper class packages so
that OpenCV can be used in C#.


Features

Image data manipulation


Image and video I/O


products, solvers, eigenvalues, SVD
Various dynamic data structures


file and camera based input, image/video file output
Matrix and vector manipulation, and linear algebra
routines


allocation, release, copying, setting, conversion
lists, queues, sets, trees, graphs
Basic image processing

filtering, edge detection, corner detection, sampling and
interpolation, color conversion, morphological operations,
histograms, image pyramids
3
Features (cont.)

Structural analysis


Camera calibration


eigen-methods, HMM
Basic GUI


optical flow, motion segmentation, tracking
Object recognition


finding and tracking calibration patterns, calibration, fundamental
matrix estimation, homography estimation, stereo correspondence
Motion analysis


connected components, contour processing, distance transform,
various moments, template matching, Hough transform, polygonal
approximation, line fitting, ellipse fitting, Delaunay triangulation
display image/video, keyboard and mouse handling, scroll-bars
Image labeling

line, conic, polygon, text drawing
4
Emgu CV


Emgu CV is a cross platform .Net wrapper to the Intel
OpenCV image processing library. Allowing OpenCV
functions to be called from .NET compatible
languages such as C#, VB, VC++, IronPython etc.
Home Page:


Online Documentation:


http://www.emgu.com/wiki/index.php/Main_Page
http://www.emgu.com/wiki/files/2.2.1/document/Index.html
Download the executable version and install it.
 Version 2.1.0 is on our course website:
http://faculty.uscupstate.edu/svandelden
Getting Started… Create a new
Windows Form Application
Add the Emgu Libraries as project References
Setting up the Window

Open your toolbox and add a
SplitContainer to your
Window.


Split it horizontally
This creates two “Panel”s
where we can add elements.

Idea: we are going to add the
video stream to the below panel
and results/data etc to the top
panel.
You can
change the
properties
of any of
the
elements in
the
Windows
form by
clicking on
the
Properties
to the
right…
When you are building the Windows Form, you are in “Design” view… Right
click on it and choose “View Code” to see the underlining code:
Notice that this is a partial
class.
The InitializeComponent()
method is automatically
created and creates the
Windows Form that you’ve
been building.
To view the other partial class, click on the “designer” cs file in the
Solution Explorer.
If you expand this, you
can see all the code that
sets up the Windows
Form. NOTE: Never
modify this code directly
at all, because it gets recreated every time you
make changes to the
design of the form!!
From your Toolbox, add an ImageBox to the lower panel.
Drag he Panel sizes so that it looks like this.
Note: if you are
having problems
selecting the
SplitContainer,
right click on the
form and choose
“Select
SplitContainer”
Add a button labeled “Start Camera” to Panel1, then double click
on it to create and view a button1_Click method (event handler)…
Include the EMGU libraries.
Create two data attributes :
Capture _capture is the
EMGU CV class for
connecting to a camera
The boolean will be used in
toggling the camera off and
on.
Button Click event handler.
Add a user-defined method called ProcessFrame that
will be doing our computer vision stuff. For now, just
have it get the camera’s current frame and add it to the
ImageBox on the Windows Form.
Add this code to the button’s click method:
Instantiate the
EMGU CV
Camera object
Toggle the
camera on and
off.
Try Running the program and click the Start Camera button.
Try applying a built-in EMGU CV computer
vision algorithm to the image.

For example, convert to gray scale image
and threshold (cut off 100 and include pixels
up to 255:
Try it out..
Try Using the Inverted Threshold method (so that the
background is black) and also do an “closing” – dilation
followed by erosion.
Camera/Robot Calibration
Camera Calibration
Camera/Robot Calibration


We will calibrate the camera’s coordinate
system with the robot’s world coordinate
system.
We assume that the depth (Z) of the points
on robot world system are known and equal.



Flat table top of the robot’s work area
This is significant because the robot world points
form a plane in 3D space.
Since depth is known, we only need to calibrate
camera (X,Y) pairs to robot (X,Y) pairs
3x3 Perspective Transformation

Emgu has a built-in method that will compute
the 3x3 perspective transformation based on
at least 4 camera-robot (X,Y) pairs.

Need at least 4 points (8 values in total) because
we need to recover 8 values:
a
d
g
b
e
h
c
f
1
The Transformation

The transformation captures/recovers:
Translation:
Scaling:
Rotation:
Shear:
The Emgu CV

HomographyMatrix PerM =
CameraCalibration.GetPerspectiveTransform(
PointF[ ] CameraPoints,
PointF[ ] RobotPoints
);

NOTE: “Homography” – square, invertible matrix.
Example Program

Provide for you is a C# calibration program
that



connects to the V+ robot via TCP/IP
displays the video from the camera
allows you to click on the image


When you do so, the picture X,Y along with the robot
X,Y location of the tool frame are captured.
generate the calibration matrix based on the
recovered points.
Activity

Use this program to calibrate your camera to
your robot.