Introduction to
IPL and OpenCV libraries
Bogdan Raducanu
Centre de Visiò per Computador
E-mail: bogdan@cvc.uab.es
Cover Story
OpenCV was of key use in the vision system of "Stanley"
What is IPL?
- The IPL (Image Processing Library ) is a collection of functions
implementing several image processing algorithms. It was developed by
INTEL.
- Is optimized for MMX and different processor types (there is a DLL for
each type of INTEL processor)
- The images are stored in a specific data structure. In order to work with the image,
we need to know the information contained in the structure header
(contains image specific characteristics)
Image structure
IplImage *
IplImage
• Width
• Height
• Bits per pixel
• Channel sequence
•…
• pointer to image data
Data
The “IplImage” structure includes a header containing image information and attributes:
- nChannels: number of channels
(1 for grayscale, 3 for RGB, 4 for CMYK, ...)
- depth: number of bits/pixel and data type
- IPL_DEPTH_1U (1-bit)
- IPL_DEPTH_8U (8-bit unsigned)
- IPL_DEPTH_8S (8-bit signed)
- IPL_DEPTH_16U (16-bit unsigned)
- IPL_DEPTH_16S (16-bit signed)
- IPL_DEPTH_32S (32-bit signed)
- IPL_DEPTH_32F (32-bit float)
- colorModel: “GRAY”, “RGB”, “CMYK”, etc.
- channelSeq: “GRAY”, “BGR”, “BGRA”, “RGB”, “RGBA”, “HSV”, “YUV”, etc.
- dataOrder: RGBRGBRGB... or RRR...GGG...BBB
- origin: Top-Left or Bottom-Left (IPL_ORIGIN_TL ó IPL_ORIGIN_BL)
- scanline alignment: DWORD or QWORD
- width (in pixels)
- height (in pixels)
- ROI: (could be NULL)
- maskROI: (could be NULL)
- imageSize: image size (in bytes)
- imageData: pointer to pixel data
Function categories in IPL:
- create/destroy an image and access its content
- arithmetical/logical operations
- filtering
- morphological operations
- color space conversion
- histogram
- linear/geometrical transformations
- image statistics
Create/destroy an image and access its content:
- iplCreateImageHeader
- iplAllocateImage
- iplCreateROI
- iplSetROI
- iplCopy
- iplClone
- iplDeallocateImage
- iplPutPixel
- iplGetPixel
Example (create/destroy an image):
#include ”ipl.h”
…
IplImage *img = iplCreateImageHeader(
3, 0, IPL_DEPTH_8U, “RGB”, “BGR”, IPL_DATA_ORDER_PIXEL,
IPL_ORIGIN_TL, IPL_ALIGN_QWORD, 150, 100, NULL, NULL, NULL, NULL );
iplAllocateImage(img, 0, 0);
/////// Use the image ////////
iplDeallocate(img, IPL_IMAGE_ALL);
• We created an image of 150x100 pixels with 3 channels. The color model is
RGB and data type is 8 bits/pixel, unsigned. The channel order is
BGRBGRBGR… starting from the upper row. The data are aligned in memory as
QWORD (64 bits). There is no ROI defined
• We allocated memory for data, but without initialize it.
• With the IPL_IMAGE_ALL parameter, we freed the header of the structure, the
data and the existing ROIs (if any)
- The content of an IplImage can be accessed in several ways:
- using the functions GetPixel and PutPixel:
-Inconvenience: slow access
- go directly to the memory address corresponding to the pixel al pixel:
- Inconvenience:
•this operation is ‘complex’ because we have to compute the
memory address beforehand
•careful with the data types
- Advantage:
•is faster to access a big chunk of data
- Example: Let’s assume we have a RGB image with 8 bits/pixel, unsigned:
#include ”ipl.h”
…
IplImage *img = iplCreateImageHeader(
3, 0, IPL_DEPTH_8U, “RGB”, “BGR”, IPL_DATA_ORDER_PIXEL,
IPL_ORIGIN_TL, IPL_ALIGN_QWORD, 150, 100, NULL, NULL, NULL,
NULL );
iplAllocateImage(img, 0, 0);
unsigned char *R,*G,*B;
B = (unsigned char *) img->imageData;
G = B+1;
R = G+1;
for (int i=0; i<15000; i++, R+=3, G+=3, B+=3) {
///// here we can use/modify the pixel data
}
iplDeallocate(img, IPL_IMAGE_ALL);
Remarks:
- in order to be able to use the IPL functions, you must include the header file “ipl.h” in your source code
- include the “ipl.lib” in the project settings
- the structures’ names use the “Ipl” (“I” uppercase) prefix, meanwhile the functions’ names use the “ipl” (“i”
lowercase) prefix
- the remaining function categories will be presented in the section dedicated to OpenCV library
- for more information, consult the IPL manual:
C:\Archivos de Programa\Intel\plsuite\doc\iplman.pdf
OpenCV library
Why OpenCV?
- IPL is a “low-level” library (it allows basic operations)
- OpenCV is a library which contains more complex data structures and “high-level” functions: optical flow,
pattern recognition 2D-3D real-time tracking, camera calibration, etc.
- it comes with some extensions that allow:
- accessing a camera or working with AVI files
- graphical user interface (“HighGUI”) allowing a faster and easier way
to interact/visualize the images
OpenCV is in general compatible with the IPL library. It is based also on the “IplImage” structure. But it must be
employed taking into account the following restrictions:
- the image statistics functions require that “IplImage” be defined either with a single channel or three channels
of the following data types: IPL_DEPTH_8U, IPL_DEPTH_8S or IPL_DEPTH_32F.
- OpenCV supports only interleaved images
- the following attributes: colorModel, channelSeq, BorderMode, and BorderConst are ignored
- the attibutes maskROI and tileInfo must be set to 0.
- the ROIs of the input and output image must be the same.
Remark: the structures’ name uses the “Cv” (“C” uppercase) prefix, meanwhile the functions’ name uses the
“cv” (“c” lowercase) prefix.
Create/Destroy an image and access its content
- cvCreateImage
- cvCreateImageHeader
- cvReleaseImageHeader
- cvReleaseImage
- cvCreateImageData
- cvReleaseImageData
- cvSetImageROI
- cvCopyImage
- cvCloneImage
- cvGetImageRawData
#include ”ipl.h”
#include “cv.h”
#include “cxcore.h”
…
IplImage *img = cvCreateImage(cvSize(150, 100), IPL_DEPTH_8U, 3);
/////// Use the image ////////
cvReleaseImage(&img);
Remark: In some cases (when we work with images captured from the
camera), would be more convenient to initialize the IplImage structure using
the IPL functions.
Arithmetical/Logical operations
- Arithmetical operations:
- unary; cvAddS, cvSubS, ...
- binary: cvAdd, cvMul, cvSub, cvMatMulAdd, cvInvert, ...
- Logical operations
- unary: cvAndS, cvOrS, cvXorS, ...
- binary: cvAnd, cvOr, cvXor, ...
Remark:
Most of the OpenCV functions are defined to support both the IplImage and
CvMat data types. That’s possible because of the definition of CvArr data
type:
typedef void CvArr;
The OpenCV functions look-up for the first integer of the structure being
passed, in order to distinguish between the two data types. In the case
of IplImage, this integer is equal to the size of IplImage structure,
meanwhile it is equal to 0x4224xxxx in the case of CvMat.
Image filtering
- Based on convolution with fixed kernel:
- cvLaplace, cvSobel, cvSmooth
Morphological operations
- cvErode, cvDilate
- cvMorphologyEx (advanced operations: opening, closing, Top-Hat, etc.)
- user-defined structuring elements:
- cvCreateStructuringElementEx
- cvReleaseStructuringElement
Example: application of the erosion function
Color space conversion
- cvCvtColor allows the following color-space conversions:
- CV_RGB2GRAY
- CV_RGB2HSV
- CV_RGB2YCrCb
Histogram
- cvCreateHist, cvReleaseHist
- cvCalcHist, cvCopyHist
- cvCompareHist, cvThreshHist
- cvGetMinMaxHistValue, cvNormalizeHist
Linear/geometrical transformations
- cvFFT (Fast Fourier Transform)
- cvDCT (Discrete Cosine Transform)
- cvResize, cvMirror, cvConvertScale
Feature extraction
- cvCanny (border detection)
- cvHoughLines (line detection)
- cvFindCornerSubPix (corner detection)
Example of border detection
Image statistics
- cvNorm (C- , L1- and L2-norm)
- cvMoments (spatial and central moments)
- cvMinMaxLoc (find the min/max values)
Drawing functions
- cvLine, cvRectangle
- cvCircle, cvEllipse
- cvPolyLine, cvFillPoly
- cvInitFont, cvPutText
Motion analysis
- estimators: Kalman, Condensation
- cvKalmanXX, cvCondensXX
- movement patterns
- cvCalcMotionGradient, cvMotionHistoryUpdate
- optical flow
- cvCalcOpticalFlowPyrLK (implements the Lucas-Kanade method based on pyramidal decomposition)
- tracking
- cvMeanShift, cvCamShift, cvSnakeImage
- background substraction
3D reconstruction
- camera calibration
- cvCalibrateCamera, cvFindExtrinsecCameraParams, cvUnDistort
- hand detection
- cvFindHandRegion
- pose estimation
- cvPOSIT
- finding pixel correspondence in a pair of stereo images
- cvFindStereoCorrespondence
Example of distorsion correction
Object detection (faces)
The object detection algorithm implemented in OpenCV is based on the following papers:
- Paul Viola and Michael J. Jones. Rapid Object Detection using a Boosted Cascade of Simple Features. IEEE CVPR,
2001
and
- Rainer Lienhart and Jochen Maydt. An Extended Set of Haar-like Features for Rapid Object Detection. IEEE ICIP
2002, Vol. 1, pp. 900-903, Sep. 2002.
Idea
A classifier (namely a cascade of boosted classifiers working with haar-like features)
is trained with a few hundreds of sample views of a particular object (i.e., a face or a car), called positive
examples, that are scaled to the same size (say, 20x20), and negative examples - arbitrary images of the same
size.
- The weak classifier outputs a "1" if the region is likely to show the object of interest (i.e., face/car), and "0"
otherwise
- The classifier is designed so that it can be easily "resized" in order to be able to find the objects of interest at
different sizes, which is more efficient than resizing the image itself.
- The word "cascade" in the classifier name means that the resultant classifier consists of several simpler
classifiers (stages) that are applied subsequently to a region of interest until at some stage the candidate is
rejected or all the stages are passed.
Data structures and functions implemented in OpenCV for face detection
- CvHaarClassifierCascade (structure representing a cascade of classifiers)
- cvLoadHaarClassifierCascade (reads from a file a cascade of classifiers and stores it in the
CvHaarClassifierCascade structure). The cascade is stored in a XML file format:
C:\Archivos de programa\Intel\OpenCV\data\haarcascades\...)
- cvHaarDetectObjects (detects the objects in the image)
- cvReleaseHaarClassifierCascade (frees the memory occupied by the CvHaarClassifierCascade structure)
HighGUI library
Allows a fast and easy interaction/visualization with/of images
- Reading an image from a file
- cvLoadImage (const char* filename, int iscolor CV_DEFAULT(1));
- Writing an image to a file
- cvSaveImage (const char* filename, const CvArr* image);
It supports several formats: BMP, GIF, JPG, TIFF, etc.
- Open a window for visualization
- cvNamedWindow (const char* name, int flags);
- Visualize the image
- cvShowImage (const char* name, const CvArr* image);
The structure of OpenCV
The ‘cv.h’ contains:
- general image processing functions: filter, color conversion, morphological operators, structural analysis, motion
analysis, pattern recognition (object detection), camera calibration and 3D reconstruction
The ‘cxcore.h’ contains:
- basic structures, arithmetical/logical operators (copy, transformation), dynamic structures (sets, graphs, trees),
drawing functions, error handling and system functions
The ‘cvaux.h’
- stereo correspondence, texture descriptors, 2D-3D trackers, background segmentation, morphing, etc.
The ‘highgui.h’ contains:
- user interface
Installation of IPL/OpenCV libraries and environment settings for MSVC++
In order to keep track with the following settings, OpenCV folder and IPL folder (named ‘plsuite’) must be
installed in C:\Archivos de Programa\Intel\
- in the ‘c’ source file add the header files needed: ‘ipl.h’, ‘cv.h’, ‘cxcore.h’, ‘highgui.h’, ‘cvaux.h’,
‘cvhaartraining.h’
- from the ‘Project’ menu, choose ‘Settings’ and click on the ‘Link Tab’. In the field: ‘Object/library modules’
add the following:
ipl.lib cv.lib cxcore.lib highgui.lib cvaux.lib cvhaartraining.lib (if you use the face detector functions)
- from the menu ‘Tools’ choose ‘Options’ and then click on the ‘Directories’ tab. In the field ‘Show directories for’ choose
‘Include files’, then edit the following paths:
- C:\ARCHIVOS DE PROGRAMA\INTEL\PLSUITE\INCLUDE
- C:\ARCHIVOS DE PROGRAMA\INTEL\OPENCV\CV\INCLUDE
- C:\ARCHIVOS DE PROGRAMA\INTEL\OPENCV\OTHERLIBS\HIGHGUI
- C:\ARCHIVOS DE PROGRAMA\INTEL\OPENCV\CXCORE\INCLUDE
- C:\ ARCHIVOS DE PROGRAMA\INTEL\OPENCV\apps\HaarTraining\include
(if you are using the face detector)
- from the menu ‘Tools’ choose ‘Options’ and then click on the ‘Directories’ tab. In the
field ‘Show directories for’ choose ‘Library files’, then edit the following paths:
- C:\ARCHIVOS DE PROGRAMA\INTEL\PLSUITE\LIB\MSVC
- C:\ARCHIVOS DE PROGRAMA\INTEL\OPENCV\LIB
- the last step consists in adding the paths for DLLs in the ‘Environment Variables’
- from ‘Control Panel’, choose ‘System’, then ‘Advanced’ and finally click on ‘Environment Variable’. From
dialog box that appears, in the ‘System Variables’ section, click and edit the ‘PATH’ item
- you have to add the following route:
C:\Archivos de programa\Intel\OpenCV\bin
C:\Archivos de programa\Intel\plsuite\bin
Examples
Three examples are included (provided in separate files):
- first is about getting a live stream from a webcam (you must make sure you have DirectX library installed
- second is about detecting faces in an image by using the detector which comes with the OpenCV library
- third is about image convolution
More information...
- http://www.cvc.uab.es/~bogdan/CV/cv.html
- IPL/OpenCV online documentation (html, pdf files)
- C:\Archivos de programa\Intel\OpenCV\samples
- http://www.site.uottawa.ca/~laganier/tutorial/opencv+directshow/cvision.htm
- http://groups.yahoo.com/group/OpenCV/ (you have to register)