OpenCV Introduction
Hang Xiao
Oct 26, 2012
History

1999 Jan : lanched by Intel, real time machine vision library for UI, optimized code for
intel

2000 Jun : OpenCV alpha 3。

2000 Dec : OpenCV beta 1 for linux

2006 : the first 1.0 version supports Mac OS

2008 mid : obtain corporate support from Willow Garage

2009 Sep : OpenCV 1.2（beta2.0

2009 Oct : Version 2.0 released。

2010 Dec : OpenCV 2.2。

2011 Aug : OpenCV 2.3。

2012 Apr : OpenCV 2.4.
Overview
 Goals
 Develop a universal toolbox for research and development in the field
of Computer Vision
 Algorithms
 More than 350 algorithms, 500 API
 Programming language
 C/C++, C#, Ch , Python, Ruby, Matlab, and Java (using JavaCV)
 OS support
 Windows, Android, Maemo, FreeBSD, OpenBSD, iOS, Linux and Mac
OS.
 Licence
 BSDlisence, free for commercial and non-commmercial
Overview - Applications
 2D and 3D feature toolkits
 Egomotion estimation
 Facial recognition system
 Gesture recognition
 Human–computer interaction (HCI)
 Mobile robotics
 Motion understanding
 Object identification
 Segmentation and Recognition
 Stereopsis Stereo vision: depth perception from 2 cameras
 Structure from motion (SFM)Motion tracking
Overview - A statistical machine
learning library
 Boosting (meta-algorithm)
 Decision tree learning
 Gradient boosting trees
 Expectation-maximization algorithm
 k-nearest neighbor algorithm
 Naive Bayes classifier
 Artificial neural networks
 Random forest
 Support vector machine (SVM)
Outline
 Image Analysis
 Structural Analysis
 Object Recognition
 Motion Analysis and Object Tracking
 3D Reconstruction
Outline
 Image Analysis
 Structural Analysis
 Object Recognition
 Motion Analysis and Object Tracking
 3D Reconstruction
Image Analysis
 Thresholds
 Statistics
 Pyramids
 Morphology
 Distance transform
 Flood fill
 Feature detection
 Contours retrieving
Image Thresholding
 Fixed threshold;
 Adaptive
threshold;
Image Thresholding Examples
Source picture
Fixed threshold
Adaptive threshold
Statistics
min, max, mean value, standard
deviation over the image
Norms C, L1, L2
Multidimensional histograms
Spatial moments up to order 3 (central,
normalized, Hu)
Multidimensional Histograms
 Histogram operations
 calculation, normalization, comparison, back project
 Histograms types:
 Dense histograms
 Signatures (balanced tree)
 EMD algorithm
 The EMD computes the distance between two distributions, which
are represented by signatures.
 The signatures are sets of weighted features that capture the
distributions. The features can be of any type and in any number of
dimensions, and are defined by the user.
 The EMD is defined as the minimum amount of work needed to
change one signature into the other
EMD – a method for the
histograms comparison
pi  P, 1  i  P , q j  Q, 1  j  Q ,  two historam s
EMD( P, Q) 
f
i, j
ij
 d ( pi , q j )
f
,
ij
i, j
f ij  weightcoefficients,
d ( pi , q j )  the distancebetween the elem ents pi and q j .
Image Pyramids
Gaussian and
Laplacian pyramids
Image
segmentation by
pyramids
Image Pyramids
Gaussian and Laplacian
Pyramid-based color
segmentation
On still pictures
And on movies
Morphological Operations
 Two basic morphology operations using structuring element:

erosion

dilation
 More complex morphology operations:
 opening : erosion + dilation
 closing : dilation + erosion
 morphological gradient : the difference between the dilation and the erosion of an
image
 top hat : the difference between an input image and its opening
 black hat : the difference between the closing and its input image
Opening
Closing
Top Hat
Morphological Gradient
Black Hat
Morphological Operations Examples
 Morphology - applying Min-Max. Filters and its combinations
Image
I
Erosion IB
Closing I•B= (IB)B Grad(I)= (IB)-(IB)
Dilatation IB
Opening IoB= (IB)B
TopHat(I)= I - (IB) BlackHat(I)= (IB) - I
Distance Transform
 Calculate the distance for all non-feature points to
the closest feature point
 Two-pass algorithm, 3x3 and 5x5 masks, various
metrics predefined
Flood Filling
Simple
Gradient
Feature Detection
 Fixed filters (Sobel operator, Laplacian);
 Optimal filter kernels with floating point
coefficients (first, second derivatives, Laplacian)
 Special feature detection (corners)
 Canny operator
 Hough transform (find lines and line segments)
 Gradient runs
Canny Edge Detector
Hough Transform
Detects lines in a binary image
•Standard Hough
Transform
•Probabilistic Hough
Transform
Another Sample of the
Hough Transform Using
Source picture
Result
Contour Retrieving
 The contour representation:
 Chain code (Freeman code)
 Polygonal representation
Initial Point
Chain code for the curve:
34445670007654443
Contour representation
Hierarchical representation of
contours
Image Boundary
(W1)
(B2)
(W2)
(B3)
(W3)
(B4)
(W5)
(W6)
Contours Examples
Source Picture
(300x600 = 180000 pts total)
Retrieved Contours
(<1800 pts total)
After Approximation
(<180 pts total)
And it is rather fast: ~70 FPS for 640x480 on complex scenes
Outline
 Image Analysis
 Structural Analysis
 Object Recognition
 Motion Analysis and Object Tracking
 3D Reconstruction
Structural Analysis
Contours processing
 Approximation
 Hierarchical representation
 Shape characteristics
 Matching
Geometry
 Contour properties
 Fitting with primitives
 PGH: pair-wise geometrical histogram for the
contour.
Contour Processing
 Approximation:
 RLE algorithm (chain code)
 Teh-Chin approximation (polygonal)
 Douglas-Peucker approximation (polygonal);
 Contour moments (central and normalized up to order 3)
 Hierarchical representation of contours
 Matching of contours
Hierarchical Representation
of Contours
 A contour is represented with a binary tree
 Given the binary tree, the contour can be retrieved with arbitrary precision
 The binary tree is quasi invariant to translations, rotations and scaling
Contours matching
Matching based on hierarchical
representation of contours
Geometry
 Properties of contours: (perimeter, area,
convex hull, convexity defects, rectangle of
minimum area)
 Fitting: (2D line, 3D line, circle, ellipse)
 Pair-wise geometrical histogram
Pair-wise geometrical
histogram (PGH)
p (i , j )
f PGH  [ E r (1), E r ( 2), E r ( N ), Ec (1), Ec ( 2), Ec ( M )] T ,
E r (i ) 
 j  p(i, j) /  p(i, j),
j
Ec ( j ) 
j
 i  p(i, j) /  p(i, j).
i
i
Outline
 Image Analysis
 Structural Analysis
 Object Recognition
 Motion Analysis and Object Tracking
 3D Reconstruction
Object Recognition
 Eigen objects
 Hidden Markov Models
Eigen Objects
u i  {u1i , u2i , uni }  R n , 1  i  m, m  n,
C  {cij }  cov ariance matrix,
cij 

m
(uli  ul )  (ulj  ul ), ul  1 / m
l
eli  1 / i


ulk ,
k 1
vki  (ulk  ul ),
k
e i  {e1i , e1i ,, eni }, i  1, m1  m  eigen basis,
i and  i  { 1i ,  2i ,, mi }  eigen values and eigen vectors.
.
Eigen objects (continued)
Hidden Markov Model
Definitions
S  {s1, s2 ,...,sN }
- The set of states
O  {o1, o2 ,...,oM }
- The set of measurements
qt  O
- The state at time t
A  {aij \ aij  P(qt  s j | qt 1  si )}
B  {bij \ bij  P(oi | s j )}
  { i \  i  P(q0  si )}
- The transition probability matrix
- The conditional probability
matrix
- The starting states distribution
Embedded
HMM for Face
Recognition
Model-
- Face ROI partition
Face recognition
using Hidden Markov Models
One person – one HMM
 Stage 1 – Train every HMM

1
…

n
Stage 2 – Recognition
i
Pi
- probability
Choose max(Pi)
Outline
 Image Analysis
 Structural Analysis
 Object Recognition
 Motion Analysis and Object Tracking
 3D Reconstruction
Motion Analysis and
Object Tracking
 Background subtraction
 Motion templates
 Optical flow
 Active contours
 Estimators
Background Subtraction
 Background model (normal distribution)
 Background statistics functions:
 Average
 Standard deviation
 Running average
Motion Templates
 Object silhouette
 Motion history images
 Motion history gradients
 Motion segmentation algorithm
silhouette
MHI
MHG
Motion Segmentation
Algorithm
 Two-pass algorithm labeling all motion segments
Motion Templates Example
 Motion templates allow
to retrieve the dynamic
characteristics of the
moving object
Optical Flow
 Block matching technique
 Horn & Schunck technique
 Lucas & Kanade technique
 Pyramidal LK algorithm
 6DOF (6 degree of freedom) algorithm
Optical flow equations:
I ( x  dx, y  dy, t  dt)  I ( x, y , t );
 I / t  I / x  ( dx / dt)  I / y  ( dy / dt);
G  X  b,
X  (x, y ), G 

 I x2 , I x I y 

, b 
 I x I y , I y2 



I x 
It  
I y 
Pyramidal Implementation of the
optical flow algorithm
Image Pyramid
Representation
J image
I image
Iterative Lucas – Kanade
Scheme
Location of point u on image uL=u/2L
Spatial gradient matrix
G

 I x2 , I x I y 


 I x I y , I y2 


Standard Lucas – Kanade scheme for
optical flow computation at level L dL
Generic Image
Guess for next pyramid level L – 1
g L1  2( g L  d L )
Finally,
V U  d
(L-1)-th Level
Image pyramid building
L-th Level
d  d 0  g0
Optical flow computation
6DOF Algorithm
Parametrical optical flow equations:
X   (s ).
I  I / s  I / X   / s
N

i 1 ROI
N
I iT  I i  ds 

i 1 ROI
I t  I iT
i
Active Contours
 Snake energy:
 Internal energy:
 External energy:
 Two external energy types:
E  Eint  Eext
Eint  Econt  Ecurv
Eext  Eimg  Econ
Eimg   I ,
Eimg   grad( I ) ,
E    Econt    Ecurv    Eimg  min
Estimators
Kalman filter
ConDensation filter
Kalman object tracker
Outline
 Image Analysis
 Structural Analysis
 Object Recognition
 Motion Analysis and Object Tracking
 3D Reconstruction
3D reconstruction
 Camera Calibration
 View Morphing
 POSIT
Camera Calibration
 Define intrinsic and extrinsic camera parameters.
 Define Distortion parameters
p  A[ RT ]P,
 f x 0 cx 
 r11 r12 r13 
t1 


A  0 f y c y , R   r21 r22 r23 , T  t2 , P  [ X , Y , Z ], p  [u, v ]
0 0 1 
 r31 r32 r33 
t3 


u~  u  (u  c x )  [k1  r 2  k2  r 4  2 p1 y  p2 ( r 2 / x  2 x )],
v~  v  (v  c )  [k  r 2  k  r 4  2 p x  p ( r 2 / y  2 y )],
y
r2  x2  y2.
1
2
2
1
Camera Calibration
Now, camera calibration can be done by holding
checkerboard in front of the camera for a few seconds.
And after that you’ll get:
3D view of etalon
Un-distorted image
View Morphing
POSIT Algorithm
Perspective projection:

xi  ( f / Zi )  X i , yi  ( f / Zi )  Yi
Weak-perspective projection:
xi  s  X i , yi  s  Yi , s  f / Z .
OpenCV Websites
 http://opencv.org
OpenCV official webpage.
 http://opencvlibrary.sourceforge.net/
OpenCV documentation and FAQs.
OpenCV Examples

adaptiveskindetector : detect skin area

fback_c : dense Franeback optical flow

contours : calculate contours on different levels

delaunay : delaunay triangle

find_obj : SURF Detector and Descriptor using either FLANN or brute force
matching on planar objects

morphology : open/close, erode/dilate

motempl ：motion templates

mser_sample : Maximal Extremal Region interest point detector

polar_transforms : illustrates Linear-Polar and Log-Polar image transforms

pyramid_segmentation : color pyramid segmentation
Thanks !!!