Images
- photometric aspects of image formation
- gray level images
- linear/nonlinear filtering
- edge detection
- corner detection
CS223, Jana Kosecka
Image
Brightness values
I(x,y)
CS223, Jana Kosecka
Image model
Mathematical tools
Analog intensity function
Analysis
Temporal/spatial sampled function
Linear algebra
Quantization of the gray levels
Numerical methods
Point sets
Set theory, morphology
Random fields
Stochastic methods
List of image features, regions
Geometry, AI, logic
CS223, Jana Kosecka
Basic Photometry
Radiometric model of image formation
CS223, Jana Kosecka
Basic ingredients
Radiance – amount of energy emitted along certain direction
Iradiance – amount of energy received along certain direction
BRDF – bidirectional reflectance distribution –
portion of the energy coming from direction
reflected to direction
Lambertian surfaces – the appearance depends only on radiance, not
on the viewing direction
Image intensity for a Lambertian surface
CS223, Jana Kosecka
Images
• Images contain noise – sources sensor quality, light
fluctuations, quantization effects
CS223, Jana Kosecka
Image Noise Models
• Additive noise: most commonly used
• Multiplicative noise:
• Impulsive noise (salt and pepper):

Iˆ(i, j ) if x  l
I (i, j )  
imin  y (imax  imin ) x  l
• Noise models: gaussian, uniform
• Noise Amount: SNR = s/ n
CS223, Jana Kosecka
Image filtering
• How can we reduce the noise in the image
• Image acquisition noise due to light fluctuations
and sensor noise can be reduced by acquiring a
sequence of images and averaging them
• Computation of simple features
• First stage of visual processing
CS223, Jana Kosecka
Image Processing
1D signal and its sampled version
f = { f(1), f(2), f(3), …, f(n)}
f = {0, 1, 2, 3, 4, 5, 5, 6, 10 }
CS223, Jana Kosecka
Discrete time system
• Maps 1 discrete time signal to another
f[x]
h
g[x]
• Special class of systems – linear , time-invariant systems
Superposition principle
Shift (time) invariant – shift in input causes shift in output
CS223, Jana Kosecka
Convolution sum:
unit impluse – if x = 0 it’s 1 and zero everywhere else
Every discrete time signal can be written as a sum of scaled
and shifted impulses
The output the linear system is related to the input and the transfer
function via convolution
f
h
g
f
Convolution sum:
Notation:
CS223, Jana Kosecka
filter
g
Averaging filter
Original image
Smoothed image
CS223, Jana Kosecka
Averaging filter
and 0 everywhere else
Box filter
Ex. cont.
Averaging filter center pixel weighted more
CS223, Jana Kosecka
Convolution in 2D
f
9 10 11 9 99 11
10 9 9 11 10 10
1/9
X X X X X
X
X 10
X
X
X
X
X
X
X X X X X X
X
10 11 10 0 0 1
9 10 11 1 0 1
10 9 10 0 2 1
11 10 9 10 9 11
g
h
1
1
1
1
1
1
1
1
1
1/9.(10x1 + 11x1 + 10x1 + 9x1 + 10x1 + 11x1 + 10x1 + 9x1 + 10x1) =
CS223, Jana Kosecka
1/9.( 90) = 10
Example:
10 11 10 0 0
9 10 11 1 0
I
O
10 9 10 0 2 1
11 10 9 10 9 11
9 10 11 9 99 11
10 9
9 11 10 10
X X X X X
X 10 7 4 1 X
X
1
1
F
1
1/9 1
1
1
1
1
1
1
1
X
X
X
X
X
X
X
X
X X X X
1/9.(10x1 + 0x1 + 0x1 + 11x1 + 1x1 + 0x1 + 10x1 + 0x1 + 2x1) =
1/9.( 34) = 3.7778
CS223, Jana Kosecka
Example:
10 11 10 0 0
9 10 11 1 0
I
O
10 9 10 0 2 1
11 10 9 10 9 11
9 10 11 9 99 11
10 9
9 11 10 10
X X X X X
X 10 7 4 1 X
X
1
1
F
1
1/9 1
1
1
1
1
1
1
1
X
X
X
X
X
X
X
20 X
X X X X
1/9.(10x1 + 9x1 + 11x1 + 9x1 + 99x1 + 11x1 + 11x1 + 10x1 + 10x1) =
1/9.( 180) = 20
CS223, Jana Kosecka
How big should the mask be?
• The bigger the mask,
– more neighbors contribute
– bigger noise spread.
– more blurring.
– more expensive to compute
Limitations of averaging
• Signal frequencies shared with noise are lost
• Impulsive noise is diffused but not removed
CS223, Jana Kosecka
Gaussian Filter
• Better option for blurring
– The coefficients are samples of a 1D Gaussian.
– Gives more weight at the central pixel and less weights to
the neighbors.
– The further away the neighbors, the smaller the weight.
– Gaussian filter
Samples from the continuous Gaussian
• Gaussian filter is the only one that has the same shape
in the space and frequency domains.
• There are no secondary lobes – i.e. a truly low-pass filter
CS223, Jana Kosecka
How big should the mask be?
• The std. dev of the Gaussian  determines the amount of
smoothing.
• The samples should adequately represent a Gaussian
• For a 98.76% of the area, we need
m = 5
5.(1/)  2    0.796, m 5
5-tap filter
g[x] = [0.136, 0.6065, 1.00, 0.606, 0.136]
CS223, Jana Kosecka
Image Smoothing
• Convolution with a 2D Gaussian filter
• Gaussian filter is separable, convolution can be
accomplished as two 1-D convolutions
CS223, Jana Kosecka
Non-linear Filtering
• Replace each pixel with the MEDIAN value of all the pixels
in the neighborhood
• Non-linear
• Does not spread the noise
• Can remove spike noise
• Expensive to run
CS223, Jana Kosecka
Example:
10 11 10 0 0
9 10 11 1 0
I
10
11
9
10
9
10
10
9
10 0
9 10
11 9
9 11
1
1
O
2 1
9 11
99 11
10 10
X
X X
X 10
X X
X
X
X
X
X
X
X
X X X X
X
median
10,11,10,9,10,11,10,9,10
sort
9,9,10,10,10,10,10,11,11
CS223, Jana Kosecka
X
X
Example:
10 11 10 0 0
9 10 11 1 0
I
10
11
9
10
9
10
10
9
10 0
9 10
11 9
9 11
1
1
X X X X X
1 X
X 10 10 1
X
O
2 1
9 11
99 11
10 10
X
X
X
X
X
X
X
X
X X X X
median
11,10,0,10,11,1,9,10,0
sort
0,0,1,9,10,10,10,11,11
CS223, Jana Kosecka
Example:
10 11 10 0 0
9 10 11 1 0
I
10
11
9
10
9
10
10
9
10 0
9 10
11 9
9 11
1
1
O
2 1
9 11
99 11
10 10
X
X X
X 10
X X
X
X
X
X
X
9
X
10 X
X X X X
X
median
10,9,11,9,99,11,11,10,10
sort
9,9,10,10,10,11,11,11,99
CS223, Jana Kosecka
X
X
Image Features
Local, meaningful, detectable parts of the image.
We will look at edges and corners
CS223, Jana Kosecka
Image Features – Edges, Corners
• Look for detectable, meaningful parts of the image
• Edges are detected at places where the image values exhibit
sharp variation
Gray value
column
Edge
column
Edge
CS223, Jana Kosecka
Edge detection (1D)
F(x)
Edge= sharp variation
F ’(x)
x
Large first derivative
x
CS223, Jana Kosecka
Digital Approximation of 1st derivatives
df ( x)
f ( x  x)  f ( x)
 lim
x  0
dx
x
f(x)
X-1
df ( x) f ( x  1)  f ( x  1)

dx
2
Convolve with:
-1
0
CS223, Jana Kosecka
1
X
X+1
Edge Detection (2D)
Vertical Edges:
Convolve with:
-1
0
Horizontal Edges:
-1
Convolve with:
0
1
CS223, Jana Kosecka
1
Noise cleaning and Edge Detection
I(x,y)
Noise
Filter
Edge
Detection
We need to also deal with noise
-> Combine Linear Filters
CS223, Jana Kosecka
E(x,y)
Convolve with:
-1
0
1
-1
0
1
-1
0
1
Noise Smoothing
Noise Smoothing & Edge Detection
Vertical Edge Detection
This mask is called the (vertical) Prewitt Edge Detector
Outer product of box filter [1 1 1]T and [-1 0 1]
CS223, Jana Kosecka
Convolve with:
-1
-1
-1
0
0
0
1
1
1
Noise Smoothing
Horizontal Edge Detection
Noise Smoothing & Edge Detection
This mask is called the (horizontal) Prewitt Edge Detector
CS223, Jana Kosecka
Sobel Edge Detector
Convolve with:
and
-1
0
1
-2
0
2
-1
0
1
-1
-2
-1
0
0
0
1
2
1
CS223, Jana Kosecka
Gives more weight
to the center pixels
Example
0 0 0 0
0
0 0 0 0
0
0 0 50 50 50
Ix 
0 0 50 50 50
0 0 50 50 50
Iy 
-1
-1
-1
0
0
0
1
1
1
0 0 0
0 0
0 50 50 0 0
0 100 100 0 0
0 150 150 0 0
0 150 150 0 0
CS223, Jana Kosecka
0 0 0 0 0
0 50 100 150 150
0 50 100 150 150
0 0 0 0 0
0 0 0 0 0
-1 -1 -1
0
1
0
1
0
1
Image Derivatives
We know better alternative to smoothing
Smooth using Gaussian filter
g(x) is a 1-D Gaussian filter, g(x,y) – 2-D Gaussian filter
Taking a derivative – linear operation (take the
derivative of the filter)
CS223, Jana Kosecka
Gaussian and its derivative
CS223, Jana Kosecka
Vertical edges
CS223, Jana Kosecka
First derivative
Gradient Magnitude
Horizontal edges
• Image Gradient
CS223, Jana Kosecka
Gradient Orientation
CS223, Jana Kosecka
Orientation histogram
CS223, Jana Kosecka
Corner detection
• A point on a line is hard to match.
Intuition:
• Right at corner, gradient is ill defined.
• Near corner, gradient has two different values.
CS223, Jana Kosecka
Formula for Finding Corners
We look at matrix:
Gradient with respect to x, times
gradient with respect to y
Sum over a small region, the
hypothetical corner
 I
C
 I x I y
2
x
Matrix is symmetric
I I
I
CS223, Jana Kosecka
x y
2
y



First, consider case where:
 I
C
 I x I y
2
x
I I
I
x y
2
y
 1 0 


  0 2 
This means all gradients in neighborhood are:
(k,0) or (0, c) or
(0, 0) (or off-diagonals cancel).
What is region like if:
1.
?
2.
?
3.
and
?
4.
and
?
CS223, Jana Kosecka
General Case:
In general case from linear algebra, it follows that
because C is symmetric:


1
1
CR 
0
0
R

2 
With R being a rotation matrix. So every case can
be intrepreted like one on last slide.
CS223, Jana Kosecka
Corner Detection
•
•
•
•
•
•
Filter image.
Compute magnitude of the gradient everywhere.
We construct C in a window.
Compute eigenvalues 1 and 2.
If they are both big, we have a corner
Or if smalest eigenvalue  of C is bigger than  - mark pixel
as candidate feature point
CS223, Jana Kosecka
Point Feature Extraction
Harris Corner Detector
• Alternatively feature quality function (Harris Corner Detector)
CS223, Jana Kosecka
Harris Corner Detector - Example
CS223, Jana Kosecka
Edge Detection
gradient magnitude
original image
• Compute image derivatives
• if gradient magnitude >  and the value is a local max. along gradient
direction – pixel is an edge candidate
• how to detect one pixel thin edges ?
CS223, Jana Kosecka
Canny Edge Detector
• The magnitude image Es has the magnitudes of the smoothed
gradient.
• Sigma determines the amount of smoothing.
• Es has large values at edges:
– Find local maxima
Th
CS223, Jana Kosecka
Nonmaximum supression
•
•
The inputs are Es & Eo Magnitude and orientation
Consider 4 directions D={ 0,45,90,135} wrt x
•
For each pixel (i,j) do:
1.
2.
•
Find the direction dD s.t. d Eo(i,j) (normal to the edge)
If {Es(i,j) is smaller than at least one of its neigh. along d}
•
IN(i,j)=0
• Otherwise, IN(i,j)= Es(i,j)
The output is the thinned edge image IN
x
CS223, Jana Kosecka
x
Thresholding
• Edges are found by thresholding the output of
NONMAX_SUPRESSION
• If the threshold is too high:
– Very few (none) edges
• High MISDETECTIONS, many gaps
• If the threshold is too low:
– Too many (all pixels) edges
• High FALSE POSITIVES, many extra edges
CS223, Jana Kosecka
Hysteresis Thresholding
Es(i,j)<H
Es(i,j)>L
Es(i,j)> H
Es(i,j)<L
Es(i,j)>L
Strong edges reinforce adjacent weak edges
CS223, Jana Kosecka
Finding lines in an image
• Option 1:
– Search for the line at every possible
position/orientation
– What is the cost of this operation?
• Option 2:
– Use a voting scheme: Hough transform
CS223, Jana Kosecka
Other edge detectors - second-order
derivative filters (1D)
• Peaks of the first-derivative of the input signal,
correspond to “zero-crossings” of the secondderivative of the input signal.
F(x)
F’’(x)
F ’(x)
x
CS223, Jana Kosecka
Edge Detection (2D)
y
1D
F(x)
I(x)
dI(x)
dx
d2I(x)
dx2
2D
x
> Th
=0
x
I(x,y)
|I(x,y)| =(Ix 2(x,y) + Iy2(x,y))1/2 > Th
tan  = Ix(x,y)/ Iy(x,y)
2I(x,y) =Ix x (x,y) + Iyy (x,y)=0
Laplacian
CS223, Jana Kosecka
Notes about the Laplacian:
•
2I(x,y) is a SCALAR
–  Can be found using a SINGLE mask
–  Orientation information is lost
• 2I(x,y) is the sum of SECOND-order derivatives
– But taking derivatives increases noise
– Very noise sensitive!
• It is always combined with a smoothing operation:
• Filter Laplacian of Gaussian LOG filter
I(x,y)
Smooth
Laplacian
CS223, Jana Kosecka
O(x,y)
g ( x)  e

x
1D Gaussian
2
2 2
g ' ( x)  

1
2 xe
x2
2 2

2
x2
2
x
1  2 2
g ' ' ( x)  ( 3  )e
 
2
x

2

e
x2
2 2
2D
“Mexican Hat”
CS223, Jana Kosecka
Derivative of Gaussian Filters
Measure the image gradient and its direction at different
scales (use a pyramid).
CS223, Jana Kosecka
The Laplacian Pyramid
• Building a Laplacian pyramid:
– Create a Gaussian pyramid
– Take the difference between one Gaussian pyramid level
and the next (before subsampling)
• Properties
– Also known as the difference-of-Gaussian function, which
is a close approximation to the Laplacian
– It is a band pass filter - each level represents a
different band of spatial frequencies
• Reconstructing the original image:
– Reconstruct the Gaussian pyramid starting at top layer
CS223, Jana Kosecka
CS223, Jana Kosecka
Gaussian
pyramid
Laplacian
Pyramid
(note top image
is from Gaussian)
CS223, Jana Kosecka
Add more oriented filters
(Malik & Perona, 1990)
CS223, Jana Kosecka
Alternative: Gabor filters
Gabor filters: Product of a Gaussian
with sine or cosine
Top row shows anti-symmetric
(or odd) filters, bottom row the
symmetric (or even) filters.
No obvious advantage to any one type
of oriented filters.
CS223, Jana Kosecka
An edge is not a line...
• How can we detect lines ?
CS223, Jana Kosecka