Ekstrahering og repræsentation
af features
MM2: Convolution and
filtering of images
1
Agenda
• Point processing
– Image computing
• Neighborhood processing
– Convolution
• Applications
2
Fundamental Steps in Computer Vision
Today:
• Point processing
–
Point 1: 22,33
Point 2: 24, 39
…..
Segmentation
•
•
Filtering
Convolution
– Method
Filtering
– Application
Representation
and description
Actor sitting
Preprocessing
Knowledge base
Problem
domain Image
acquisition
Recognition
and
Interpretation
3
Result
Point processing
• Only one pixel in the input has an effect on the output
• For example:
– Changing the brightness, thresholding, histogram stretching
Input
1
2
1
1
2
2
1
0
2
5
0 1 3
4 2 2
1 0 1
1 0 2
3 1 2
Output
12
9
4
Image Computing
•
•
•
•
Arithmetic: An image calculator: + - / etc.
Logic operations (OR, AND, etc. )
Image computation is done pixel-wise
For example:
a1 a2 …
a3 a4 …
… … …
a1 a2 …
a3 a4 …
… … …
b1 b2 …
+ b3 b4 …
… … …
+
10
=
=
a1
+b1
a3
+b3
a2
+b2
a4
+b4
a1
+10
a3
+10
a2
+10
a4
+10
5
Image Computing
• The size of the output image
+
=
• Sometimes a program requires the images
to have the same sizes
6
Image Computing: Caution!!
• Undesired effects may happen, when computing with
images: Overflow / Underflow
• Example: Two Gray value images I1, I2 with pixel
ranges: [0,255]. Their Pixel sum may be larger than
255!! For example: 124+202=326!
• Subtraction results may be smaller than 0!!
For example: 124-202=-78!!
7
Solution to the
Overflow/Underflow Problem
•
Use an intermediate image:
–
–
Pixel values are float values (32 bits/ 4 bytes)
Can store almost any number
1. Write computation results into intermediate
image
2. Rescale intermediate image to values 0…255
and write results into normal Gray value image.
8
Gray Value Scaling and Shifting
Intermediate
float image
Gray value
image
… -1 … 0
0
1
… …
… 1
…
255 256
… … … …
min
…
max
…
9
Different operations
• Monadic (one image one
constant)
–
–
–
–
–
–
–
–
–
–
Add
Sub
Mul
Div
And
Or
Xor
Min
Max
xxx…
Im + 17
Im - 17
Im * 17
Im / 17
Im AND 17
Im OR 17
Im XOR 17
Min(Im,17)
Max(Im,17)
• Dynadic (two images)
–
–
–
–
–
–
–
–
–
–
Add
Sub
Mul
Div
And
Or
Xor
Min
Max
xxx…
A+B
A-B
A*B
A/B
A AND B
A OR B
A XOR B
Min(A,B)
Max(A,B)
10
In ImageJ
• Involving two images:
• Process => Image Calculator
+
=
• Involving an image and a number:
• Process => Math
+ 10 =
11
Measure noise in your setup
• Subtract two images of the same scene at
different time instances => Noise in scene
• Help you to judge the nature of the noise
After
Before
-
-
Noise
=
=
12
Image averaging: remove noise
• Assume noise has zero mean
• The noise tends to cancel
out if we average a number
of images
• Example: real value: 100.
• Value in different images:
103, 109, 98, 87, 95, 112.
• Average = (I1 + I2 + I3 + I4 +
I5 + I6 ) / 6 = 104
• The more images the better
• Requirement: nothing is
moving (changing) in the
scene!
Original
Noise added
8
16
64
128
13
Segmentation of the object
• Background subtraction
• Image containing object - Background image = Object
Image w. Obj.
Object
Background
-
-
=
=
14
• Advanced background subtraction: used avg. bg. image
Combine two images
• Used when adding text etc. to an image
– Mixing two images by averaging
– But what if we want to put more weight on one image?
– Alpha blending (or weighted averaging):
g ( x, y)    f1 ( x, y)  (1   )  f 2 ( x, y)
–  [0,1]
– If   0.5 then this operation is standard averaging
– Many programs (CV+CG) support alpha blending
• Image representation: RGBA (4 bytes) per pixel
15
Neighborhood processing
16
Neighborhood processing
• Several pixels in the input has an effect on
the output
Input
1
2
1
1
2
2
1
0
2
5
0 1 3
4 2 2
1 0 1
1 0 2
3 1 2
Output
12
9
17
Convolution
18
Convolution (1D)
Filter coefficients
(mask, kernel, template, window)
Filter
Input Signal/Image-row
1 2 1
1 1 2 2 1 1 2 2 1 1
Output Signal/Image-row
Filter Response
5
4
19
Normalise filter response
Max values in image
Filter coefficients
…
255 255
A
B
255
…
C
Max filter response = A 255 + B 255 + C 255 <=>
= 255 (A+B+C)
If max filter response = 255 (one byte) then
New filter response = filter response / (A+B+C)
20
Convolution (1D)
1 2 1
1 1 2 2 1 1 2 2 1 1
5
4
7
4
21
Convolution (1D)
This process is called
Convolution!!
1 2 1
1 1 2 2 1 1 2 2 1 1
5
4
7
4
7
4
5
4
5
4
7
4
7
4
5
4
22
Math of convolution
n
g ( x )  h * f ( x )   h( n) f ( x  n)
n
g(x): output, h: filter, * means convolution,
f(x): input, n = |_ width of filter / 2 _|
|_ _|: rounds down, for example: |_ 1.7_| = 1
For example: Filter (h):
width = 3 => n=1
1
2
h(-1)=1 h(0)=2
1
h(1)=1
23
Correlation (1D)
Filter coefficients
(mask, kernel, template, window)
Filter
Input Signal/Image-row
1 2 1
1 1 2 2 1 1 2 2 1 1
Output Signal/Image-row
Filter Response
5
4
24
Correlation versus Convolution
1 2 1
Correlation
n
g ( x )  h  f ( x )   h( n) f ( x  n)
n
1 1 2 2 1 1 2 2 1 1
1 2 1
Convolution
n
g ( x )  h * f ( x )   h( n) f ( x  n)
n
1 1 2 2 1 1 2 2 1 1
In image processing we use CORRELATION
but (nearly) always call it CONVOLUTION!!!!!
Note: When the filter is symmetric: correlation = convolution!
25
Convolution/correlation on images
• The filter is now 2D
• Kernel (mask), kernel coefficients
• Size: 3x3, 5x5, 7x7, ….
1
9
Input
1
2
1
1
2
2
1
0
2
5
0 1 3
4 2 2
1 0 1
1 0 2
3 1 2
Output
1 1 1
1 1 1
1 1 1
12
9
26
Math. of 2D
Convolution/Correlation
n
Convolution
m
g ( x, y )  h * f ( x, y )   h(m, n) f ( x  m, y  n)
n m
Correlation
n
m
g ( x, y )  h  f ( x, y )   h(m, n) f ( x  m, y  n)
n m
27
Problems at the borders
• Why is the output image smaller than the input?
– We are lacking information
• The bigger the kernel the bigger the problem
• Does it matter? Yes, if we are going to combine the images afterwards
1
2
1
Input
1
2
2
1
0
2
5
0
4
1
1
3
1
2
0
0
1
3
2
1
2
2
Output
12
9
11
9
14
9
13
9
11
9
13
9
16
9
12
9
11
9
28
Problems at the borders
• Solutions
– Add a value: 0, 255, neighbor (input/output)
• Change histogram, very different value, new pattern, etc.
– Truncate kernel: 3x3 => for example 2x3:
• Complex and not well-defined
1
2
1
Input
1
2
2
1
0
2
5
0
4
1
1
3
1
2
0
0
1
3
2
1
2
2
Output
12
9
11
9
14
9
13
9
11
9
13
9
16
9
12
9
11
9
29
Problems at the borders
• Solutions:
Circular indexing
– Complex and perhaps wrong
2 1 1
2
1
Input
1
2
1
2
2
1
0
2
5
Reflected (mirrored)
0
4
1
1
3
1
2
0
0
1
3
2
1
2
2
3 7 1
2
1
Input
1
2
4
2
1
0
2
5
5
3
2
0
4
1
1
3
2
1
2
0
0
1
1
3
2
1
2
2
1 30
7
4
3
3
2
1
Applications of
convolution/correlation
• Many many operations defined by the
programmer…. and some standard operations
–
–
–
–
–
Object detection
Smoothing filters
Edge detection
Morphology (next time)
….
31
Simple Object Detection
• Finding a specific object in the image
• 1D example: An object is given
(known) as an image, e.g.,
30 60 30
• Task: Find this object in an image:
Input
Output
20 25 30 60 30 20 40 60 10 2
3000 4350 5400 4200 4300 4800 5100 2460
32
For images this is called corelation or template matching!
Template Matching
• The filter is called a template or a mask
Input image
Output
Template
• The brighter the value in the output, the better the
match
33
Smoothing filters
• Also know as: Smoothing kernel, Mean filter,
Low pass filter
1 1 1
• The simplest filter:
1
– Spatial low pass filter
1
1 1 1
3
• Another mask:
– Gaussian filter:
1
1 2 1
4
1
1
1
1
1
1
1
1
2
16
1
2
4
2
1
2
1
9
34
Applications of smoothing
• Blurring to remove
identity or other details
• Degree of blurring = kernel size
• De-blurring / sharpening
g ( x, y)  f ( x, y)  f '' ( x, y)
• (show: camera, 3 x smooth, sharpen)
35
Applications of smoothing
• Preprocessing: enhance objects
• Smooth + Thresholding
• Remove background: g(x,y) = f(x,y) – f(x,y)
36
• (show: boat, 25mean)
Applications of smoothing
• Remove noise
37
Rank Filters
• Aka: order-statistics filters
• Not based on convolution but still
neighborhood processing
• Principle:
– Define a mask, e.g., 3x3
– Sort all pixel-values within the mask into
ascending order
– Select a pixel-value according to the filter type:
Median, min., max., range, …
38
Median Filter
• For an image, mask symmetric: 3x3, 5x5, etc.
Sorted: 0,0,1,1,1,2,2,2,4
Input
1
2
1
1
2
2
2
0
2
5
0 1 3
4 2 2
1 0 1
1 0 2
3 1 2
Output
1
39
Median Filter
• Median Filter
– Good for cleaning salt-and-pepper noise Better than the
mean filter as blurring is minimized and edges stay sharp
40
Edge Detection
Powerfull application of convolution!
41
Edge detection
• What are edges?
• Why are they interesting?
• How do we find them?
–
–
–
–
Prewitt
Sobel
Laplacian
Canny
42
What are edges?
• Local intensity change
• Strong edge = the steep areas in a 3D plot (show: blobs)
Edge detection
43
Why are they interesting?
• Edges can (many times) represent the
information in the image (the objects)
• A higher level of abstraction (less data to process!)
• Edges are features:
– “Independent” from illumination. As opposed to e.g., color
information
• Object recognition and detection (many times) use
edge features
– This is true for computer vision as well as for biological
vision systems!
44
Edges detectors
• Three steps:
– Noise reduction
– Edge enhancement
– Edge localisation
45
Edges detectors
• Three steps:
– Noise reduction
• E.g., median filter
• E.g., mean filter
• Dilemma:
–
–
–
–
Large filter => remove noise 
Large filter => remove edges 
Small filter => keep edges 
Small filter => keep noise 
– Edge enhancement
– Edge localisation
46
Edges detectors
• Three steps:
– Noise reduction
– Edge enhancement
• Calculate candidates for the edges
– Edge localisation
• Decide which edge candidates to keep
47
Edges detectors
• We’ll look at four methods (but others exist!)
• With respect to complexity (simplest first):
–
–
–
–
Prewitt
Sobel
Laplacian
Canny
48
Edge Detectors: Prewitt and Sobel
49
Edge Detectors: Prewitt and Sobel
• Simple to implement. Fast
• Based on the grey-level gradient
– A measure of the steepness of the “image-landscape”
– Calculated for each pixel => Gradient image: g(x,y)
– Use a 16-bit or 32-bit image to represent the gradients!
• The gradient is the first-order derivative: f ( x, y)  g ( x, y)
• Approximated in the x- and y-direction by:
g x ( x, y)  f ( x  1, y)  f ( x  1, y)
g y ( x, y)  f ( x, y  1)  f ( x, y  1)
x-1 x x+1
y-1
y
y+1
50
Edge Detectors: Prewitt and Sobel
g x ( x, y)  f ( x  1, y)  f ( x  1, y)
g y ( x, y)  f ( x, y  1)  f ( x, y  1)
• Gradient vector:

g  [gx , gy]
2
2
g

g

g
– Magnitude:
x
y  gx  g y
51
Edge Detectors: Prewitt and Sobel
• Noise reduction
– Mean of three gradients
• Edge enhancement
g x ( x, y)  f ( x  1, y)  f ( x  1, y)
g y ( x, y)  f ( x, y  1)  f ( x, y  1)
– Implementation of:
– Convolution with Prewitt kernels or Sobel kernels
• Edge localisation
– If the magnitude: g(x,y) > TH => edge found: e(x,y)=255
– Else: no edge: e(x,y)=0
•
(Show: blob, 32-bit, convolution, abs)
Prewitt kernels
Sobel kernels
52
Prewitt and Sobel
• Conclusion
– Pros:
• Simple to understand
• Simple to implement
• Fast
– Cons:
• Produce wide edges
– Used allot! (most common: Sobel)
53
Edge Detector: Laplacian
54
Edge Detector: Laplacian
• Where exactly is the edge (width)?
''
• Second-order derivative: f ( x, y)
– The variation of the variation of the
gray-level value!
– How fast does the gradient change?
– Find the zero-crossing
• Center of edge
0
Zero-crossing
55
Edge Detector: Laplacian
• The second-order
derivative is very
sensitive to noise!
56
Laplacian of Gaussian
• Solution:
– Combined with a 2D
Gaussian smoothing
filter
– The filter is then
called Laplacian of
Gaussian (LoG)
57
Laplacian of Gaussian
• 2D Laplacian of Gaussian
• Second-order derivative of
the 2D Gaussian
• Can also be approximated
by a discrete mask which
depends on the size of the
Gaussian and the size of
the kernel, for example:
• (show: blob, 32-bit, gauss,
convolution)
0
-1 0
-1 4
0
-1
-1 0
58
Examples of LoG
Gauss filter width:
Input
LoG
Edges
5x5 pixels
9x9 pixels
12x12 pixels
59
Laplacian of Gaussian
• Conclusion
– Pros:
• Edges well-defined due to zero-crossings
– Cons:
• How do we find the zero-crossings?
– I.e., transitions from black to white and vice versa
• Laplacian is too sensitive to noise due to second-order derivative
• Simple to implement except for the search for zero-crossings!
– Not used so often
60
Canny Edge Detector
61
Canny Edge Detector
• Noise reduction
– 2D Gaussian used for smoothing
• Edge enhancement
– Magnitude of gradient vector:
g  g x2  g y2  g x  g y
• Edge localisation
62
Canny Edge Detector
• Edge localisation
– Wide edges:
• Edges give rise to ridges in the gradient
image (show: blobs, edge, invert, region,
3D)
• Thin edges using the principle of nonmaximal suppression
63
Thresholding Dilemma
• Define a threshold in the magnitude-image
– Assume high magnitudes originate from objects
• Dilemma:
– Too high threshold:
• Remove noise 
• Remove small edges 
– Too low threshold:
• Keep noise 
• Keep small edges 
• How do we threshold the magnitude image so that
noise is suppressed and the object-edges are
64
not???
Hysteresis Thresholding
• Concept: think of object edges as a group of edges
• Use two thresholds: Thmin and Thmax
– All magnitude edge-pixel below Thmin are sat to zero
• Choose Thmin so low that no object edges are eliminated
– Find all grouped edges (non-zero edge-pixels)
– Ignore all groups which are too small (in number)
– Ignore all groups which do not contain at least one edge-pixel
with a magnitude above Thmax
• Choose Thmax so high that (most) noise edges are eliminated
• Effect:
– Isolated (noise) edge-pixels are ignored
– Both strong and weak edge-pixels are kept
65
Examples
Gauss 5x5,
Thmax = 128,
Thmin = 1
Gauss 5x5,
Thmax = 255,
Thmin = 1
Gauss 5x5,
Thmax = 255,
Thmin = 220
Gauss 9x9,
Thmax = 128,
Thmin = 1
66
Examples
• Comparison on noisy image
Canny
Sobel
67
Examples
Gauss 5x5,
Thmax = 200,
Thmin = 1
Gauss 7x7,
Thmax = 200,
Thmin = 1
68
Canny edge detector
• Conclusion:
• Pros:
– One pixel wide edges
– Edges are grouped together (often good for segmentation)
– Robust to noise!
• Cons:
– Complicated to understand and implement
– Slow
• Used allot!
69
What to remember
• Point processing: pixel-wise operations
– An image calculator…(to image, image and scalar):
AND, OR, +, -, …
• Neighborhood processing
– Convolution and correlation
• Kernel, mask, filter, template
– Template matching (object recognition)
– Filtering
•
•
•
•
Mean filter: blur, preprocessing
Median filter (good at removing noise)
Remove “background”
Sharpening
70
What to remember
• Edge = Rapid intensity change
• Edge information is one of the most important in
CV and Human Vision
• Three steps in edge detection:
– Noise reduction
– Edge enhancement
– Edge localisation
• Three types were presented:
– Based on first-order derivative
• Prewitt and Sobel
– Based on second-order derivative
• Laplacian of Gaussian
– Based on groups of edges
• Canny
71