stanford hci group / cs377s
Designing Applications that See
Lecture 3: Image Processing
Dan Maynes-Aminzade
15 January 2008
Designing Applications that See
http://cs377s.stanford.edu
Reminders
ƒ Register on Axess
ƒ Assignment #1 out today, due next Tuesday
in lecture
ƒ Newsgroup:
N
su.class.cs377s
l
377
ƒ Next lecture is an interactive workshop, so
bring your webcam if you have one
ƒ Remember to check the course calendar for
the latest readings
15 January 2008
Lecture 3: Image Processing
2
Today’s Goals
ƒ Get an overview of various low-level image
processing techniques
ƒ Understand what they are good for and
when to use them
ƒ Next lecture: hands-on demo of these
techniques in action
15 January 2008
Lecture 3: Image Processing
3
Outline
ƒ
ƒ
ƒ
ƒ
ƒ
Image basics
Color
Image filters
Features
Shapes
15 January 2008
Lecture 3: Image Processing
4
“Image Processing”
ƒ Which of these two images contains a red circle?
15 January 2008
Lecture 3: Image Processing
5
“Image Processing”
ƒ Which of these two images contains a red circle?
15 January 2008
Lecture 3: Image Processing
6
“Image Processing”
ƒ Which of these two images contains a red circle?
15 January 2008
Lecture 3: Image Processing
7
“Preattentive” Processing
ƒ Certain basic visual properties are detected
immediately by low-level visual system
ƒ “Pop-out” vs. serial search
ƒ Tasks
T k that
th t can b
be performed
f
d iin less
l
than
th
200 to 250 milliseconds on a complex
display
ƒ Eye movements take at least 200 ms to
initiate
15 January 2008
Lecture 3: Image Processing
8
Preattentive Visual Search Tasks
15 January 2008
Lecture 3: Image Processing
9
Cockpit Dials
ƒ Detection of a slanted line in a sea of
vertical lines is preattentive
15 January 2008
Lecture 3: Image Processing
10
Perspective
(courtesy of Maria Petrou)
15 January 2008
Lecture 3: Image Processing
11
What is an Image?
ƒ Digital representation of a real-world scene
ƒ Composed of discrete “picture elements”
(pixels)
ƒ Pixels
Pi l parameterized
t i db
by
ƒ Position
ƒ Intensity
ƒ Time
15 January 2008
Lecture 3: Image Processing
12
What is an Image?
ƒ A 2D domain
ƒ With samples at regular points (almost always a
rectilinear grid)
ƒ Whose values represent gray levels, colors, or
opacities
iti
ƒ Common image types:
ƒ 1 sample per point (B&W or Grayscale)
ƒ 3 samples per point (Red, Green, and Blue)
ƒ 4 samples per point (Red, Green, Blue, and
“Alpha”, a.k.a. Opacity)
15 January 2008
Lecture 3: Image Processing
13
Channels
ƒ In an images
with multiple
samples per
pixel,, we refer
p
to each set of
one type of
sample as a
"plane" or
"channel."
15 January 2008
3 samples per pixel
Blue Channel of Image,
1 sample per pixel
Lecture 3: Image Processing
Red Channel of Image,
1 sample per pixel
(courtesy Andries van Dam)
14
Spatial Sampling
(courtesy of Gonzalez and Woods)
15 January 2008
Lecture 3: Image Processing
15
Quantization
(courtesy of Gonzalez and Woods)
15 January 2008
Lecture 3: Image Processing
16
What is Image Processing?
ƒ Generally, an attempt to do one of the
following:
ƒ Restore an image (take a corrupted image and
recreate a clean original)
g )
ƒ Enhance an image (alter an image to make its
meaning clearer to human observers)
ƒ Understand an image (mimic the human visual
system in extracting meaning from an image)
15 January 2008
Lecture 3: Image Processing
17
Image Restoration
ƒ Removing sensor noise
ƒ Restoring old, archived film and images that
has been damaged
(courtesy of Gonzalez and Woods)
15 January 2008
Lecture 3: Image Processing
18
Image Restoration
15 January 2008
Lecture 3: Image Processing
19
Image Enhancement
ƒ Often used to increase the contrast in
images that are overly dark or light
ƒ Enhancement algorithms often play to
humans’ sensitivity to contrast
humans
(courtesy of Tobey Thorn)
15 January 2008
Lecture 3: Image Processing
20
Image Understanding
ƒ Image understanding includes many
different tasks
ƒ Segmentation (identifying objects in an image)
ƒ Classification (assigning labels to individual
objects or pixels)
ƒ Interpretation (extracting meaning from the
image as a whole)
15 January 2008
Lecture 3: Image Processing
21
Image Processing:
Hierarchy of Tasks
(courtesy of B. Jähne)
15 January 2008
Lecture 3: Image Processing
22
Separating Regions with Filters
15 January 2008
Lecture 3: Image Processing
23
What is Color?
ƒ A box of pencils?
15 January 2008
Lecture 3: Image Processing
24
What is Color?
ƒ A quantity related to the wavelength of light
in the visible spectrum?
(courtesy of J.M. Rehg)
15 January 2008
Lecture 3: Image Processing
25
What is Color?
ƒ A perceptual attribute of objects and
scenes constructed by the visual system?
(courtesy R. Beau Lotto)
15 January 2008
Lecture 3: Image Processing
26
Color Perception
(courtesy R. Beau Lotto)
15 January 2008
Lecture 3: Image Processing
27
How Do We Use Color?
ƒ In Biological Vision
ƒ Distinguish food from nonfood
ƒ Help identify predators and prey
ƒ Check health of others
ƒ In Computer Vision
ƒ Find a person’s skin
ƒ Segment (group together) pixel regions
belonging to the same object
(courtesy of J.M. Rehg)
15 January 2008
Lecture 3: Image Processing
28
Colorblindness
Ishihara Test for Color Blindness
15 January 2008
Lecture 3: Image Processing
29
Human Photoreceptors
Rods
Cones
15 January 2008
Lecture 3: Image Processing
30
Human Cone Sensitivities
15 January 2008
Lecture 3: Image Processing
31
Reflectance and Transmittance
Reflectance
Transmittance
(courtesy of Brian Wandell)
15 January 2008
Lecture 3: Image Processing
32
Reflectance
15 January 2008
Lecture 3: Image Processing
33
Transmittance
15 January 2008
Lecture 3: Image Processing
34
Illumination Spectra
15 January 2008
Lecture 3: Image Processing
35
Reflectance Spectra
15 January 2008
Lecture 3: Image Processing
36
Assigning Names to Spectra
15 January 2008
Lecture 3: Image Processing
37
Additive Color Mixing
15 January 2008
Lecture 3: Image Processing
38
Subtractive Color Mixing
15 January 2008
Lecture 3: Image Processing
39
The Color Matching Experiment
15 January 2008
Lecture 3: Image Processing
40
Experiment Step 1
(courtesy of Bill Freeman)
15 January 2008
Lecture 3: Image Processing
41
Experiment Step 1
(courtesy of Bill Freeman)
15 January 2008
Lecture 3: Image Processing
42
Experiment Step 1
(courtesy of Bill Freeman)
15 January 2008
Lecture 3: Image Processing
43
Experiment Step 2
(courtesy of Bill Freeman)
15 January 2008
Lecture 3: Image Processing
44
Experiment Step 2
(courtesy of Bill Freeman)
15 January 2008
Lecture 3: Image Processing
45
Experiment Step 2
(courtesy of Bill Freeman)
15 January 2008
Lecture 3: Image Processing
46
Experiment Step 2
(courtesy of Bill Freeman)
15 January 2008
Lecture 3: Image Processing
47
Conclusion from Experiment
ƒ Three primaries are enough for most people
to reproduce arbitrary colors
ƒ Why is this the case?
15 January 2008
Lecture 3: Image Processing
48
Univariance Principle
Lk = ∫ ρ k (λ ) E (λ ) dλ
15 January 2008
k = L, M , S
Lecture 3: Image Processing
For illuminant E (λ )
49
Tristimulus Color Space
ƒ Three axes represent
responses of the three
types of receptors
ƒ Horseshoe-shaped cone
is the set of all humanperceptible colors
15 January 2008
Lecture 3: Image Processing
50
Munsell Color System
ƒ Hue
(Wavelength)
ƒ Value
(Luminosity)
ƒ Chroma
(Saturation)
15 January 2008
Lecture 3: Image Processing
51
Color Space
15 January 2008
Lecture 3: Image Processing
52
How to Match a Color
ƒ Choose primaries A, B, C
ƒ Given an energy distribution, which amounts
of primaries will match it?
ƒ For
F each
h wavelength
l
th iin th
the di
distribution,
t ib ti
determine how much of A, B, and C is
needed to match light of that wavelength
alone
ƒ Add these together to produce a match
15 January 2008
Lecture 3: Image Processing
53
Matching on a Color Circle
15 January 2008
Lecture 3: Image Processing
54
RGB Color Matching
15 January 2008
Lecture 3: Image Processing
55
Geometry of Color (CIE)
ƒ Perceptual color spaces are
non-convex
ƒ Three primaries can span
the space, but weights may
be negative.
ƒ Curved outer edge consists
of single wavelength
primaries
15 January 2008
Lecture 3: Image Processing
56
CIE Chromaticity Diagram
15 January 2008
Lecture 3: Image Processing
57
RGB Color Space
15 January 2008
Lecture 3: Image Processing
58
Constructing Colors
Color can be constructed in many ways
15 January 2008
Lecture 3: Image Processing
59
Color Spaces
ƒ Use color matching functions to define a
coordinate system for color
ƒ Each color can be assigned a triple of
coordinates with respect to some color
space (e.g. RGB)
ƒ Devices (monitors, printers, projectors) and
computers can communicate colors
precisely
15 January 2008
Lecture 3: Image Processing
60
McAdam Ellipses
(10 times actual size)
(Actual size)
(courtesy of D. Forsyth)
15 January 2008
Lecture 3: Image Processing
61
Human Color Constancy
ƒ Distinguish between
ƒ Color constancy, which refers to hue and saturation
ƒ Lightness constancy, which refers to gray-level.
ƒ Humans can perceive
ƒ Color a surface would have under white light (surface
color)
ƒ Color of the reflected light (limited ability to separate
surface color from measured color)
ƒ Color of illuminant (even more limited)
(courtesy of J.M. Rehg)
15 January 2008
Lecture 3: Image Processing
62
Spatial Arrangement and Color
Perception
15 January 2008
Lecture 3: Image Processing
63
Spatial Arrangement and Color
Perception
15 January 2008
Lecture 3: Image Processing
64
Spatial Arrangement and Color
Perception
15 January 2008
Lecture 3: Image Processing
65
Spatial Arrangement and Color
Perception
15 January 2008
Lecture 3: Image Processing
66
Spatial Arrangement and Color
Perception
15 January 2008
Lecture 3: Image Processing
67
(courtesy of D. Forsyth)
15 January 2008
Lecture 3: Image Processing
68
Land’s Mondrian Experiments
ƒ Squares of color with the same color
radiance yield very different color
perceptions Photometer: 1.0, 0.3, 0.3
Photometer: 1.0, 0.3, 0.3
Colored light
White light
Audience: “Red”
Audience: “Blue”
(courtesy of J.M. Rehg)
15 January 2008
Lecture 3: Image Processing
69
Lightness Constancy Algorithm
ƒ The goal is to determine what the surfaces
in the image would look like under white
light.
ƒ Compares the brightness of patches across
their common boundaries and computes
relative brightness
ƒ Establish an absolute reference for lightness
(e.g. brightest point is “white”)
15 January 2008
Lecture 3: Image Processing
70
Lightness Constancy Example
(courtesy of John McCann)
15 January 2008
Lecture 3: Image Processing
71
Finite Dimensional Linear Models
m
E(λ ) = ∑εiψi (λ )
⎞
⎛ m
⎞⎛ n
Lk = ∫ σ k (λ )⎜ ∑ ε iψ i (λ )⎟⎜⎜ ∑ rjϕ j (λ )⎟⎟dλ
⎝ i =1
⎠⎝ j =1
⎠
i=1
=
m,n
∑ ε r ∫ σ (λ )ψ (λ )ϕ (λ )dλ
i =1, j =1
n
ρ (λ ) = ∑ rj ϕ j (λ )
=
i j
k
i
j
m,n
∑ε r g
i =1, j =1
i j
ijk
j=1
15 January 2008
Lecture 3: Image Processing
72
From Images to Objects and Regions
ƒ Attributes of regions
ƒ Bounding edges
ƒ Texture
ƒ The need to compute and
reason about spatial
aggregations of pixels
leads us to filtering
ƒ Key problem is
segmentation
15 January 2008
Lecture 3: Image Processing
73
Features and Filters: Questions
ƒ What is a feature?
ƒ What is an image filter? What is it good for?
ƒ How can we find
f d corners?
ƒ How can we find edges?
15 January 2008
Lecture 3: Image Processing
74
What is a Feature?
ƒ In computer vision, a feature is a local,
meaningful, detectable part of an image
15 January 2008
Lecture 3: Image Processing
75
Why Use Features?
ƒ High information content
ƒ Invariant to changing viewpoint or changing
illumination
ƒ Reduces
R d
computational
t ti
lb
burden
d
(courtesy of Sebastian Thrun)
15 January 2008
Lecture 3: Image Processing
76
One Type of Computer Vision
Image 1
Feature 1
Feature 2
:
Feature N
Computer
Vision
Algorithm
Image 2
Feature 1
Feature 2
:
Feature N
(courtesy of Sebastian Thrun)
15 January 2008
Lecture 3: Image Processing
77
Where Features Are Used
ƒ Calibration
ƒ Image Segmentation
ƒ Correspondence in multiple images (stereo,
structure
t t
ffrom motion)
ti )
ƒ Object detection, classification
(courtesy of Sebastian Thrun)
15 January 2008
Lecture 3: Image Processing
78
What Makes a Good Feature?
ƒ Invariance
ƒ
ƒ
ƒ
ƒ
View point (scale, orientation, translation)
Lighting condition
Object deformations
Partial occlusion
ƒ Other Characteristics
ƒ Uniqueness
ƒ Sufficiently many
ƒ Tuned to the task
(courtesy of Sebastian Thrun)
15 January 2008
Lecture 3: Image Processing
79
What is Image Filtering?
ƒ Modify the pixels in an image based on
some function of a local neighborhood of
the pixels
10 5
4 5
1 1
15 January 2008
3
1
7
7
Lecture 3: Image Processing
80
Linear Filtering
ƒ Linear case is simplest and most useful
ƒ Replace each pixel with a linear combination of its
neighbors.
ƒ The specification of the linear combination is
called the convolution kernel.
10
4
1
5
5
1
3
1
7
0
*
0
0
0 0.5 0
0 1.0 0.5
=
7
kernel
15 January 2008
Lecture 3: Image Processing
81
I(.) I(.) I(.)
I(.) I(.) I(.)
I(.) I(.) I(.)
g11 g12 g13
g21 g22 g23
g31 g32 g33
f (i,j) =
15 January 2008
g11 I(i-1,j-1)
+ g12 I(i-1,j) + g13 I(i-1,j+1) +
g21 I(i,j-1)
+ g22 I(i,j)
g31 I(i+1,j-1)
+ g32 I(i+1,j) + g33 I(i+1,j+1)
Lecture 3: Image Processing
+ g23 I(i,j+1)
+
82
1
1
1
-1
2
1
-1
-1
1
Step 1
1
1
1
-11
42
2
2
3
-1
2
-2
1
3
3
2
2
1
2
1
3
2
2
2
2
2
3
2
1
3
3
2
2
1
2
1
3
2
2
5
(courtesy of Christopher Rasmussen)
1
1
1
-1
2
1
-1
-1
1
Step 2
1
1
1
22
-2
42
2
3
2
-2
-11
3
3
2
2
1
2
1
3
2
2
5
2
2
2
3
2
1
3
3
2
2
1
2
1
3
2
2
4
(courtesy of Christopher Rasmussen)
1
1
1
-1
2
1
-1
-1
1
Step 3
1
1
1
2
22
-2
42
3
2
-11
3
-3
3
2
2
1
2
1
3
2
2
5
4
2
2
2
3
2
1
3
3
2
2
1
2
1
3
2
2
4
(courtesy of Christopher Rasmussen)
1
1
1
-1
2
1
-1
-1
1
Step 4
1
1
1
2
2
22
-2
36
1
2
1
3 -3
3
-3
1
2
2
1
2
1
3
2
2
5
4
2
2
2
3
2
1
3
3
2
2
1
2
1
3
2
2
4
-22
(courtesy of Christopher Rasmussen)
1
1
1
-1
2
1
-1
-1
1
Step 5
1
2
2
2
3
5
-1
42
1
3
3
9
-1
-2
2
2
1
2
1
3
2
2
4
2
2
2
3
2
1
3
3
2
2
1
2
1
3
2
2
4
-22
(courtesy of Christopher Rasmussen)
1
1
1
-1
2
1
-1
-1
1
Step 6
2
2
2
3
5
4
-2
2
21
3
3
9
6
2
-2
2
-2
1
2
1
3
2
2
2
2
2
3
2
1
3
3
2
2
1
2
1
3
2
2
4
-22
1
1
1
-1
2
1
-1
-1
1
Final Result
2
2
2
3
5
4
4
-2
2
1
3
3
9
6
14
5
2
2
1
2
11
7
6
5
1
3
2
2
9
12
8
5
I
I’
Why is I’ large in some places and small in others?
(courtesy of Christopher Rasmussen)
Filtering Example
15 January 2008
Lecture 3: Image Processing
90
Filtering Example
15 January 2008
Lecture 3: Image Processing
91
Filtering Example
15 January 2008
Lecture 3: Image Processing
92
Problem: Image Noise
(courtesy of Forsyth & Ponce)
15 January 2008
Lecture 3: Image Processing
93
Solution: Smoothing Filter
ƒ If object reflectance changes slowly and noise at each pixel
is independent, then we want to replace each pixel with
something like the average of neighbors
Disadvantage: Sharp (high-frequency) features lost
Original image
15 January 2008
7 x 7 averaging
neighborhood
Lecture 3: Image Processing
94
Smoothing Filter: Details
ƒ Filter types
ƒ Mean filter (box)
ƒ Median (nonlinear)
ƒ Gaussian
ƒ Can specify linear
operation by shifting
kernel over image and
taking product
15 January 2008
Lecture 3: Image Processing
1
1
1
1
1
1
1
1
1
3 x 3 box filter kernel
95
Gaussian Kernel
ƒ Idea: Weight contributions of neighboring pixels by nearness
0.003
0.013
0.022
0.013
0.003
0.013
0.059
0.097
0.059
0.013
0.022
0.097
0.159
0.097
0.022
0.013
0.059
0.097
0.059
0.013
0.003
0.013
0.022
0.013
0.003
5 x 5, σ = 1
ƒ Smooth roll-off reduces “ringing” seen in box filter
15 January 2008
Lecture 3: Image Processing
96
Gaussian Smoothing Example
Box filter
Original image
7x7
kernel
15 January 2008
σ=1
Lecture 3: Image Processing
σ=3
97
Gaussian Smoothing
Averaging
Gaussian
(courtesy of Marc Pollefeys)
15 January 2008
Lecture 3: Image Processing
98
“Box” vs. “Cone” Filters
(courtesy Andries van Dam)
15 January 2008
Lecture 3: Image Processing
99
Smoothing Reduces Noise
(courtesy of Marc Pollefeys)
15 January 2008
Lecture 3: Image Processing
100
What causes an edge?
ƒ Depth discontinuity
ƒ Change in surface
orientation
ƒ Reflectance
discontinuity
(change in surface
properties)
ƒ Illumination
discontinuity
(light/shadow)
15 January 2008
(courtesy of Christopher Rasmussen)
Lecture 3: Image Processing
101
How Can We Find Edges?
15 January 2008
Lecture 3: Image Processing
102
Spatial Structure of Edges
ƒ Edges exist at different scales from fine
(whiskers) to coarse (stripes)
ƒ Solution: Gaussian smoothing
15 January 2008
Lecture 3: Image Processing
103
Blurring Example
15 January 2008
Lecture 3: Image Processing
104
Scale
ƒ Increasing smoothing:
ƒ Eliminates noise edges
ƒ Makes edges smoother and thicker
ƒ Removes fine detail
15 January 2008
Lecture 3: Image Processing
105
Effect of Smoothing Radius
1 pixel
15 January 2008
3 pixels
Lecture 3: Image Processing
7 pixels
106
The Edge Normal
S = dx 2 + dy 2
dy
α = arctan
dx
15 January 2008
Lecture 3: Image Processing
107
Sobel Operator
-1 -2 -1
S1= 0 0 0
1 2 1
S2 =
Edge Magnitude =
Edge Direction =
15 January 2008
-1 0
-2 0
-1 0
2
S1 + S1
tan-1
Lecture 3: Image Processing
1
2
1
2
S1
S2
108
The Sobel Kernel, Explained
-1
-2
0
0
1
2
-1
0
1
= 1/4 * [-1 0 -1] ⊗
1
2
1
Sobel kernel is separable!
1
0
2
0
1
0
-1
-2
-1
= 1/4 * [ 1 2 1] ⊗
1
1
-2
2
-1
1
15 January 2008
1
0
-1
Averaging done parallel to edge
Lecture 3: Image Processing
109
Combining Kernels
( I ⊗ g ) ⊗ h = I ⊗ ( g ⊗ h)
[
0.0030
0.0133
0.0219
0.0133
0.0030
0.0133
0.0596
0.0983
0.0596
0.0133
15 January 2008
0.0219 0.0133
0.0983 0.0596
0.1621 0.0983
0.0983 0.0596
0.0219 0.0133
0.0030
0.0133
0.0219
0.0133
0.0030
]
Lecture 3: Image Processing
⊗
[1
−1
]
110
Robinson Compass Masks
-1 0 1
-2 0 2
-1 0 1
0 1
-1 0
-2 -1
2
1
0
1 2 1
0 0 0
-1 -2 -1
2 1 0
1 0 -1
0 -1 -2
1 0 -1
2 0 -2
1 1 -1
0 -1
-1 0
2 1
-2
-1
0
-1 -2 -1
0 0 0
1 2 1
-2 -1 0
-1 0 1
0 1 2
15 January 2008
Lecture 3: Image Processing
111
Robert’s Cross Operator
1 0
0 -1
S=
+
0 1
-1 0
[ I(x, y) - I(x+1, y+1) ]2 + [ I(x, y+1) - I(x+1, y) ]2
or
S = | I(x, y) - I(x+1, y+1) | + | I(x, y+1) - I(x+1, y) |
15 January 2008
Lecture 3: Image Processing
112
Claim Your Own Kernel!
1
1
1
5
5
5
-1
1
-2
1
-3
0
-3
0
-1
-1
-1
-3
-3
-3
1
Prewitt 1
Kirsch
0
2
-1
0
1
Frei & Chen
1
1
1
1
2
1
0
0
0
0
0
0
-1
-1
-1
-1
-2
-1
Prewitt 2
2
Sobel
(courtesy of Sebastian Thrun)
15 January 2008
Lecture 3: Image Processing
113
What’s Not a Convolution?
ƒ Nonlinear systems
ƒ For example, radial distortion of a fish-eye lens
f
(courtesy of M. Fiala)
15 January 2008
Lecture 3: Image Processing
114
John Canny’s Edge Detector
ƒ John Canny, “Finding Edges and Lines in
Images,” Master’s Thesis, MIT, June 1983.
ƒ Developed a practical edge detection
algorithm with three properties
ƒ Gaussian smoothing
ƒ Non-maximum suppression (remove edges
orthogonal to a maxima)
ƒ Hysteresis thresholding – Improved recovery of
long image contours
15 January 2008
Lecture 3: Image Processing
115
Sobel Example
(courtesy of Sebastian Thrun)
15 January 2008
Lecture 3: Image Processing
116
Canny Example
(courtesy of Sebastian Thrun)
15 January 2008
Lecture 3: Image Processing
117
Comparison
Sobel
Canny
(courtesy of Sebastian Thrun)
15 January 2008
Lecture 3: Image Processing
118
Canny Edge Detection
Steps:
1. Apply derivative of Gaussian
2. Non-maximum suppression
• Thin multi
multi-pixel
pixel wide “ridges”
ridges down to single pixel
width
3. Linking and thresholding
• Low, high edge-strength thresholds
• Accept all edges over low threshold that are
connected to edge over high threshold
15 January 2008
Lecture 3: Image Processing
119
Non-Maximum Suppression
ƒ Select the single maximum point across the
width of an edge.
15 January 2008
Lecture 3: Image Processing
120
Linking to the Next Edge Point
ƒ Assume the marked
point q is an edge
point.
ƒ Take the normal to
the gradient at that
point and use this to
predict continuation
points (either r or p).
15 January 2008
Lecture 3: Image Processing
121
Edge Hysteresis
ƒ Hysteresis: A lag or momentum factor
ƒ Idea: Maintain two thresholds:
kHIGH and kLOW
ƒ Use
U kHIGH to
t find
fi d strong
t
edges
d
tto start
t t edge
d
chain
ƒ Use kLOW to find weak edges which continue
edge chain
ƒ Typical ratio of thresholds is roughly
kHIGH / kLOW = 2
15 January 2008
Lecture 3: Image Processing
122
Canny Edge Detection (Example)
gap is gone
Strong +
connected
weak edges
Original
image
Strong
edges
only
Weak
edges
(courtesy of G. Loy)
15 January 2008
Lecture 3: Image Processing
123
123
Canny Edge Detection (Example)
Using default thresholds in Matlab
15 January 2008
Lecture 3: Image Processing
124
Choosing Parameters
15 January 2008
Lecture 3: Image Processing
125
Choosing Parameters
Fine scale,
High threshold
15 January 2008
Lecture 3: Image Processing
126
Choosing Parameters
Coarse scale,
High threshold
15 January 2008
Lecture 3: Image Processing
127
Choosing Parameters
Coarse scale,
Low threshold
15 January 2008
Lecture 3: Image Processing
128
Image Arithmetic
ƒ Just like matrices, we can do pixelwise
arithmetic on images
ƒ Some useful operations
ƒ Differencing
Differencing: Measure of similarity
ƒ Averaging: Blend separate images or smooth a
single one over time
ƒ Thresholding: Apply function to each pixel, test
value
15 January 2008
Lecture 3: Image Processing
129
Image Type Conversion
ƒ Many processing functions work on just one
channel, so the options are:
ƒ Run on each channel independently
ƒ Convert from color → grayscale weighting each channel
b perceptual
by
t l importance
i
t
15 January 2008
Lecture 3: Image Processing
130
Thresholding
ƒ Grayscale → Binary: Choose threshold
based on histogram of image intensities
15 January 2008
Lecture 3: Image Processing
131
Image Comparison: Histograms
15 January 2008
Lecture 3: Image Processing
132
Color Similarity
ƒ One measure is “Chrominance Distance”:
distance from color to a reference color
15 January 2008
Lecture 3: Image Processing
133
Image Comparison
ƒ One approach to template matching is a
sum of squared differences:
15 January 2008
Lecture 3: Image Processing
134
Correlation for Template Matching
ƒ Note that SSD formula can be written:
h the
h last
l term is b
h mismatch
h is
ƒ When
big, the
small—the dot product measures correlation:
ƒ By normalizing by the vectors’ lengths, we
are measuring the angle between them
15 January 2008
Lecture 3: Image Processing
135
Normalized Cross-Correlation
ƒ Shift template
image over search
image, measuring
normalized
correlation at each
point
ƒ Local maxima
indicate template
matches
15 January 2008
Lecture 3: Image Processing
136
Template Matching Example
(courtesy of Sebastian Thrun)
15 January 2008
Lecture 3: Image Processing
137
Statistical Image Comparison:
Color Histograms
ƒ Steps
ƒ Histogram RGB/HSI triplets over two images to be
compared
ƒ Normalize each histogram by respective total number
of pixels to get frequencies
ƒ Similarity is Euclidean distance between color
frequency vectors
ƒ Insensitive to geometric changes, including
different-sized images
15 January 2008
Lecture 3: Image Processing
138
Connected Components
ƒ After thresholding, method to identify
groups or clusters of like pixels
ƒ Uniquely label each n-connected region in
binary image
ƒ 4- and 8-connectedness
15 January 2008
Lecture 3: Image Processing
139
Connected Components Example
15 January 2008
Lecture 3: Image Processing
140
Binary Operations
ƒ Dilation, erosion
ƒ Dilation: All 0’s next to a 1 → 1 (Enlarge foreground)
ƒ Erosion: All 1’s next to a 0 → 0 (Enlarge background)
15 January 2008
Lecture 3: Image Processing
141
Image Moments: Region Statistics
ƒ Zeroth-order: Size/area
ƒ First-order: Position (centroid)
ƒ Second-order: Orientation
15 January 2008
Lecture 3: Image Processing
142
Other Effects
ƒ Art Effects
ƒ Posterizing
ƒ Faked “Aging”
ƒ “Impressionist”
pi el remapping
pixel
ƒ Technical Effects
ƒ Color remapping
ƒ Grayscale
conversion
ƒ Contrast
balancing
(courtesy Andries van Dam)
15 January 2008
Lecture 3: Image Processing
143
Finding Corners
ƒ Edge detectors perform poorly at corners.
ƒ Corners provide repeatable points for
matching, so are worth detecting.
ƒ Problem:
P bl
ƒ Exactly at a corner, gradient
is ill-defined.
ƒ However, in the region around
a corner, gradient has two or
more different values.
15 January 2008
Lecture 3: Image Processing
144
Harris Corner Detector
⎡ ∑I
C=⎢
⎢⎣∑ I x I y
2
x
15 January 2008
∑I I
∑I
Lecture 3: Image Processing
⎤
x y
2 ⎥
y ⎥
⎦
145
Simple Case
ƒ First, consider case where:
⎡ ∑I
C=⎢
⎢⎣∑ I x I y
2
x
∑I I
∑I
x y
2
y
⎤ ⎡λ1 0 ⎤
⎥=⎢
⎥
⎥⎦ ⎣ 0 λ2 ⎦
ƒ This means dominant gradient directions
align with x or y axis
ƒ If either λ is close to 0, then this is not a
corner, so look for locations where both are
large.
15 January 2008
Lecture 3: Image Processing
146
General Case
ƒ It can be shown that since C is rotationally
symmetric:
⎡λ1 0 ⎤
C=R ⎢
R
⎥
⎣ 0 λ2 ⎦
−1
ƒ So every case is like a rotated version of the
one on last slide.
15 January 2008
Lecture 3: Image Processing
147
So, To Detect Corners
ƒ Filter image with Gaussian to reduce noise
ƒ Compute magnitude of the x and y
gradients at each pixel
ƒ Construct C in a window around each pixel
p
(Harris uses a Gaussian window – just blur)
ƒ Solve for product of lS (determinant of C)
ƒ If lS are both big (product reaches local
maximum and is above threshold), we have
a corner (Harris also checks that ratio of lS is
not too high)
15 January 2008
Lecture 3: Image Processing
148
Gradient Orientation
Closeup
15 January 2008
Lecture 3: Image Processing
149
Corner Detection
ƒ Corners are
detected
where the
product of the
p
ellipse axis
lengths reaches
a local
maximum.
15 January 2008
Lecture 3: Image Processing
150
Harris Corners
(courtesy of Sebastian Thrun)
15 January 2008
Lecture 3: Image Processing
151
Example (σ=0.1)
(courtesy of Sebastian Thrun)
15 January 2008
Lecture 3: Image Processing
152
Example (σ=0.01)
(courtesy of Sebastian Thrun)
15 January 2008
Lecture 3: Image Processing
153
Example (σ=0.01)
(courtesy of Sebastian Thrun)
15 January 2008
Lecture 3: Image Processing
154
Features?
Global, not Local
15 January 2008
Lecture 3: Image Processing
155
Vanishing Points
15 January 2008
Lecture 3: Image Processing
156
Vanishing Points
A. Canaletto [1740], Arrival of the French Ambassador in Venice
15 January 2008
Lecture 3: Image Processing
157
From Edges to Lines
15 January 2008
Lecture 3: Image Processing
158
Hough Transform
b = − mx + y
b' = − m' x + y
b ' ' = − m' ' x + y
15 January 2008
Lecture 3: Image Processing
159
Hough Transform: Quantization
b
m
(courtesy of Sebastian Thrun)
15 January 2008
Lecture 3: Image Processing
160
Hough Transform: Algorithm
ƒ For each image point, determine
ƒ most likely line parameters b,m (direction of
gradient)
ƒ strength (magnitude of gradient)
ƒ Increment parameter counter by strength
value
ƒ Cluster in parameter space, pick local
maxima
15 January 2008
Lecture 3: Image Processing
161
Hough Transform: Results
15 January 2008
Lecture 3: Image Processing
162
Hough Transform?
15 January 2008
Lecture 3: Image Processing
163
Summary
ƒ Many kinds of mathematical operations can
be performed on images to extract basic
information about shapes and features
ƒ Image processing does not itself lead to
image understanding…
ƒ But it’s often a good start
15 January 2008
Lecture 3: Image Processing
164
Reading for Next Lecture
ƒ Introduction to the MATLAB Image
Processing Toolbox
ƒ Assignment #1 Question #4: MATLAB
introduction
15 January 2008
Lecture 3: Image Processing
165