Computer Vision:
Image Formation
and Image Noise
Dr. Mikhail Genkin
COMP 4102
Winter 2021
Today’s topics
• Image Formation
• Grayscale and color image
• Noise function
• Image filtering
How is an image created?
The image formation process that produced a particular image depends on
• lighting conditions
• scene geometry
• surface properties
• camera optics
[Source: R. Szeliski]
CCD (Charge-Coupled Device) Cameras
• Small solid state cells convert light energy into electrical
charge (sensing elements always rectangles and are usually
square)
• The image plane acts as a digital memory that can be read
row by row by a computer
What is an image?
What is an image?
Digital Camera
We’ll focus on these in this class
The Eye
[Source: A. Efros]
Image Digitization
Sampling – measuring the value of an image at a finite number of
points.
Quantization – representing the measured value at the sampled point,
by an integer.
Pixel – picture element, usually in the range [0,255]
• A grid (matrix) of intensity values
Computer Vision
• Common to use one byte per value: 0=black, 255=white)
[Source: N. Snavely]
Digital Images
(q also called Bit Depth)
At 8 bits, 256 (2 8) different tones can be assigned to each pixel.
Grayscale Image
10 5 9
100
A digital image is represented by an integer array 𝐸 of 𝑚 𝑏𝑦 𝑛.
𝐸(𝑖, 𝑗), a pixel, is an integer in the range [0, 255].
• We can think of a (grayscale) image as a function 𝑓 ∶ 𝑅2 → 𝑅 giving the
intensity at position (𝑥, 𝑦)
f (x, y)
x
18 / 82
y
• A digital image is a discrete (sampled, quantized) version of this function
[Source: N. Snavely]
Image Transformations
• As with any function, we can apply operators to an image
g (x,y) = f (x,y) + 20
g (x,y) = f (-x,y)
• We’ll talk about special kinds of operators, correlation and convolution (linear
filtering)
[Source: N. Snavely]
Color Image
B
G
R
https://en.wikipedia.org/wiki/Grayscale#Colorimetric_(perc
eptual_luminance-preserving)_conversion_to_grayscale
Let’s get to know images
• Image noise
• Image filters
Noise
In Computer Vision we can think of Noise in several contexts:
• Any entity (in images, data, or intermediate results) that is
not interesting for the main computation
• Spurious pixel fluctuations, introduced by image acquisition
• Numerical inaccuracies, introduced by the computer’s
limited precision, round-offs or truncations, etc
• Entities that do not belong to any meaningful region (edges
that do not correspond to actual object contours)
Image noise
Clinical images
Image noise
Additive and random noise:
𝐼መ 𝑥, 𝑦 = 𝐼 𝑥, 𝑦 + 𝑛(𝑥, 𝑦)
• 𝐼(𝑥, 𝑦) : the true pixel values
• 𝑛(𝑥, 𝑦) : the (random) noise at pixel (𝑥, 𝑦)
Lets see some Noise function!
Random Variables
• A random variable is a variable whose values are
determined by the outcome of a random experiment.
• A random variable can be discrete or continuous.
Examples
1.
A coin is tossed ten times. The random variable X is the
number of tails that are noted. X can only take the values
0, 1, ..., 10, so X is a discrete random variable.
2.
A light bulb is burned until it burns out. The random
variable Y is its lifetime in hours. Y can take any positive
real value, so Y is a continuous random variable.
Mean, Variance of Random Variable
measures how far a set of numbers are
spread out from their average value
measure the amount of variation or
dispersion of a set of values
Gaussian Distribution
• Single variable
Gaussian Distribution
• Bivariate with zero-means and variance 𝜎 2
Gaussian Noise
• Is used to model additive random noise
• The probability of 𝑛(𝑥, 𝑦) is 𝑒
−𝑛2
2𝜎2
• It has zero mean
• The noise at each pixel is independent
Gaussian Noise
Impulsive Noise
• Alters random pixels
• Makes their values very different from the true ones
Salt and Pepper Noise
• Is used to model impulsive noise
Impulsive Noise
Image Filtering
Image filtering
• Modify the pixels in an image based on some function of a local
neighborhood of each pixel
8
5
10
4
5
1
3
1
7
Local image data
𝑓(𝑥)
7
Modified image data
[Source: L. Zhang]
Applications of Filtering
• Enhance an image ( e.g., denoise, resize)
• Extract information (e.g., texture, edges)
• Detect patterns (e.g., template matching)
Noise reduction
• Given a camera and a still scene, how can you reduce noise?
• Take lots of images and average them!
• What's the next best thing?
[Source: S. Seitz]
• Replace each pixel by the average of its neighbors
Moving average in 1D: [1, 1, 1, 1, 1]/5
Non-uniform weights [1, 4, 6, 4, 1] / 16
Original
Original
Smoothed
Smoothed
[Source: S. Marschner]
Moving Average in 2D
[Source: S. Seitz]
Raquel
Moving Average in 2D
[Source: S. Seitz]
Raquel
Moving Average in 2D
[Source: S. Seitz]
Raquel
Moving Average in 2D
[Source: S. Seitz]
Raquel
Moving Average in 2D
[Source: S. Seitz]
Raquel
Moving Average in 2D
[Source: S. Seitz]
Raquel
Averaging result
• Linear filtering
• Non-linear filtering
Correlation
𝑚Τ2
𝐹 o 𝐼(𝑥) = σ𝑖=−𝑚Τ2 F(i)I(𝑥 + 𝑖)
𝐼𝐹 𝑖, 𝑗 = 𝐹 o 𝐼 =
𝑚 Τ2
𝑚 Τ2
෍
෍
ℎ=−𝑚Τ2 𝑘=−𝑚Τ2
F(h,k)I (i +h, j + k)
Linear Filtering – convolution
• The output is the linear combination of the neighbourhood pixels
𝐼𝐴 𝑖, 𝑗 = 𝐼 ∗ 𝐴 =
𝑚 Τ2
𝑚 Τ2
෍
෍
A(h,k)I (i − h, j − k)
ℎ=−𝑚Τ2 𝑘=−𝑚Τ2
• The coefficients come from a constant matrix A, called kernel.
• This process, denoted by ‘*’, is called (discrete) convolution.
1
3
0
1
0
2
10
2
1
0.1 -1
4
1
1
1
0
Image
Kernel
-1
=
5
-1
Filter Output
Handle Border Pixels
Near the borders of the image, some pixels do not have
enough neighbours. Two possible solutions are:
• Set the value of all non-included pixels to zero.
• Set all non-included pixels to the value of the
corresponding pixel in the input image.
Smoothing by Averaging
m/2
1=
m/2
  A(h,k)
h=−m / 2 k =−m / 2
*
1
9
1
1
1
1
1
1
1
1
1
=
Convolution can be understood as weighted averaging.
Gaussian Filter
Discrete Gaussian kernel:
G(h, k) =
−
1
2
2
e
h 2 +k 2
2 2
Where G(h,k) is an element of an m x m array
Gaussian Filter
=
*
𝜎=1
Gaussian Kernel is Separable
IG = I  G =
=
m/2
m/2
 G(h, k)I (i − h, j − k) =
h=−m / 2 k =−m / 2
=
=
m/2
m/2
 e
2 2
h=−m / 2 k =−m / 2
− h2
m/2
m/2
2
e

h=−m / 2
since
−
h 2 +k 2
−
e
2
− k2
2 2 I (i − h, j − k)
e

k=−m / 2
h 2 +k 2
2 2
I (i − h, j − k) =
=e
−
h2
2
2
−
e
k2
2 2
Why is separability useful?
Convolving rows and then columns with a 1D Gaussian kernel.
I

1
38
9 18 9
1
1
=
Ir
1
Ir

1
38
9
18
9
=
result
1
The complexity increases linearly with 𝑚 instead of with 𝑚2 .
Which kernels are Separable?
• A kernel is separable if it can be written as the outer
product of two 1D kernels
• The complexity of filtering an n×n image with an m×m kernel
O(𝑛2 𝑚2 )
• if the kernel is separable?
O(𝑛2 m)
• This is a very important computational advantage
• Can have larger kernels (say up to m=10) on large images
• Can also do multiple operations with different kernels
Outer product – examples
Outer product – more examples
• Most important kernels used in practice are separable
Gaussian vs. Average
• Gaussian and average are smoothing linear filters
• In this case sum of all kernel entries is one
• So that new pixel is in same value range as old
Gaussian Smoothing
Smoothing by Averaging
Gaussian Scale Space (increasing 𝜎)
Gaussian Scale Space (increasing 𝜎)
Noise Filtering
•
Goal is to remove noise and still
preserve image structure (edges)
After Averaging
Gaussian Noise
Gaussian smoothing preserves edges better
than average filter
Gaussian filter best at removing Gaussian noise
(can prove this)
After Gaussian Smoothing
Noise Filtering
•
Neither Gaussian nor average filter
removes salt and pepper noise
After averaging
Salt-and-pepper noise
After Gaussian smoothing
Nonlinear Filtering – median filter
Replace each pixel value 𝐼(𝑖, 𝑗) with the median of the
values found in a local neighbourhood of (𝑖, 𝑗).
Median Filter
Salt-and-pepper noise
After median filtering
Remove noise and preserve edges!
Salt-and-Pepper Noise Removal by Median-type Noise Detectors and Edge-preserving
Regularization
Raymond H. Chan, Chung-Wa Ho, and Mila Nikolova
IEEE Transactions on Image Processing, 14 (2005), 1479-1485.
Reading for Today
• Szeliski, Ch3(3.1.1- 3.1.2)
Next session
• Linear and non-linear filtering
• A bit of algebra!