Images - photometric aspects of image formation - gray level images - linear/nonlinear filtering - edge detection - corner detection CS223, Jana Kosecka Image Brightness values I(x,y) CS223, Jana Kosecka Image model Mathematical tools Analog intensity function Analysis Temporal/spatial sampled function Linear algebra Quantization of the gray levels Numerical methods Point sets Set theory, morphology Random fields Stochastic methods List of image features, regions Geometry, AI, logic CS223, Jana Kosecka Basic Photometry Radiometric model of image formation CS223, Jana Kosecka Basic ingredients Radiance – amount of energy emitted along certain direction Iradiance – amount of energy received along certain direction BRDF – bidirectional reflectance distribution – portion of the energy coming from direction reflected to direction Lambertian surfaces – the appearance depends only on radiance, not on the viewing direction Image intensity for a Lambertian surface CS223, Jana Kosecka Images • Images contain noise – sources sensor quality, light fluctuations, quantization effects CS223, Jana Kosecka Image Noise Models • Additive noise: most commonly used • Multiplicative noise: • Impulsive noise (salt and pepper): Iˆ(i, j ) if x l I (i, j ) imin y (imax imin ) x l • Noise models: gaussian, uniform • Noise Amount: SNR = s/ n CS223, Jana Kosecka Image filtering • How can we reduce the noise in the image • Image acquisition noise due to light fluctuations and sensor noise can be reduced by acquiring a sequence of images and averaging them • Computation of simple features • First stage of visual processing CS223, Jana Kosecka Image Processing 1D signal and its sampled version f = { f(1), f(2), f(3), …, f(n)} f = {0, 1, 2, 3, 4, 5, 5, 6, 10 } CS223, Jana Kosecka Discrete time system • Maps 1 discrete time signal to another f[x] h g[x] • Special class of systems – linear , time-invariant systems Superposition principle Shift (time) invariant – shift in input causes shift in output CS223, Jana Kosecka Convolution sum: unit impluse – if x = 0 it’s 1 and zero everywhere else Every discrete time signal can be written as a sum of scaled and shifted impulses The output the linear system is related to the input and the transfer function via convolution f h g f Convolution sum: Notation: CS223, Jana Kosecka filter g Averaging filter Original image Smoothed image CS223, Jana Kosecka Averaging filter and 0 everywhere else Box filter Ex. cont. Averaging filter center pixel weighted more CS223, Jana Kosecka Convolution in 2D f 9 10 11 9 99 11 10 9 9 11 10 10 1/9 X X X X X X X 10 X X X X X X X X X X X X X 10 11 10 0 0 1 9 10 11 1 0 1 10 9 10 0 2 1 11 10 9 10 9 11 g h 1 1 1 1 1 1 1 1 1 1/9.(10x1 + 11x1 + 10x1 + 9x1 + 10x1 + 11x1 + 10x1 + 9x1 + 10x1) = CS223, Jana Kosecka 1/9.( 90) = 10 Example: 10 11 10 0 0 9 10 11 1 0 I O 10 9 10 0 2 1 11 10 9 10 9 11 9 10 11 9 99 11 10 9 9 11 10 10 X X X X X X 10 7 4 1 X X 1 1 F 1 1/9 1 1 1 1 1 1 1 1 X X X X X X X X X X X X 1/9.(10x1 + 0x1 + 0x1 + 11x1 + 1x1 + 0x1 + 10x1 + 0x1 + 2x1) = 1/9.( 34) = 3.7778 CS223, Jana Kosecka Example: 10 11 10 0 0 9 10 11 1 0 I O 10 9 10 0 2 1 11 10 9 10 9 11 9 10 11 9 99 11 10 9 9 11 10 10 X X X X X X 10 7 4 1 X X 1 1 F 1 1/9 1 1 1 1 1 1 1 1 X X X X X X X 20 X X X X X 1/9.(10x1 + 9x1 + 11x1 + 9x1 + 99x1 + 11x1 + 11x1 + 10x1 + 10x1) = 1/9.( 180) = 20 CS223, Jana Kosecka How big should the mask be? • The bigger the mask, – more neighbors contribute – bigger noise spread. – more blurring. – more expensive to compute Limitations of averaging • Signal frequencies shared with noise are lost • Impulsive noise is diffused but not removed CS223, Jana Kosecka Gaussian Filter • Better option for blurring – The coefficients are samples of a 1D Gaussian. – Gives more weight at the central pixel and less weights to the neighbors. – The further away the neighbors, the smaller the weight. – Gaussian filter Samples from the continuous Gaussian • Gaussian filter is the only one that has the same shape in the space and frequency domains. • There are no secondary lobes – i.e. a truly low-pass filter CS223, Jana Kosecka How big should the mask be? • The std. dev of the Gaussian determines the amount of smoothing. • The samples should adequately represent a Gaussian • For a 98.76% of the area, we need m = 5 5.(1/) 2 0.796, m 5 5-tap filter g[x] = [0.136, 0.6065, 1.00, 0.606, 0.136] CS223, Jana Kosecka Image Smoothing • Convolution with a 2D Gaussian filter • Gaussian filter is separable, convolution can be accomplished as two 1-D convolutions CS223, Jana Kosecka Non-linear Filtering • Replace each pixel with the MEDIAN value of all the pixels in the neighborhood • Non-linear • Does not spread the noise • Can remove spike noise • Expensive to run CS223, Jana Kosecka Example: 10 11 10 0 0 9 10 11 1 0 I 10 11 9 10 9 10 10 9 10 0 9 10 11 9 9 11 1 1 O 2 1 9 11 99 11 10 10 X X X X 10 X X X X X X X X X X X X X X median 10,11,10,9,10,11,10,9,10 sort 9,9,10,10,10,10,10,11,11 CS223, Jana Kosecka X X Example: 10 11 10 0 0 9 10 11 1 0 I 10 11 9 10 9 10 10 9 10 0 9 10 11 9 9 11 1 1 X X X X X 1 X X 10 10 1 X O 2 1 9 11 99 11 10 10 X X X X X X X X X X X X median 11,10,0,10,11,1,9,10,0 sort 0,0,1,9,10,10,10,11,11 CS223, Jana Kosecka Example: 10 11 10 0 0 9 10 11 1 0 I 10 11 9 10 9 10 10 9 10 0 9 10 11 9 9 11 1 1 O 2 1 9 11 99 11 10 10 X X X X 10 X X X X X X X 9 X 10 X X X X X X median 10,9,11,9,99,11,11,10,10 sort 9,9,10,10,10,11,11,11,99 CS223, Jana Kosecka X X Image Features Local, meaningful, detectable parts of the image. We will look at edges and corners CS223, Jana Kosecka Image Features – Edges, Corners • Look for detectable, meaningful parts of the image • Edges are detected at places where the image values exhibit sharp variation Gray value column Edge column Edge CS223, Jana Kosecka Edge detection (1D) F(x) Edge= sharp variation F ’(x) x Large first derivative x CS223, Jana Kosecka Digital Approximation of 1st derivatives df ( x) f ( x x) f ( x) lim x 0 dx x f(x) X-1 df ( x) f ( x 1) f ( x 1) dx 2 Convolve with: -1 0 CS223, Jana Kosecka 1 X X+1 Edge Detection (2D) Vertical Edges: Convolve with: -1 0 Horizontal Edges: -1 Convolve with: 0 1 CS223, Jana Kosecka 1 Noise cleaning and Edge Detection I(x,y) Noise Filter Edge Detection We need to also deal with noise -> Combine Linear Filters CS223, Jana Kosecka E(x,y) Convolve with: -1 0 1 -1 0 1 -1 0 1 Noise Smoothing Noise Smoothing & Edge Detection Vertical Edge Detection This mask is called the (vertical) Prewitt Edge Detector Outer product of box filter [1 1 1]T and [-1 0 1] CS223, Jana Kosecka Convolve with: -1 -1 -1 0 0 0 1 1 1 Noise Smoothing Horizontal Edge Detection Noise Smoothing & Edge Detection This mask is called the (horizontal) Prewitt Edge Detector CS223, Jana Kosecka Sobel Edge Detector Convolve with: and -1 0 1 -2 0 2 -1 0 1 -1 -2 -1 0 0 0 1 2 1 CS223, Jana Kosecka Gives more weight to the center pixels Example 0 0 0 0 0 0 0 0 0 0 0 0 50 50 50 Ix 0 0 50 50 50 0 0 50 50 50 Iy -1 -1 -1 0 0 0 1 1 1 0 0 0 0 0 0 50 50 0 0 0 100 100 0 0 0 150 150 0 0 0 150 150 0 0 CS223, Jana Kosecka 0 0 0 0 0 0 50 100 150 150 0 50 100 150 150 0 0 0 0 0 0 0 0 0 0 -1 -1 -1 0 1 0 1 0 1 Image Derivatives We know better alternative to smoothing Smooth using Gaussian filter g(x) is a 1-D Gaussian filter, g(x,y) – 2-D Gaussian filter Taking a derivative – linear operation (take the derivative of the filter) CS223, Jana Kosecka Gaussian and its derivative CS223, Jana Kosecka Vertical edges CS223, Jana Kosecka First derivative Gradient Magnitude Horizontal edges • Image Gradient CS223, Jana Kosecka Gradient Orientation CS223, Jana Kosecka Orientation histogram CS223, Jana Kosecka Corner detection • A point on a line is hard to match. Intuition: • Right at corner, gradient is ill defined. • Near corner, gradient has two different values. CS223, Jana Kosecka Formula for Finding Corners We look at matrix: Gradient with respect to x, times gradient with respect to y Sum over a small region, the hypothetical corner I C I x I y 2 x Matrix is symmetric I I I CS223, Jana Kosecka x y 2 y First, consider case where: I C I x I y 2 x I I I x y 2 y 1 0 0 2 This means all gradients in neighborhood are: (k,0) or (0, c) or (0, 0) (or off-diagonals cancel). What is region like if: 1. ? 2. ? 3. and ? 4. and ? CS223, Jana Kosecka General Case: In general case from linear algebra, it follows that because C is symmetric: 1 1 CR 0 0 R 2 With R being a rotation matrix. So every case can be intrepreted like one on last slide. CS223, Jana Kosecka Corner Detection • • • • • • Filter image. Compute magnitude of the gradient everywhere. We construct C in a window. Compute eigenvalues 1 and 2. If they are both big, we have a corner Or if smalest eigenvalue of C is bigger than - mark pixel as candidate feature point CS223, Jana Kosecka Point Feature Extraction Harris Corner Detector • Alternatively feature quality function (Harris Corner Detector) CS223, Jana Kosecka Harris Corner Detector - Example CS223, Jana Kosecka Edge Detection gradient magnitude original image • Compute image derivatives • if gradient magnitude > and the value is a local max. along gradient direction – pixel is an edge candidate • how to detect one pixel thin edges ? CS223, Jana Kosecka Canny Edge Detector • The magnitude image Es has the magnitudes of the smoothed gradient. • Sigma determines the amount of smoothing. • Es has large values at edges: – Find local maxima Th CS223, Jana Kosecka Nonmaximum supression • • The inputs are Es & Eo Magnitude and orientation Consider 4 directions D={ 0,45,90,135} wrt x • For each pixel (i,j) do: 1. 2. • Find the direction dD s.t. d Eo(i,j) (normal to the edge) If {Es(i,j) is smaller than at least one of its neigh. along d} • IN(i,j)=0 • Otherwise, IN(i,j)= Es(i,j) The output is the thinned edge image IN x CS223, Jana Kosecka x Thresholding • Edges are found by thresholding the output of NONMAX_SUPRESSION • If the threshold is too high: – Very few (none) edges • High MISDETECTIONS, many gaps • If the threshold is too low: – Too many (all pixels) edges • High FALSE POSITIVES, many extra edges CS223, Jana Kosecka Hysteresis Thresholding Es(i,j)<H Es(i,j)>L Es(i,j)> H Es(i,j)<L Es(i,j)>L Strong edges reinforce adjacent weak edges CS223, Jana Kosecka Finding lines in an image • Option 1: – Search for the line at every possible position/orientation – What is the cost of this operation? • Option 2: – Use a voting scheme: Hough transform CS223, Jana Kosecka Other edge detectors - second-order derivative filters (1D) • Peaks of the first-derivative of the input signal, correspond to “zero-crossings” of the secondderivative of the input signal. F(x) F’’(x) F ’(x) x CS223, Jana Kosecka Edge Detection (2D) y 1D F(x) I(x) dI(x) dx d2I(x) dx2 2D x > Th =0 x I(x,y) |I(x,y)| =(Ix 2(x,y) + Iy2(x,y))1/2 > Th tan = Ix(x,y)/ Iy(x,y) 2I(x,y) =Ix x (x,y) + Iyy (x,y)=0 Laplacian CS223, Jana Kosecka Notes about the Laplacian: • 2I(x,y) is a SCALAR – Can be found using a SINGLE mask – Orientation information is lost • 2I(x,y) is the sum of SECOND-order derivatives – But taking derivatives increases noise – Very noise sensitive! • It is always combined with a smoothing operation: • Filter Laplacian of Gaussian LOG filter I(x,y) Smooth Laplacian CS223, Jana Kosecka O(x,y) g ( x) e x 1D Gaussian 2 2 2 g ' ( x) 1 2 xe x2 2 2 2 x2 2 x 1 2 2 g ' ' ( x) ( 3 )e 2 x 2 e x2 2 2 2D “Mexican Hat” CS223, Jana Kosecka Derivative of Gaussian Filters Measure the image gradient and its direction at different scales (use a pyramid). CS223, Jana Kosecka The Laplacian Pyramid • Building a Laplacian pyramid: – Create a Gaussian pyramid – Take the difference between one Gaussian pyramid level and the next (before subsampling) • Properties – Also known as the difference-of-Gaussian function, which is a close approximation to the Laplacian – It is a band pass filter - each level represents a different band of spatial frequencies • Reconstructing the original image: – Reconstruct the Gaussian pyramid starting at top layer CS223, Jana Kosecka CS223, Jana Kosecka Gaussian pyramid Laplacian Pyramid (note top image is from Gaussian) CS223, Jana Kosecka Add more oriented filters (Malik & Perona, 1990) CS223, Jana Kosecka Alternative: Gabor filters Gabor filters: Product of a Gaussian with sine or cosine Top row shows anti-symmetric (or odd) filters, bottom row the symmetric (or even) filters. No obvious advantage to any one type of oriented filters. CS223, Jana Kosecka An edge is not a line... • How can we detect lines ? CS223, Jana Kosecka