2. image formation
what is computer vision?
What is the goal of Computer Vision?
Developing computational models and algorithms to
interpret digital images
understand the visual world we live in
What is the Problem of Ambiguity in image Data?
Our image data is not only too little to fully recover and understand the “state of the
visible world”.
It may even be of poor quality:
- Low resolution
- (Sensor) noise Etc.
-> even if we have noisless images, context and macro knowledge is needed to
understand state of the world
-> Our image data is always ambiguous.
Why is CV is an inverse problem?
Goal: We want to devise models and algorithms to understand the visual world much
like we do.
This means
we have to use a lot of different cues...
we have to deal with ambiguity...
we have to exploit what is a plausible and meaningful interpretation...
... in order to extract information about the visual world from a small amount of
data.
How do we turn these cues around?
Pinhole Camera
How does a Pinhole Camera work?
idea: put pinhole in front of film to block rays -> reduces blurring (size of opening =
aperture)
What are two characteristics of the resulting image from a pinhole camera?
resulting image is inverted
digital representation of image is spatially discrete -> unlike real world
image sensor performs sampling of continuous intensity function -> turns it into
array of discrete intensity values (pixels)
How does the setup of a Pinhole Camera look like?
What are some projection properties of a pinhole camera (3D -> 2D) and what have
we lost?
what have we lost
angles
distances
Many-to-one
all points along the same ray map to the same point in image
Points → points
But projection of points on focal plane is undefined
Lines → lines (collinearity is preserved)
But line through focal point projects to a point
Planes → planes (or half-planes)
- But plane through focal point projects to line
What is a vanishing point when projecting 3D -> 2D?
Parallel lines converge at a vanishing point
- Each direction in space has its own vanishing point
- But parallels also parallel to the image plane remain parallel
- All directions in the same plane have vanishing points on the same line
modeling perspective projection
How does the coordinate system for modeling perspective projection look?
put the optical center ( ) at the origin
put the image plain (
) in front of
How can you compute the Projection of
through
to
?
projected depth
but
not a linear transformation because of division by z ->
Transform to Homogeneous Coordinates
Projection is a matrix multiplication in homogeneous coordinates:
What are Homogeneous Coordinates and how are they created and converted
back?
add one coordinate (e.g. homogeneous scene coordinates)
converting from homogeneous coordinates: divide by last coordinate
Coordinate Systems
What Coordinate System Transformations are involved when taking an image?
Extrinsic camera transformation: takes world into camera coordinates.
Intrinsic camera transformation: describes the image formation process.
3D world coordinates -> 3D camera coordinates -> 2D image coordinates
Extrinsic Camera Transformation: World -> Camera
How do camera and world frame relate (+ formula)?
camera frame is a rotated and translated world frame
: coordinates of points in camera frame
: coordinates of points in world frame
: camera center in world frame
: translation
: rotation
Intrinsic Camera Transformation: Camera -> Image
What is a normalized camera coordinate system, a principal point/ -axis/ -offset in
Intrinsic Camera Transformation?
Principal axis: line from the camera center perpendicular to the image plane
Normalized (camera) coordinate system: camera center is at the origin and the
principal axis is the z-axis
Principal point (p): point where principal axis intersects the image plane (origin of
normalized coordinate system)
Principal point offset:
- Camera coordinate system: origin is at the principal point
- Image coordinate system: origin is in the corner
How to account for Principal Point offset in the Calibration Matrix?
Matrix to map Camera center to image center
: projected depth
: principal point
Take pixels/unit measure into account. Multiply pixels/unit matrix with units matrix
to map a point from Camera to Image space, where
equations
describes a system of linear
What Intrinsic parameters does the calibration matrix take into account?
Principal point coordinates:
Focal length:
Pixel magnification factors:
Skew (non-rectangular pixels)
Other factors (not modeled with linear calibration)
Radial distortion
Coordinate Systems summary
How does the Perspective projection pipeline look from world to image frame?
Concatenate transformations into a single projection matrix
:
What is Orthographic Projection?
Special case of perspective projection: Distance from center of projection to image
plane is infinite.
Also called parallel projection
Camera calibration
What is the Problem with Camera calibration?
Given points with known 3D coordinates
estimate the camera parameters
and known image projections
,
How to calulcate the Projection matrix P for Camera Calibration?
use collinearity between
and
1. write using cross product
2. write as equation system -> two linearly independent equations
How many degrees of freedom does the Projection matrix P have and how many
correspondences are needed for a minimal solution?
has 11 degrees of freedom (12 parameters, but scale is arbitrary)
6 Correspondences needed for a minimal solution
more points: least-squares estimation
What Problem occurs when solving Camera Calibration and how can it be solved?
Problem: trivial solution
Solution: add constraint on norm of
Find
that minimizes
(value can be arbitrary)
subject to
-> homogeneous least squares
In practice: Find the right nullspace of A using SVD
Decompose equation system matrix
Take last right singular vector
, assuming that singular values are sorted
Homogeneous least squares: Constrained optimization
How does the Constrained optimization of Camera Parameters turn into an
Eigenvalue Problem and how to solve it?
Goal: Find
that minimizes
subject to
: Rewriting the constraint (Norm of 1) allows us to write the problem using a
Lagrange multiplier and derive the necessary condition
and
: gives us an eigenvalue problem -> implies that
an eigenvalue.
To minimize
subject to
with the smallest eigenvalue .
, we thus choose
is an eigenvector of
as the eigenvector of
Use SVD
Decompose equation system matrix
Take last right singular vector
, assuming that singular values are sorted
What Steps are left after we have P (camera matrix)?
we still have to figure out the intrinsic and extrinsic parameters
First split the projection matrix into a 3x3 matrix and a 3x1 vector
Next, decompose M into upper triangular part K (calibration) and orthonormal part R
(rotation) using RQ-decomposition
Finally, find c as the nullspace of P by means of SVD.
Summary: What have we got now?
We have a projection matrix P, which maps any point in the 3D world coordinate frame
onto the (infinite) image plane of the camera
Given a 3D point in the world, we can find out where its projection is in the image
One goal of vision: given a point in the images, find out where the pre-image is in the
world
We cannot do this from one image: The depth is lost, and the projection matrix can only
tell us the ray through an image point x
3. Cameras
What is the effect of the Aperture and why not make the aperture as small as
possible?
shrinking the aperture makes images more clear
influences depth of field
Why not make the aperture as small as possible?
- Less light gets through
- Diffraction effects (last image)
How does the Lens work, and what points are in focus?
A lens focuses light onto the film
Rays passing through the center are not deviated
All parallel rays converge to one point on a plane located at the focal length
There is a specific distance at which objects are “in focus” calculated by thin lens
formula:
with D = the distance from the object to the lens, D' = the distance from the lens to the
image, f = focal length
any point satisfying this equation is in focus
How to adjust depth of field and what is the effect?
Changing the aperture size affects depth of field
A smaller aperture increases the range in which the object is approximately in focus
But small aperture reduces amount of light – need to increase exposure
How to influence Field of view (Zoom) and how to calculate it?
FOV depends on focal length and the size of the camera retina (sensor)
Smaller FOV = larger focal length
What are flaws of Lenses?
Chromatic aberration: Lens has different refractive indices for different wavelengths:
causing color fringing
Spherical aberration: Spherical lenses don’t focus light perfectly. Rays farther from the
optical axis focus closer. Causes blur away from the image center.
Vignetting
Radial Distortion: Caused by imperfect lenses. Deviations are most noticeable for rays
that pass through the edge of the lens
Digital Camera sensors
Why do we have to perform sampling in image sensors?
Images arriving at our CCD or CMOS sensor are spatially discrete with individual
pixels.
We usually do not work with spatially continuous functions since our cameras do not
sense in this way.
Instead, use (spatially) discrete images
- Sample the 2D domain on a regular grid
- 1D analogy:
How do CCD vs. CMOS sensors work?
Charge Coupled Device (CCD)
transports the charge across the chip and reads it at one corner of the array
An analog-to-digital converter (ADC) then turns each pixel's value into a digital
value by measuring the amount of charge at each photosite and converting that
measurement to binary form.
Complementary metal oxide semiconductor (CMOS)
uses several transistors at each pixel to amplify and move the charge using more
traditional wires.
The CMOS signal is digitized right away, so it needs no separate ADC. Also often
faster (video applications)
Color sensing in Cameras
What is the idea behind the Bayer grid?
Estimate missing components from neighboring values (demosaicing)
What is the Problem with demosaicing and how to avoid it?
Fine black and white detail in the image misinterpreted as color information
avoid using optical low-pass filter (artificial blur)
What are approaches on how to record colors?
Prism: Requires three chips and precise alignment. More expensive, big & heavy
Foveon X3
CMOS sensor
Takes advantage of the fact that red, blue and green light penetrate silicon to
different depths
Better image quality
X-Trans
CMOS sensor
Use color filters as in Bayer sensor, but with pseudo-random spatial arrangement
Avoids some demosaicing artifacts that are due to the regular Bayer grid Requires
more complex demosaicing procedure
What are Issues with digital cameras?
Noise
low light is where you most notice noise
light sensitivity (ISO) / noise tradeoff
Resolution: Are more megapixels better?
requires higher quality lens
noise issues
In-camera processing
oversharpening can produce halos
RAW vs. compressed
file size vs. quality tradeoff
Blooming
charge overflowing into neighboring pixels
white balance
4. Image Filtering & Image Pyramids
What are 3 Linear Operations? (+ formula)
homogeneity:
additivity:
superposition:
Convolution
What is the idea of Convolution?
Replace each pixel by a linear combination of its neighbors (and itself).
2D convolution (discrete):
What are properties of convolution?
linear:
associative:
commutative:
shift-invariant:
can be represented as matrix-vector product:
Linear Filtering
What is the goal of filtering?
reduce noise
What assumption underlies filtering?
The pixel’s neighborhood contains information about its intensity
How does the averaging filter work?
replaces each pixel with an average of its neighborhood.
Mask with positive entries that sum to 1.
If all weights are equal, it is called a box filter
How does Gaussian Averaging work?
Isotropic Gaussian (rotationally symmetric)
Weighs nearby pixels more than distant ones
Smoothing kernel proportional to
What does an efficient implementation of linear filtering in 2D look like?
Both the box filter and the Gaussian filter are separable:
1. convolve each row with a 1D filter
2. convolve each column with a 1D filter
What are different boundary-handling strategies when filtering?
Zero padding adds black borders around the image (0 values)
Mirror padding adds a boundary around the image by mirror-reflecting the image on the
original image border
Non-linear filtering & morphology
How does the median filter work?
Replace each pixel with the median in a neighborhood around it.
How do morphological filters work? (only binary images)
Perform convolution with a “structuring element” s
This is a binary mask (often circle or square)
perform thresholding to recover a binary image
Multi-scale image representation
What is the broad idea behind multi-scale image representation and what can it be
used for?
Represent images at multiple scales (resolutions)
Motivation: search
How does the Gaussian Pyramid work?
images are weighted down using a Gaussian smoothing (Gaussian blur) and then
scaled down.
Each pixel containing a local average corresponds to a neighborhood pixel on a lower
level of the pyramid
How does the Laplacian pyramid work?
similar to a Gaussian pyramid but saves the difference image of the blurred versions
between each levels.
What does the result of the Laplacian Pyramid look like, and what can it be used
for?
The Laplacian pyramid provides a simple frequency decomposition into subbands.
Each level / subband contains only image structure of a particular range of spatial
frequencies: E.g., the finest level contains all the high-frequency detail.
We can get back the original image by reversing the decomposition.
This technique can be used in image compression or edge detection
What is Aliasing?
Cannot shrink an image by taking every second pixel: High frequencies cannot be
represented anymore
If we do that anyway, characteristic errors appear:
- Spatial frequencies are misinterpreted (aliasing)
- Typically, small phenomena look bigger; fast phenomena can look slower.
- Common examples:
- Wagon wheels rolling the wrong way in movies.
- Checkerboards misrepresented in ray tracing.
- Striped shirts look strange on color television.
5. Edge detection
What does a simple approach to edge detection look like?
1. “Filter” image to find brightness changes
2. “Fit” lines to the raw measurements
3. “Project” model into the image and “match” to lines
What are 3 characteristics of good edge detection?
good detection: filter responds to edge, not to noise
good localization: detected edge near true edge
single response: one per edge
How to detect edges in 1D?
Edges Correspond to fast changes, where the magnitude of the derivative is large
Algorithmically: find peaks in the 1st derivative
1. smooth with Gaussian
2. calculate derivative
3. finds its local optima
better: calculate derivative of Gaussian kernel before smoothing -> saves one step
How to calculate the Gradient for Edge detection in 2D?
Partial derivatives approximation
in x direction:
in y direction:
with Sobel filters (Gaussian smoothing in opposite direction) based on the idea of finite
differences:
Again smooth with Gaussian:
The scale of the smoothing filter affects derivative estimates, and also the semantics of
the edges recovered. Strong edges persist across scales
What does the Gradient in 2D edge detection look like?
Magnitude:
Direction:
Gradient direction is perpendicular to edge and magnitude measures edge strength
What are the assumptions and conclusions of ‘Optimal’ Edge Detection (Canny)
and what is the Detection/localization tradeoff?
Assume:
linear filtering
additive i.i.d. Gaussian noise
Edge detection should have:
good detection: filter responds to edge, not to noise
good localization: detected edge near true edge
single response: one per edge
Then: Optimal detector is approximately derivative of Gaussian (Canny 1986)
Detection/localization tradeoff:
- more smoothing improves detection
- and hurts localization
How does Non-maximum suppression work?
Check if pixel is local maximum along gradient direction
- choose the largest gradient magnitude along the gradient direction
- requires checking interpolated pixels p and r
How does Hysteresis work?
Initially use high threshold to start edges
noise will typically have lower gradient magnitudes, thus not identified as an edge.
Label all pixels as edge that
- 1. have gradient magnitudes above a second, lower threshold
- 2. are connected to a pixel above higher threshold
How to compute the 2nd order derivatives for edge detection?
1st derivative:
-> mask:
2nd derivative:
How to apply the Laplacian for edge detection?
Laplacian of Gaussian (LoG):
Approximate the LoG by a Difference of Gaussians (DoG) at different scales
-> mask:
6. Appearance-Based Recognition & PCA
Template methods
How to compute similarity between two vectors? (Template Matching)
Dot product: When we filter, we measure the angle (cosine of it, really) between the
filter template and the image patch, however scaled by the length of the vectors.
Filtering an image is a set of dot products
Alternative: Sum of Squared Differences (SSD) between Image and Template vectors
What are Problems with computing similarity using the raw data and what is a
possible solution?
Storage and computation costs become unreasonable as the number of objects
increases.
It may require a very large ensemble of ‘training’ images.
-> Find a lower dimensional representation that captures the variability in the data.
Search using this low-dimensional model.
Linear dimensionality reduction
What is the basic idea of linear dimensionality reduction?
Find a lower dimensional representation that captures the variability in the data.
Search using this low-dimensional model.
Given that differences are structured, we can use ‘basis images’ to transform images
into other images in the same space.
If I give you the mean and one vector to represent the data, what vector would you
choose? (two ways to formulate it)
A vector which minimizes the error between projection and raw data:
Minimizing the error is equivalent to maximizing the variance of the projection:
Principal component analysis
How can we calculate PCA using the Covariance matrix (C)?
compute the mean of the data, and subtract it -> compute the covariance matrix ->
decompose it -> choose a the first D eigenvalues:
C is real, symmetric, and positive-definite.
Thus we can apply Eigendecomposition:
How to choose the number of Dimensions (D) to inlcude in PC-Space?
Larger D leads to better approximation.
1. choose D based on application performance, i.e. choose the smallest D that
makes the application work well enough.
2. choose D so that the eigenbasis captures some fraction of the variance
What is the Problem of doing PCA using the Covariance Matrix?
While we may work in spaces that have 100000s of dimensions, we often only have
100s or 1000s of examples.
We do not need to compute all eigenvectors and eigenvalues. Computing a few dozen
of the largest ones usually suffices.
-> Express the covariance matrix differently:
explicitly calculate the covariance matrix
-> SVD of
-> never have to
How to calculate PCA using SVD?
SVD is a generalization of the eigendecomposition to rectangular matrices
center the Data matrix ( )
The left-singular vectors give us the eigenvectors of the covariance matrix.
The singular values let us easily compute the eigenvalues of the covariance matrix:
-> s is standard deviation of PCs
What is the advantage of computing PCA using SVD instead of
Eigendecomposition and how can it be further improved?
Advantage:
We never have to explicitly build and store the covariance matrix
is of managebale size and so is
BUT:
is still too large
We only need the first N left-singular vectors, because only N singular values can
be non-zero. -> “Economy” decomposition, makes it all manageable memory-wise
and computationally.
We will typically need even fewer than N singular values and left-singular vectors > Special variants that only compute a partial decomposition.
How to use PCA in simple image based Search / Mapping (EigenIrgendwas)?
Project each training image onto the low-dimensional subspace.
Store the vectors of coefficients.
For each image region:
Project it onto the low-dimensional subspace.
Compare this to each stored coefficient vector (cheap).
If the smallest distance is less than some threshold, then it is a mouth.
Advantage to Naive Raw Data approach: Smaller Feature space
What are Limitations of View-Based Approaches?
Appearance Manifolds: Viewpoint changes (Translation, rotation ...), Illumination,
Clutter, Occlusion, Noise
View-Based Approaches ...
are severely challenged by these common variations: To make them work, we
would need an unmanageable amount of examples (training data).
do not generalize well: Almost any variation that hasn’t been captured in the
training data will not be handled gracefully.
Training data is expensive: Humans have to gather and label it.
-> PCA will not work -> Appearance-Based Instance Recognition
Appearance-Based Instance Recognition Global Representations
What is the basic assumption of appearance-based instance recognition?
Objects can be represented by a set of images (“appearances”).
For recognition, it is sufficient to just compare the 2D appearances.
No 3D model is needed.
What is the idea behind Global Representation?
Move away from representing the object simply by its pixels.
Represent each object (view) by a global feature descriptor.
For recognizing objects, just match the (global) descriptors.
Some modes of variation are built into the descriptor, while others must be incorporated
into the training data or the recognition process.
7. Matching Interest Points
Given a sliding window how can we compute the local interest point uniqueness
between two windows?
when shifting the window by
we can calculate the sum of the squared
differences (SSD) “error”/energy of
where
can be approximated by Taylor expansion
What do the Eigenvectors and -values of the structure Tensor H represent? (keep in
mind how it has been derived)
Eigenvalues and eigenvectors of define shifts within the smallest and largest change
( value)
direction of largest increase in
amount of increase in direction
direction of smallest increase in
-
amount of increase in direction
How can we use the Eigenvectors and -values of the structure Tensor H in the
context of detecting interest points?
We want
to be large for small shifts in each direction
- the minimum of
should be large over all unit vectors
- this minimum is given by
What are the 5 steps to detect interest points?
1. Compute the gradient at each point in the image
2. Create the structure tensor
from the entries in the gradient
3. Compute the eigenvalues.
4. Find points with large response (
5. Choose those points where
> threshold)
is a local maximum as interest points
What is the Harris detector (+formula) and how does it compare to
is a variant of the harris operator for interest point detection
typically
the trace is the sum of the diagonals:
?
very similar to
but less expensive (no square root)
aka. Harris Corner Detector
is Rotation invariant
What is another interest point detector despite the Harris detector (+formula)?
Hessian determinant
Why is the Harris detector rotation invariant?
Rotating the image leads to rotation of the gradient -> the determinant and the trace of
the structure Tensor H don't change
What do the Harris or Hessian detector accomplish and what can they be used for?
Finds interesting, i.e. discriminative points (Harris detector was the “de-facto” standard
for a long time)
Used for recognition, correspondence for stereo, sparse optical flow/motion, etc.
What are the Limitations of the Harris and Hessian detector?
Distinctiveness vs. invariance to transformation
Harris & Hessian find distinctive points, but they are not invariant to scale, affine, and
projective transformations
Scale invariant Interest Point detection
What are the downsides of matching image patches by varying the patch size?
Computationally inefficient/prohibitive
Inefficient, but possible for matching
Prohibitive for retrieval in large databases
Prohibitive for recognition
What is the goal of automatic scale selection in the context of interest point
matching?
Find scale that gives local maximum of some criterion.
Detector finds location and scale of interest points
In both images: independent automatic scale detection
by finding “characteristic” scale of an interest point
What is the principle behind scale space?
Level of details decreases monotonically as the scale of Gaussian smoothing is
increased
How does Harris Laplace interest point matching work?
scale invariant interest point matching
Harris Laplace (HarLap): detecting multiscale Harris points
- compute Harris interest points for each level of gaussian smoothing
- select points which maximize the laplacian pyramid
Interest Point matching
How do we decide if two image regions match?
1. Compute local descriptors / features (describing the region around interest point)
2. Compare them (using some distance) e.g. Jaccard index:
What are other popular interest point matching techniques despite HarLap?
SIFT (more later): Detect local maxima (in space and scale) in Laplacian pyramid
(similar to HarLap). Ensures that eigenvalues are not too dissimilar
SURF (speeded up robust features): Local scale space maxima of Hessian determinant
Both approaches are blob (rather than corner) detectors
What are important properties of local descriptors?
Distinctiveness
Visually similar regions should have similar descriptors
Different regions should have different descriptors
Invariance
Visually similar regions should have similar descriptors despite the transformation
(geometric, photometric) i.e., rotation, brightness
Robustness
Visually similar regions should have similar descriptors despite the noise
(geometric, photometric)
Dimensionality
Descriptors should be low dimensional, i.e. small number of histogram bins.
Efficiency (large databases)
Generalization property
What are examples of 2 local descriptors?
Image patch (vector of pixels)
- Invariant only when computed on normalized patches
- Distinctive & easy to implement
- But high-dimensional and not very robust
- Match descriptors using: Euclidean distance or Cross-correlation
Bank of filter responses (“jet”)
- Invariant only when computed on normalized patch
How does Scale Invariant Feature Transform (SIFT) work? (basic idea + full version)
Orientation histogram (basic idea):
- Take 16x16 square window around detected interest point
- Compute edge orientation (angle of the gradient - 90°) for each pixel
- Throw out weak edges (threshold gradient magnitude)
- Create histogram of surviving edge orientations
Full version:
- Divide the 16x16 window into a 4x4 grid of cells (2x2 case shown below)
- Compute an orientation histogram for each cell
- 16 cells * 8 orientations = 128 dimensional descriptor
What are the advantages of Scale Invariant Feature Transform (SIFT)?
Extraordinarily robust matching technique
Can handle changes in viewpoint (up to about 60 degree out of plane rotation)
Can handle significant changes in illumination (sometimes even day vs. night;
below)
Fast and efficient — can run in real time
What is the motivation behind Shape Context and what are the basic steps?
Motivation
Cope with complex geometric transformations
Cannot compare pixel to pixel
Represent histogram of edge locations (log polar coordinate system)
- Run edge detector
- Make histogram insensitive to minor shape variations
How can we measure the performance of a feature matcher (descriptor +
distance/classifier)?
ROC Curves
Generated by counting
matches, for different thresholds
Want to maximize area under the curve (AUC)
8. Single-View Geometry
Geometric intuition
How does the matrix for projecting a plane look like and what are its
characteristics? (Geometric intuition)
The mapping is given by a (3x3) matrix H and is called a homography (or projectivity)
the most general projective 2D transformation
maps a square to a general quadrangle
has 8 degrees of freedom