CSE 185
Introduction to Computer Vision
Stereo
Multiple views
• Taken at the same time
or sequential in time
• stereo vision
• structure from motion
• optical flow
Why multiple views?
• Structure and depth are inherently ambiguous from
single views.
Why multiple views?
• Structure and depth are inherently ambiguous from
single views.
P1
P2
P1’=P2’
Optical center
Shape from X
• What cues help us to perceive 3d shape
and depth?
• Many factors
–
–
–
–
–
–
–
–
Shading
Motion
Occlusion
Focus
Texture
Shadow
Specularity
…
Shading
For static objects and known light source, estimate surface normals
Focus/defocus
Images from
same point of
view, different
camera
parameters
far focused
near focused
3d shape / depth
estimates
estimated depth map
Texture
estimate surface
orientation
Perspective effects
perspective, relative size, occlusion, texture gradients contribute to
3D appearance of the scene
Motion
Motion parallax: as the viewpoint moves side to side, the
objects at distance appear to move slower than the ones
close to the camera
Occlusion and focus
Estimating scene shape
• “Shape from X”: Shading, Texture, Focus, Motion…
• Stereo:
– shape from motion between two views
– infer 3d shape of scene from two (multiple)
images from different viewpoints
scene point
Main idea:
image plane
optical center
Outline
• Human stereopsis
• Stereograms
• Epipolar geometry and the epipolar constraint
– Case example with parallel optical axes
– General case with calibrated cameras
Human eye
Rough analogy with human visual system:
Pupil/Iris – control
amount of light
passing through lens
Retina - contains
sensor cells, where
image is formed
Fovea – highest
concentration of
cones
Human stereopsis: disparity
Human eyes fixate on point in space – rotate so that
corresponding images form in centers of fovea.
Human stereopsis: disparity
Disparity occurs when
eyes fixate on one object;
others appear at different
visual angles
Random dot stereograms
• Julesz 1960: Do we identify local brightness
patterns before fusion (monocular process) or
after (binocular)?
• To test: pair of synthetic images obtained by
randomly spraying black dots on white objects
Random dot stereograms
Random dot stereograms
1. Create an image of suitable size. Fill it with random dots. Duplicate the image.
2. Select a region in one image.
3. Shift this region horizontally by a small amount. The stereogram is complete.
Random dot stereograms
• When viewed monocularly, they appear random;
when viewed stereoscopically, see 3d structure.
• Conclusion: human binocular fusion not directly
associated with the physical retinas; must
involve the central nervous system
• Imaginary “cyclopean retina” that combines the
left and right image stimuli as a single unit
• High level scene understanding not required for
Stereo
Stereo photograph and viewer
Take two pictures of the same subject from two slightly
different viewpoints and display so that each eye sees
only one of the images.
Invented by Sir Charles Wheatstone, 1838
Image from fisher-price.com
http://www.johnsonshawmuseum.org
http://www.johnsonshawmuseum.org
Public Library, Stereoscopic Looking Room, Chicago, by Phillips, 1923
present stereo images on the screen by simply putting the right and left images in an animated gif
http://www.well.com/~jimg/stereo/stereo_list.html
Autostereograms
Exploit disparity as
depth cue using single
image.
(Single image random
dot stereogram, Single
image stereogram)
Autosteoreogram
Designed to create the
visual illusion of a 3D
scene
Using smooth
gradients
Estimating depth with stereo
• Stereo: shape from motion between two views
• We’ll need to consider:
• Info on camera pose (“calibration”)
• Image point correspondences
scene point
image plane
optical
center
Stereo vision
Two cameras, simultaneous
views
Single moving camera and
static scene
Camera parameters
Camera
frame 2
Extrinsic parameters:
Camera frame 1  Camera frame 2
Camera
frame 1
Intrinsic parameters:
Image coordinates relative to
camera  Pixel coordinates
• Extrinsic params: rotation matrix and translation vector
• Intrinsic params: focal length, pixel sizes (mm), image center
point, radial distortion parameters
We’ll assume for now that these parameters are
given and fixed.
Outline
• Human stereopsis
• Stereograms
• Epipolar geometry and the epipolar constraint
– Case example with parallel optical axes
– General case with calibrated cameras
Geometry for a stereo system
• First, assuming parallel optical axes,
known camera parameters (i.e., calibrated
cameras):
World
point
Depth of p
image point
(left)
image point
(right)
Focal
length
optical
center
(left)
optical
center
(right)
baseline
Geometry for a stereo system
• Assume parallel optical axes, known camera parameters
(i.e., calibrated cameras). What is expression for Z?
Similar triangles (pl, P, pr) and
(Ol, P, Or):
T  xl  xr T

Z f
Z
disparity
T
Z f
xr  xl
Depth from disparity
image I(x,y)
Disparity map D(x,y)
image I´(x´,y´)
(x´,y´)=(x+D(x,y), y)
So if we could find the corresponding points in two images,
we could estimate relative depth…
Basic stereo matching algorithm
• If necessary, rectify the two stereo images to transform
epipolar lines into scanlines
• For each pixel x in the first image
– Find corresponding epipolar scanline in the right image
– Examine all pixels on the scanline and pick the best match x’
– Compute disparity x-x’ and set depth(x) = fB/(x-x’)
Correspondence search
Left
Right
scanline
Matching cost
• Slide a window along the right scanline and
compare contents of that window with the
reference window in the left image
• Matching cost: SSD or normalized correlation
Correspondence search
Left
Right
scanline
SSD
Correspondence search
Left
Right
scanline
Norm. corr
Effect of window size
• Smaller window
+ More detail
– More noise
• Larger window
+ Smoother disparity maps
– Less detail
W=3
W = 20
Failures of correspondence
search
Textureless surfaces
Occlusions, repetition
Non-Lambertian surfaces, specularities
Results with window search
Data
Window-based matching
Ground truth
Beyond window-based matching
• So far, matches are independent for each
point
• What constraints or priors can we add?