COMPUTATIONAL NEUROSCIENCE
FINAL PROJECT – DEPTH VISION
Omri Perez
2013
INTRO
DEPTH CUES
Pictorial Depth Cues
 Physiological Depth Cues
 Motion Parallax
 Stereoscopic Depth Cues

PHYSIOLOGICAL DEPTH CUES
Two Physiological Depth Cues:
1. Accommodation
2. Convergence
PHYSIOLOGICAL DEPTH CUES
Accommodation
–
PHYSIOLOGICAL DEPTH CUES
Accommodation
 relaxed lens = far away
 accommodating lens = near
What must the visual system be able to
compute unconsciously?
PHYSIOLOGICAL DEPTH CUES
Convergence
–
PHYSIOLOGICAL DEPTH CUES
Convergence
 small angle of convergence = far away
 large angle of convergence = near
–
–
What two sensory systems is the brain
integrating?
What happens to images closer or
farther away from fixation point?
MOTION DEPTH CUES
Parallax
MOTION DEPTH CUES
–
Parallax
 Points at different locations in the visual
field move at different speeds depending
on their distance from fixation
 http://www.youtube.com/watch?v=ktdnA6
y27Gk&NR=1
Seeing
Seeing in
in
Stereo
Stereo
SEEING IN STEREO
It’s
It’svery
veryhard
hardto
toread
readwords
wordsififthere
there
are
aremultiple
multipleimages
imageson
onyour
yourretina
retina
SEEING IN STEREO
It’s
It’svery
veryhard
hardto
toread
readwords
wordsififthere
there
are
aremultiple
multipleimages
imageson
onyour
yourretina
retina
But how many images are there on your
retinae?
BINOCULAR DISPARITY

Your eyes have a different image on each
retina

hold pen at arms length and fixate the spot
how many pens do you see?
 which pen matches which eye?

BINOCULAR DISPARITY

Your eyes have a different image on each
retina

now fixate the pen
how many spots do you see?
 which spot matches which eye?

BINOCULAR DISPARITY

Binocular disparity is the difference
between the two images
BINOCULAR DISPARITY
Binocular disparity is the difference
between the two images
 Disparity depends on where the object
is relative to the fixation point:

 objects
closer than fixation project images
that “cross”
 objects farther than fixation project
images that do not “cross”
BINOCULAR DISPARITY

Corresponding retinal points
BINOCULAR DISPARITY

Corresponding retinal points
BINOCULAR DISPARITY

Corresponding retinal points
BINOCULAR DISPARITY

Corresponding retinal points
BINOCULAR DISPARITY

Points in space that have
corresponding retinal points define a
plane called the horopter or Panum’s
fusional area
The Horopter
BINOCULAR DISPARITY

Points not on the horopter will be
disparate on the retina (they project
images onto non-corresponding points)
BINOCULAR DISPARITY

Points not on the horopter will be
disparate on the retina (they project
images onto non-corresponding points)

The nature of the disparity depends on
where they are relative to the horopter
BINOCULAR DISPARITY
points nearer than 
horopter have crossed
disparity
points farther than 
horopter have
uncrossed disparity
BINOCULAR DISPARITY

Why don’t we see double vision?
BINOCULAR DISPARITY

Why don’t we see double vision?

Images with a small enough disparity
are fused into a single image
BINOCULAR DISPARITY

Why don’t we see double vision?

Images with a small enough disparity are
fused into a single image

The region of space that contains images
with close enough disparity to be fused is
called Panum’s Area
BINOCULAR DISPARITY
Panum’s Area 
extends just in front
of and just behind
the horopter
STEREOPSIS

Our brains interpret crossed and
uncrossed disparity as depth

That process is called stereoscopic
depth perception or simply stereopsis
STEREOPSIS

Stereopsis requires that the brain can
encode the two retinal images
independently
STEREOPSIS

Primary visual cortex (V1) has bands of
neurons that keep input from the two eyes
separate
CORTICAL HYPER COLUMNS IN V1


The basic processing unit of depth perception
The cortical column consists of a complete set of
orientation columns over a cycle of 180º and of right
and left dominance columns in the visual cortex. A
hypercolumn may be about 1 mm wide.
OUR GOAL

To compute the binocular depth of stereo
images
Left
Right
SIMULATING RECEPTIVE FIELDS IN V1


To emulate the receptive fields of V1 neurons we use
the Gabor function.
Sinus
Even symmetry
.*
10
10
20
20
20
30
30
30
40
40
40
50
10
Odd symmetry
Gabor

10
50

2D Gaussian
20
30
40
50
50
10
20
30
40
50
10
10
10
20
20
20
30
30
30
40
40
40
50
50
10
20
30
40
50
10
20
30
40
50
10
20
30
40
50
50
10
20
30
40
50
FILTERING THE IMAGES WITH GABOR FILTERS
Left
Right

We filter by
doing a 2D
convolution of
the filters with
the image. The
different results
are averaged
together.
ESTIMATION OF BINOCULAR DISPARITY
2D cross correlation (xcorr2)
Tip: In most cases, peak
cross correlation results
in the x axis (columns)
between the left and
right eye should only be
positive!
BASIC ALGORITHM – MAXIMUM OF CROSS
CORRELATION
RANDOM DOT STEREOGRAM (RDS)
20
40
60
80
100
120
140
50
100
150
200
3 VARIATIONS:
1. maximum of cross correlation
 2. First neuron to fire in a 2D LIF array (winner take all). The
input is the cross correlation result.
 3. Population vector of 2D LIF array after X simulation steps.
Same input.
Notice the horizontal smearing in 2 and 3 is because of cross over
activity when switching patches

YOUR TASK
1.
2.
3.
4.
Load a pair of stereo images. You can use the supplied function
image_loader.m which makes sure the image is in grayscale and has a
proper dynamic range.
Generate the filters. You can use the supplied function generate_filters.m
to generate the array of filters. I urge you to try out different sizes (3rd
parameter) than the default ones in the function.
Filter each of the two images using the filters from the previous stage. You
can use the function filter_with_all_filters.m to do this.
Now, using the two filtered images, iterate over patches, calculate the
cross correlation matrix and determine the current depth using the
methods 1-3 described in the previous slide. You can tweak the overlap of
the patches to reduce computation time.
Note for methods 2 and 3: a. you can use the supplied function
LIF_one_step_multiple_neurons.m to simulate the LIF neurons b. For methods
2 and 3 it is wise to normalize the xcorr2 results, e.g. by dividing by the
maximum value or some such normalization.
YOUR TASK - CONTINUED
5.
Incorporate all these into a function of the form:
result = find_depth_with_LIF( Left_im_name,Right_im_name, method_num,patch_size,
use_filters )
Where result is the matrix representing the estimated depth (pixel shifts), Left and
Right_im_name are the names of the stereo images. method_num is the number of the
method (1-3, see above). patch_size the ratio of the patch size to the image dimensions.
E.g. in an image that is 640x480 a ratio of 1/15 will produce a patch of size ~43x32. Please
note that in matlab the indexing convention is rowsxcols (and not x,y) so the image is actually 480x640.
use_filters is a flag that determines whether to filter the images (step 3 in previous slide)
before computing the depth map (useful for debugging, however should be set to true when
generating the final results).



In addition to the supplied stereo image pairs, you should also generate a left and right
random dot stereogram image pair using the supplied function RDS_generator.m together
with a mask (I supplied you with an example mask, RDS_Pac-Man_mask.png)
You can find other stereo image pairs online, e.g.
http://vasc.ri.cmu.edu/idb/html/stereo/index.html
Bonus: You can add a fourth depth estimation method of our choice. This can be something
you read somewhere or an original idea. For example you can use one of the methods 1-3
but change the patch scan so it won’t be an orderly right to left then down one row and right
to left. Instead it can be a random scan which, among other things, will cause several
regions to be left uncalculated but other regions more tightly sampled.
WHAT TO HAND IN (E-MAIL IN)
1.
2.
Your code together with any images (regular and RDS) you used and the
supplied images.
A document showing the depth results on the two supplied stereo image
pairs and one RDS you generated, for each of the 3 methods .
(If you chose to do the bonus then show the results for the bonus method
as well). Don’t forget when showing results for the RDS to relate them to
the mask used to generate it.
The document should contain a concise explanation of what you did, your
algorithms and interpretation of the results.
HOW TO HAND IN

The project should be submitted by mail to
Omri.

Good Luck and a succesful test period (and
vacation?) !!!