Short Course on
Omnidirectional Vision
International Conference
on Computer Vision
October 10th, 2003
Dr. Christopher Geyer
Univeristy of California,
Berkeley
Prof. Tomáš Pajdla
Prof. Kostas Daniilidis
Center for Machine Perception
GRASP Lab
Czech Technical University University of Pennsylvania
Outline
Intro:
Part 1:
Part 3:
Part 4:
A tour of omnidirectional systems
Christopher: Structure-from-motion with
parabolic mirrors
10 minute break
Kostas: Images as homogeneous spaces
Tomáš: Panoramic and other non-central
cameras
Conclusion
3
Introduction
In this course we will take a detailed look at omnidirectional sensors for computer vision. Omnidirectional
sensors come in many varieties, but by definition must
have a wide field-of-view.
~360º FOV
~180º FOV
>180º FOV
wide FOV dioptric
cameras (e.g. fisheye)
catadioptric cameras (e.g.
cameras and mirror systems)
polydioptric cameras (e.g.
multiple overlapping cameras)
4
Introduction
Q: Why are perspective systems insufficient and why is
field of view important?
A: Perspective systems are one imaging modality of
many, we are interested in sensors better suited to
specific tasks. Sensor modality should enter into design
of computer vision systems
For example, perhaps for
flight wide field-of-view
sensors are appropriate, and
in general useful for mobile
robots.
5
Which one?
From the Page of Omnidirectional Vision
6
http://www.cis.upenn.edu/~kostas/omni.html
(Poly-)Dioptric solutions
One to two fish-eye cameras or many synchornized cameras
Pros:
- High resolution
per viewing angle
Cons:
- Bandwidth
- Multiple cameras
7
(Poly-)Dioptric solutions
One to two fish-eye cameras or many synchornized cameras
Homebrewed polydioptric cameras are cheaper,
but require calibrating and synchronizing;
commercial designs tend to be expensive
8
Catadioptric solutions
Usually single camera combined with convex mirror
Pros:
- Single image
Cons:
- Blindspot
- Low resolution
9
Confused?
Q: What kind of sensor should one use?
A: Depends on your application.
1. If you are primarily concerned with:
– resolution
– surveillance (coverage)
and can afford the bandwidth & expense,
you might stick with polydioptric solutions
2. If you are concerned with
– bandwidth
–servoing, SFM
investigate catadioptric or single wide
FOV dioptric solutions
10
Other myths and hesitations…
Myth: Catadioptric images are by necessity highly distorted.
Truth: Actually no; parabolic mirrors induce no distortion
(perpendicular to the viewing direction).
Myth: Omnidirectional cameras are more complicated than
perspective cameras, and harder to do SFM with.
Truth: Actually no; parabolic mirrors are easy to model,
calibrate and do SFM with.
Truth: Omnidirectional systems have lower resolution
Tradeoff: Balance resolution and field of view for your needs
11
Goals for this part of the course:
Demystifying catadioptric cameras
Simplify:
Catadioptric projections can be described by simple,
intuitive models
Revelations:
Modeling catadioptric
projections can actually
give us insight into
perspective cameras
SFM:
To give a framework
for studying structurefrom-motion in parabolic
cameras
12
Part I:
Modeling central
catadioptric cameras
Outline of Part I
1.
2.
3.
4.
5.
6.
7.
Properties of arbitrary camera projections, caustics
The fixed viewpoint constraint
The central catadioptric projections
Models of their projections
A “unifying model” of central catadioptric projection
Consequences of the model
Application
14
Review:
The projection induced by a camera
f
The projection induced
by a camera is the
function from space
to the image plane, e.g.
16
Review:
The projection induced by a camera
The projection induced
by a camera is the
function from space
to the image plane, e.g.
f-1(p)
The least restrictive
assumption that can be
made about any camera
model is that the inverse
image of a point is a
line in space 17
Review:
The projection induced by a camera
For many cameras,
all such lines do not
necessarily intersect
in a single point
18
Some optics: Caustics
For many cameras,
all such lines do not
necessarily intersect
in a single point
Their envelope is
called a (dia-)caustic
and represents a
locus of viewpoints
19
Review: Central projections
If all the lines intersect
in a single point, then
the system has a single
effective viewpoint and
it is a central projection
20
Review: Central projections
If all the lines intersect
in a single point, then
the system has a single
effective viewpoint and
it is a central projection
If a central projection
takes any line in space
to a line in the plane,
then it must be a
perspective projection
21
When is a catadioptric camera equivalent
– up to distortion – to a perspective one?
If the projection induced
by a catadioptric camera is
at most a scene independent
distortion of a perspective
projection, then it must at
least be a central projection
g
22
When is a catadioptric camera equivalent
– up to distortion – to a perspective one?
If the projection induced
by a catadioptric camera is
at most a scene independent
distortion of a perspective
projection, then it must at
least be a central projection
The lines in space along
which the image is
constant intersect in a
single effective viewpoint
23
When is a catadioptric camera equivalent
– up to distortion – to a perspective one?
Question:
Which combinations of
mirrors and cameras
give rise to a system
with a single effective
viewpoint?
24
Central catadioptric solutions
parabolic mirror &
orthographic camera
hyperbolic mirror &
perspective camera
elliptic mirror &
perspective camera
Theorem [Simon Baker & Shree Nayar, CVPR 1998]:
A catadioptic camera has a single effective viewpoint if
and only if the mirror’s cross-section is a conic section
25
The fixed viewpoint constraint [Baker]
y
y = f (x)
Suppose that the height
of the mirror at x is f (x)

x
And the single effective
viewpoint lies a distance
 from the camera focus
26
The fixed viewpoint constraint
The condition that the a
ray emanating from the
focus is reflected in a
direction incident with
the mirror focus can be
described by an ODE
27
The fixed viewpoint constraint
The solutions to this ODE
can be shown to be restricted
to conic sections, e.g.,
28
Modeling a parabolic projection
q
q
space point
image point
29
Modeling a hyperbolic projection
image point
space point
30
Modeling an elliptic projection
image point
space point
31
Questions about
catadioptric projections
Q: What are the properties of the projections induced by
these types of sensors?
Q: How can we extend a theory of structure-from-motion
and self-calibration for uncalibrated catadioptric
cameras?
Note: there is no difference for calibrated catadioptric
cameras, since they can be warped to calibrated
perspective images
Q: Are there simplified models for all catadioptric
projections?
32
Abstracting catadioptric projections
In each case the projection to one surface (the mirror)
followed by a projection to another surface (the image
plane).
33
Abstracting catadioptric projections
In other words they can be written as the composition
of two functions
– f is a non-linear function (projection to a quadric) and g is a
linear function (projection to a plane)
34
Abstracting catadioptric projections
(projective linear)
(non-linear)
35
Switcharoo...
Can we commute the decomposition such that the nonlinear projection becomes independent of the eccentricity?
(linear)
(non-linear &
parameter
dependent)
(linear)
(non-linear but
parameter
independent)
36
Decomposing catadioptric projections
The general catadioptric projection can be written:
The result can be written in homogeneous coordinates:
(homogeneous coordinates)
(linear projective
transformation)
(non-linear but
parameter 37
independent)
Alternative decomposition
f´
f ´ centrally projects to the sphere; it yields a
homogeneous point whose fourth coordinate
is the distance of the space point
38
Alternative decomposition
g´
g´ centrally projects to the image plane from a
point on the axis of the sphere, the height of the
point is determined by the eccentricity
39
Consequences of this model (1 of 5)
elliptic mirror
parabolic mirror
stereographic
projection
hyperbolic mirror
planar mirror
central/perspective
projection
1. Central catadioptric projections and perspective
projections are represented in one framework
40
Consequences of this model (2 of 5)
2. (a) The projection of a
line in space to the sphere
is a great circle;
(b) The central projection
of a great circle to the
image plane is a conic
section
(c) Since stereographic projection sends
great circles to circles in the image, the
parabolic projection of a line is a circle
41
Consequences of this model (3 of 5)
3. (a) The Jacobian from
the viewing sphere to the
image plane is easily
calculated
(b) The Jacobian for the parabolic/stereographic projection
is proportional to a rotation;
parabolic projection is locally
distortionless, i.e. conformal 42
Consequences of this model (4 of 5)
4. (a) Height function
(b) Satisfies
to the same height
is not one-to-one
: reciprocal eccentricities map
(c) Elliptical and hyperbolic mirrors are indistinguishable
from the projections they induce
43
Consequences of this model (5 of 5)
Recall properties of
perspective projection:
Domain:
Range:
(projective space)
(the projective plane)
a. Antipodal points have same image
b. Equator projects to line at infinity
44
Consequences of this model (5 of 5)
Recall properties of
perspective projection:
Domain:
Range:
(projective space)
(the projective plane)
a. Antipodal points have same image
b. Equator projects to line at infinity
5. Parabolic projection:
Domain:
(exclude plane at )
Range:
(ext’d real plane)
a. Antipodal points have inverted imgs
b. Equator projects to circle proportional to focal length
45
Modeling central catadiopric cameras
1. Unifying model of central catadioptric cameras
2. Line images are conics
3. Conformality of stereographic projection
4. Indistinguishability of elliptic and hyperbolic
projections
5. Inadequacy of projective plane for catadioptric
systems
46
End of Part I
Questions?
Part II:
Focus on Parabolic Mirrors
Outline of Part II
1.
2.
3.
4.
5.
6.
7.
Point, circle and line image representation
The image of the absolute conic
Lorentz transformations and their conformality
Infinitessimal generators of Lorentz transformations
Comparison of Lorentz transformations and rotations
Complex representation & estimation
Linearization of the parabolic projection
49
Projective geometry:
A framework for perspective imaging
Linear projection formula because of
the use of homogeneous coordinates
X
If
then
and
where
and K is the calibration matrix
p
z
(R,t)
y
x
50
Projective geometry:
A framework for perspective imaging
Linear projection formula because of
the use of homogeneous coordinates
If
then
and
where
and K is the calibration matrix
q
p
Where, for example, the line between
two points can be represented by the
cross product of two points:
z
y
x
Lines lie in the dual space
51
Projective geometry:
A framework for perspective imaging
Some things maybe we take for granted:
• Representation of lines & points
• Condition that a line coincide with a point
• Construction of the point coinciding with two lines
• Dually, construction of the line between two points
• Conditions for the coincidence of three lines
• Dually, conditions for the collinearity of three points
• Homographies and the invariance of the cross-ratio
• Absolute conic and all that jazz
52
Projective geometry:
A framework for perspective imaging
Over 2 decades this framework has been used to derive:
• Multiview geometry: multilinear constraints in multiple
views
• E.g., fundamental matrices in two views
• Theories of self-calibration
• Simplifications for special motions: homographies, etc.
Q: How can we extend these results to catadioptric
cameras?
A: Like the perspective theory, start with a framework for
the representation of features
53
But what about non-linearity?
It seems an obstacle to this vision is the non-linearity of
the projection equation:
(projection mapping induced by
a parabolic catadioptric camera)
Recall though that the perspective projection is also
non-linear:
homogenization
54
Representation of circles
Start with a circle in the image plane;
this sphere is not necessarily calibrated
55
Representation of circles
The inverse stereographic
projection of a circle is a circle
56
Representation of circles
Through this circle there passes a unique plane; all such planes
57
are in 1-to-1 correspondence with circles in the image plane
Representation of circles
This plane is in 1-to-1 correspondence with its pole:
58
the vertex of the cone tangent to the sphere at the circle
Representation of circles
This plane is in 1-to-1 correspondence with its pole:
59
the vertex of the cone tangent to the sphere at the circle
Representation of circles
The circle’s center is collinear with the representation and
60
the north pole, the radius varies with position along the line
Representation of circles
Easy:
Solve for u, v and r
Given the position in
space, determine the
circle center and radius
61
Representation of circles
imaginary locus, r imaginary
zero radius, r = 0
Three cases:
a. inside sphere
b. on sphere
c. outside sphere
real locus, r > 0
62
Partition of feature space
Point features and circles have point representations
63
in the same space; recall in projective plane, dual space req’d
Representation of image points
So now we have a representation of image points
For if
then
64
Representation of image points
Recall that the sphere is the locus
of points which satisfy the equation:
In projective space this is the set of
points lying on the quadratic surface
given by a quadratic form
65
Partition of feature space
66
Coincidence condition
ax + by + c = 0
or
p = (x,y)
r
(u,v)
What is the condition that
p lies on a circle of radius
r and center (x,y)?
p = (x,y)
67
Coincidence condition
r
We want:
(u,v)
p = (x,y)
with some algebra one finds:
68
Angle of intersection
Hence, circles orthogonal iff
69
Meets and joins
Projective plane:
Parabolic plane:
Circle through two points not unique
In space there is only one line through two points;
70
Why isn’t this true of their projection? Contradiction?
Images of lines in space
All lines intersect the fronto-parallel horizon
(projection of the equator) antipodally
71
72
image center
focal
length
73
d
r
2f
74
Line image constraint
When taking into account
image center and circle
center, the constraint
(u,v)
(x,y)
becomes:
75
Line image constraint
When taking into account
image center and circle
center, the constraint
(u,v)
(x,y)
…and can be written as
becomes:
where
 represents an imaginary circle76
Line image constraint
77
Line image constraint
Implies calibration by fitting
plane to circle representations
78
Interpretation:
Absolute and calibrating conics
absolute
conic
calibrating
conic
79
Summary up to now
• We have a system of
representation for:
– Image features
– Real radii circles
– Imaginary radii circles
• Conditions for coincidence
• Formula for angle of intersection
• Condition that a circle be a line image,
absolute conic
80
Summary up to now
• We have a system of
representation for:
– Image features
– Real radii circles
– Imaginary radii circles
Questions?
• Conditions for coincidence
• Formula for angle of intersection
• Condition that a circle be a line image,
absolute conic
81
Uncalibrated cameras
...the
correct
Applying
the ray
inverse
through
space
stereographic proj-
ection gives…
What is the
transformation
between the
uncalibrated
and calibrated
points?
…the inverse
When
we try
does
not
give
to
take
the
the correct
inverse
if the
ray
in
space
camera is
uncalibrated…
The
transformation
…
and
it is a
k linear
= one,
s ° k´
is a
member
° s of
i.e. Lorentz
y=Ax
the
k´O(3,1)
linear
group
82
Why should it be linear?
Choose an arbitrary circle and find
its inverse stereographic image
83
Why should it be linear?
Translate the circle; find the inverse
stereographic image of the translated circle
84
Why should it be linear?
The translation in the plane induces a transformation of the sphere which preserves planes
85
Uncalibrated cameras
This argument applies to
scaling, rotation and translation
Thus a similarity transformation in the
plane induces some projective linear
transformation A of circle space
It also sends any point satisfying
to some point satisfying
86
Sphere preserving transformations
where
The set of all such matrices is closed under matrix multiplication, inversion and contains the identity; it is a group
87
Lorentz and orthogonal groups
where
The set of all such matrices is closed under matrix multiplication, inversion and contains the identity; it is a group
Lorentz group
Orthogonal group (in 4-dimensions)
88
The Lorentz group
4 connected components
(this is only a sketch)
89
The Lorentz group
90
The Lorentz Lie group
Suppose we have a
curve satisfying:
A(0) = I
91
The Lorentz Lie group
Differentiate both sides of
to obtain:
Implying:
92
The Lorentz Lie group
93
The Lorentz Lie group
For any matrix Lie group,
a local one-to-one map
from its Lie algebra back
to the Lie group is given
by the exponential map.
ex
p
94
Infinitessimal generators
of the Lorentz group
Rotations about
the x-axis
y-axis
z-axis
Generated by skew-symmetric matrices:
95
Infinitessimal generators
of the Lorentz group
Translations along
the x-axis
y-axis
Scaling about
the origin
Generated by :
96
Lorentz group consistently
transforms circle space
One last property of Lorentz transformations
is that they transform representations of circles
consistent with the transformations of image points
97
Lorentz group consistently
transforms circle space
Questions?
One last property of Lorentz transformations
is that they transform representations of circles
consistent with the transformations of image points
98
Inverting the projection
With insight into properties of parabolic projections, let’s
reconsider the problem of inverting an uncalibrated projection
Recall that we can decompose
the parabolic projection as:
n
s
s is stereographic projection
&
n is projection to the sphere
99
Inverting the projection
With insight into properties of parabolic projections, let’s
reconsider the problem of inverting an uncalibrated projection
Recall that we can decompose
the parabolic projection as:
n
s
k
k is a calibration
transformation
100
Inverting the projection
However we now know that there exists some
projective linear k´ such that s  k´ = k  s
101
Inverting the projection
s-1(x)
s-1(k(x))
k(x)
x
We have the points in the plane and
their inverse stereographic images
102
Inverting the projection
Problem: obtain s-1(x) as a
linear transformation of s-1(k(x)).
s-1(x)
s-1(k(x))
k(x)
x
We have the points in the plane and
their inverse stereographic images
103
Inverting the projection
Problem: obtain s-1(x) as a
linear transformation of s-1(k(x)).
Knowns: k(x)
Unknowns: x, k
Non-linear in unknown:
s-1(x) = s-1(k-1(k(x)))
s-1(x)
s-1(k(x))
k(x)
x
We have the points in the plane and
their inverse stereographic images
104
Inverting the projection
s-1(x)
K´s-1(x) = s-1(k(x))
k(x)
x
k and K´ commute about s-1
105
Inverting the projection
s-1(x) = K´-1 s-1(k(x))
K´s-1(x) = s-1(k(x))
k(x)
x
Therefore the calibrated point is a linear
transformation of the lifting of the uncalibrated point
106
Inverting the projection
s-1(x) = K´-1 s-1(k(x))
K´s-1(x) = s-1(k(x))
Linear in unknown:
s-1(x) = K´-1 s-1(k(x))
k(x)
x
Therefore the calibrated point is a linear
transformation of the lifting of the uncalibrated point
107
Linearization of the inverse projection
P
PK´-1 s-1(k(x))
k(x)
x
Then the ray (in P2), as a function of the uncalibrated
image point is, is a linear transformation of the lifting
108
Linearization of the inverse projection
PK´-1 s-1(k(x)) is equivalent to the
perspective projection of the space point
PK´-1 s-1(k(x))
k(x)
x
Then the ray (in P2), as a function of the uncalibrated
image point is, is a linear transformation of the lifting
109
Linearization of the inverse projection
K´-1 is and unknown but linear transformation
(and can be absorbed into linear constraints)
s-1(x) is a non-linear but known transformation
PK´-1 s-1(k(x))
k(x)
x
Then the ray (in P2), as a function of the uncalibrated
image point is, is a linear transformation of the lifting
110
uncalibrated rays
We do not claim
that there is a
linear transformation from
uncalibrated RAYS
(i.e. elements of P2)
to calibrated RAYS
(elements of P2)
calibrated rays
111
uncalibrated points
Instead, we claim
that there is a
linear transformation from
uncalibrated lifted
image points
(i.e. elements of P3)
to calibrated RAYS
(elements of P2)
calibrated rays
112
Calibration transformation
113
Calibration transformation
on the absolute conic
114
Calibration transformation
on the absolute conic
The point  is sent to the
origin (0,0,0,1) in P3 in
The origin is in the nullspace of the projection P
Hence PK´-1  = 0
115
Circle space
1.
2.
3.
Representation of point features
Conditions for incidence, etc.
Line image constraint
116
End of Part II
Questions?
Outline of Part III
1.
2.
3.
4.
5.
6.
7.
The parabolic catadioptric fundamental matrix
Self-calibration
Kruppa equations trivially satisfied
Planar homographies & self-calibration
Multiple view geometry
Infinitessimal motions
Conformal rectification
118
Deriving the parabolic epipolar constraint
Suppose two views are separated by a rotation R
and translation t. Given a point X in space, what
constraint must the image points p1 and p2 satisfy?
119
Deriving the parabolic epipolar constraint
If we know the calibrated rays, then they are
known to satisfy the epipolar constraint for
perspective cameras (C. Longuet-Higgins)
120
Deriving the parabolic epipolar constraint
If the image points are uncalibrated, then we
know that the calibrated rays are linearly
related to the uncalibrated liftings
121
Deriving the parabolic epipolar constraint
F
(44 parabolic fundamental matrix)
122
Deriving the parabolic epipolar constraint
Consequently lifted image points
satisfy a bilinear epipolar constraint
123
Self-calibration
To each view there is
associated an IAC
which are represented
by 1 and 2
They are in the
nullspaces of
PKi-1 and so in
the nullspaces
of F and FT
124
Self-calibration
If  = 1 = 2 i.e., the intrinsic parameters are the
same, then  can be uniquely recovered from the
intersection of the nullspaces:
Unless
case:
in which
=
125
A characterization of parabolic
fundamental matrices
Recall that a 33 matrix E is an essential matrix if and
only if
for some U, V in SO(3)
Claim: A 44 matrix F is a parabolic fundamental
matrix if and only if
for some U, V in SO(3,1)
126
Simple proof
()
127
Simple proof
()
()
128
Estimation
Because we have a bilinear constraint (and in general
multilinear constraints) many methods that apply to
the estimation of structure and motion from multiple
perspective images apply, with some exceptions, to
parabolic cameras.
– Normalized epipolar constraint can be minimized
– Unfortunately no equivalent to the 8/7-point algorithm
(averaging Lorentzian singular values does not minimize
Frobenius norm)
– RANSAC and other robust methods apply
– Structure estimation identical to perspective case once
calibrated
– Robust to modest deviations from ideal assumptions (e.g., nonaligned mirror, non-parabolic mirrors, etc.)
129
Two view example
Given these two views with corresponding points
estimate the parabolic fundamental matrix
130
131
132
epipolar circle
two epipoles
133
1
2
(1 , 2)
134
135
δ
δ
136
1
2
A consequence of this is that the epipolar geometry is
completely determined by the two epipoles in each image
and the angle 
Therefore the epipolar geometry has 9 parameters
whereas the motion (5) and intrinsics for each view (6)
total 11. 2-parameter ambiguity.
137
138
139
140
141
142
What is the ambiguity?
We showed that the the epipolar geometry is
determined by nine parameters, and the motion and
camera parameters by eleven, demonstrating that there
is a two-parameter ambiguity. Meaning for any two
images there is a two-parameter family of possible
reconstructions giving rise to the images. What is this
family? Is it closed under some subset of projective
transformations?
143
This is your house
144
This is your house
on a parabolic mirror
145
This is your house on drugs
i.e. this is the ambiguity in the reconstruction
of a house; the ambiguity is not projective
146
End of Part III
Questions?
Outline of Part IV
1. Group-theoretic analysis of the parabolic fundamental
matrix
2. Quotient spaces of bilinear forms (parabolic
fundamental matrices and essential matrices)
3. Essential harmonic transform
148
The space of parabolic
fundamental matrices
What is its structure?
Is it a manifold?
How many degrees-offreedom does it have?
What ambiguities
are there in motion
estimation?
149
Group theoretic analysis
of bilinear constraints
• Let’s examine the LSVD characterization of parabolic
fundamental matrices:
implies fundamental matrices are closed under left or
right multiplication by Lorentz transformations, i.e.
is also a parabolic fundamental matrix.
Note: the same reasoning applies to essential matrices.
150
The action of SO(3,1)  SO(3,1) on
Thus SO(3,1)  SO(3,1)
acts upon the set of
fundamental matrices
F
SO(3,1)  SO(3,1)
(U,V)
151
The action of SO(3,1)  SO(3,1) on
The identity of the
group induces the
identity map
F
e = (I, I)
152
The action of SO(3,1)  SO(3,1) on
The action is
associative
F
g = (U1,V1)
h = (U2,V2)
g · h = (U1U2,V1V2)
153
The action of SO(3,1)  SO(3,1) on
The action is
(left) associative
F
g = (U1,V1)
h = (U2,V2)
g · h = (U1U2,V1V2)
154
The action of SO(3,1)  SO(3,1) on
The action is transitive:
for every F1 & F2
there exists some g
taking F1 to F2
F1
F2
g
155
SO(3,1)  SO(3,1) parameterizes
With the action  ,
SO(3,1)  SO(3,1)
parameterizes
F
156
SO(3,1)  SO(3,1) parameterizes
Because of transitivity,
the parameterization is
surjective (onto); there
is a g mapping F to F´
F
F´
157
parameterizes
Since SO(3) is itself
parameterized by
is parameterized by
(X,Y)
ex
p
158
parameterizes
In fact since
is
surjective in SO(3,1), so
then is the
parameterization of
159
Parameterization not one-to-one
The paramaterization
may be redundant;
e.g., more than one group
element may map F to F
F
g3
g2 g1
160
The set HF
So… what elements
leave F invariant?
Call it HF
F
g
HF
161
The set HF
At the very least it
contains the identity
F
162
The set HF
Also HF is closed under
(i) composition
F
gh
h
g
163
The set HF
Also HF is closed under
and (ii) inversion
F
g
g-1
164
The isotropy subgroup
Hence HF is a subgroup
It is called the isotropy
subgroup
HF
F
165
Cosets of the isotropy subgroup
Multiply every element
of HF by an element g
h1
g
h2
gh1 gh2
F
166
Cosets of the isotropy subgroup
What we obtain is a
translation of HF by g;
a coset of HF
F
HF
gHF
167
Cosets of the isotropy subgroup
F1
Claim: any two elements
of the coset g  HF map F
to the same fundamental
matrix
F
F2
Claim: F1 = F2
168
Cosets of the isotropy subgroup
F1
Since h1 is in HF and by
the associativity of the
action, g and g · h1 both
send F to the same point
F
169
Cosets of the isotropy subgroup
The same reasoning
applies to h2 and so
F1 = F2
F1
F
F2
170
Cosets of the isotropy subgroup
The same reasoning
applies to h2 and so
F1 = F2
F1
F
F2
171
Cosets of the isotropy subgroup
Consequently every
coset is in one-to-one
correspondence with
a fundamental matrix
HF
gHF
F
g·F
hHF
h·F
172
Cosets of HF partition SO(3,1)  SO(3,1)
The cosets are pairwise
disjoint and their union
is all of SO(3,1)  SO(3,1);
they form a partition
F
173
Quotient spaces
The partition of a group
into its cosets is called
the quotient space
F
174
The set of fundamental matrices
form a quotient space
Because of its one-to-one
correspondence, the set of
fundamental matrices
inherits the structure of
a quotient space
175
Quotient of Lie algebras are
automatically manifolds
The dimension of the
quotient space is the
difference in the
dimensions
of the Lie groups
176
Quotient of Lie algebras are
automatically manifolds
The dimension of the
quotient space is the
difference in the
dimensions
of the Lie groups
177
Quotient of Lie algebras are
automatically manifolds
The dimension of the
quotient space is the
difference in the
dimensions
of the Lie groups
9
= 12 –
3
178
All of these results also
apply to essential matrices
Instead, SO(3)  SO(3)
acts on the set of essential
matrices
SO(3)  SO(3)
HE
179
Harmonic analysis of bilinear forms
Is it just a novelty that essential matrices and parabolic
fundamental matrices are quotient spaces?
In other words, who cares?
We believe the description as a quotient space is
important for the following reasons:
– Simple unifying geometric description of bilinear
– Global (surjective) nowhere-singular parameterization
– These spaces are now endowed with Fourier transforms
180
The rotational harmonic transform
Recall that the Fourier transform is a projection of
functions on L2(0,):
Similarly the rotational harmonic transform (RHT)
is a projection of square integrable functions on SO(3)
— denoted L2(SO(3)) — onto an orthonormal basis:
181
The rotational harmonic transform
Recall that the Fourier transform is a projection of
functions on L2(0,):
Similarly the rotational harmonic transform (RHT)
is a projection of square integrable functions on SO(3)
— denoted L2(SO(3)) — onto an orthonormal basis:
“Wigner
d-coefficients”
Rotation invariant
measure on SO(3)
182
The rotational harmonic transform
The rotational harmonic transform obeys a number of
properties some of which are:
– Limit of partial sums converge to function
– Parseval equality
– Shift theorem
– Convolution theorem
183
Functions on the quotient space
To define a function on the space of essential
matrices we take some function on SO(3)
SO(3) and require that it be constant on
cosets of HE. Alternatively f equals its
average over all cosets.
g • HE
Recall that the subgroup
and the cosets are
184
The essential harmonic transform
The essential harmonic transform is a projection of such
a function onto the bi-rotational harmonics of SO(3)
SO(3):
and because it is constant over the cosets it satisfies
185
Applications
Q: What can we do with an essential harmonic
transform?
A: Fast convolutions.
Might it be possible to estimate an essential matrix via
a convolution of two signals to obtain a kind of
correlation value for all possible essential matrices?
Is it possible to unite signal processing and geometry?
To be continued…
186
End of Part IV
Questions?
Unifying model of catadioptric projection & consequences:
1. Line images are conics
2. Conformality of stereographic projection
3. Indistinguishability of elliptic and hyperbolic
projections
4. Inadequacy of projective plane for catadioptric
systems
209
Unifying model of catadioptric projection & consequences:
1.
2.
3.
4.
Line images are conics
Conformality of stereographic projection
Indistinguishability of elliptic and hyperbolic
projections
Inadequacy of projective plane for catadioptric
systems
Circle space
1.
2.
3.
Representation of point features
Conditions for incidence, etc.
Line image constraint
210
Unifying model of catadioptric projection & consequences:
1.
2.
3.
4.
Line images are conics
Conformality of stereographic projection
Indistinguishability of elliptic and hyperbolic
projections
Inadequacy of projective plane for catadioptric
systems
Circle space
1.
2.
3.
Representation of point features
Conditions for incidence, etc.
Line image constraint
Lorentz transformations
1.
2.
3.
4.
Linearization of projection formula
Conformal
Calibration subgroup
Decomposition into calibration
transformation and rotation
211
Unifying model of catadioptric projection & consequences:
1.
2.
3.
4.
Line images are conics
Conformality of stereographic projection
Indistinguishability of elliptic and hyperbolic
projections
Inadequacy of projective plane for catadioptric
systems
Circle space
1.
2.
3.
Representation of point features
Conditions for incidence, etc.
Line image constraint
Parabolic fundamental matrix
1.
2.
3.
4.
Bilinear epipolar constraint
Self-calibration in two views
Equal Lorentzian singular values
Trivial satisfying Kruppa equations
Lorentz transformations
1.
2.
3.
4.
Linearization of projection formula
Conformal
Calibration subgroup
Decomposition into calibration
transformation and rotation
212
Unifying model of catadioptric projection & consequences:
1.
2.
3.
4.
Line images are conics
Conformality of stereographic projection
Indistinguishability of elliptic and hyperbolic
projections
Inadequacy of projective plane for catadioptric
systems
Circle space
1.
2.
3.
Representation of point features
Conditions for incidence, etc.
Line image constraint
Group theoretic characterization
1.
2.
Lorentz transformations
1.
2.
3.
4.
Linearization of projection formula
Conformal
Calibration subgroup
Decomposition into calibration
transformation and rotation
Quotient space of fundamental
and essential matrices
Essential harmonic transform
Parabolic fundamental matrix
1.
2.
3.
4.
Bilinear epipolar constraint
Self-calibration in two views
Equal Lorentzian singular values
Trivial satisfying Kruppa equations
213
Unifying model of catadioptric projection & consequences:
1.
2.
3.
4.
Line images are conics
Conformality of stereographic projection
Indistinguishability of elliptic and hyperbolic
projections
Inadequacy of projective plane for catadioptric
systems
Circle space
1.
2.
3.
Representation of point features
Conditions for incidence, etc.
Line image constraint
Any questions?
Lorentz transformations
1.
2.
3.
4.
Linearization of projection formula
Conformal
Calibration subgroup
Decomposition into calibration
transformation and rotation
Parabolic fundamental matrix
1.
2.
3.
4.
Bilinear epipolar constraint
Self-calibration in two views
Equal Lorentzian singular values
Trivial satisfying Kruppa equations
Group theoretic characterization
1.
2.
Quotient space of fundamental
and essential matrices
Essential harmonic transform
214
Two-view geometry of catadioptric cameras
– Geyer & Daniilidis, “Mirrors in Motion” ICCV 2003
Single-view geometry of catadioptric cameras
– Geyer & Daniilidis, “Catadioptric Projective Geometry” IJCV Dec.
2001
Epipolar geometry of central catadioptric cameras
– Pajdla & Svoboda, IJCV 2002
Theory of Catadioptric Image Formation
– Baker & Nayar, IJCV
Omnidirectional vision in general
– Baker & Nayar, Panoramic Vision
Relating to complex geometry
– Hans Schwerdtfeger Geometry of Complex Numbers, Dover
– Tristan Needham, Visual Complex Analysis, Oxford University Press
– David Blair, Inversion Theory and Conformal Mapping, AMS
215
5 minute break