Omnidirectional Vision
Frank Dellaert, Georgia Tech CS 3630
Many Slides adapted from Chris Geyer and
Emanuele Menegatti
See also the Page of Omnidirectional Vision
http://www.cis.upenn.edu/~kostas/omni.html
Animal Vision - 1
• 2 Millions species
• Few hundred distinct organisational plans
BUT
only two basic types of
imaging eyes of wide use:
•Single lens camera like eyes
•Multi-lens compound eyes
From R. Dawkins:
Climbing Mount Improbable. Norton, 1996 2
Animal Vision - 2
Q: Why are perspective systems insufficient and why is field of view
important?
A: Perspective systems are one imaging modality of many, we are
interested in sensors better suited to specific tasks. Sensor modality
should enter into design of computer vision systems
For example, perhaps for flight
wide field-of-view sensors are
appropriate, and in general useful
for mobile robots.
[Slide from C. Geyer
see Reference Section]
3
Omnivision in Animals
• 500 million years ago
Trilobites
• Diurnal Insects
• Nocturnal Insects
• Crustaceous
• Gigantocypris
(not omnivision)
4
Omnivision in Paintings
“The wedding of Giovanni Arnolfini” J. Van Eyck 1390-1441
Witch Mirrors
5
Omnivision in Paintings
“The praetor and his wife” Q. Metsys 1466-1530
Witch Mirrors
6
Distortion in Paintings
Anamorphosis
7
Panoramas
(meaning of the XIX century)
Patented by Robert Barker - 1787
8
Photography
Edinburgh - Courtesy of EdVec
Mauna-Kea Observatory
9
10
Wide-Angle Lenses
Pros:
- Single image
[Slide adapted from T. Pajdla
see Reference Section]
Cons:
- Low resolution at
periphery
11
Panoramic cameras
Cirkut Camera
Panning camera
Pros:
- High resolution
per viewing angle
Roundshot
Swing lens
Cons:
- slow acquisition;
- No dynamic scene
- expensive
12
Compound-eye cameras
Ladybug
PointGrey
The Ringcam at
Microsoft Research
Viewplus
Softpia Japan & Gifu University
Pros:
- High resolution
per viewing angle
Cons:
- Bandwidth
- Multiple cameras
calibrating and
synchronizing;
13
- expensive
Compound-eye cameras (inward)
Virtualised Reality
CMU
SMART
eMotion (Spin-off Univ. of Padua)
now BTS Bioengineering
14
Catadioptric cameras
PAL
Panoramic Annular lens
Catadioprtic
Sensor
Pros:
- Single image
Two folded mirror
sensor
Cons:
- Blindspot
- Low resolution
15
Catadioptric Camera
Composed of:
• Standard Camera
• Convex Mirror
• Support
– transparent cylinder
– Lateral bar
16
Summary
. Omnidirectional sensors come in many varieties
~360º FOV
~180º FOV
>180º FOV
wide FOV dioptric
cameras (e.g. fisheye)
polydioptric cameras (e.g.
multiple overlapping cameras)
catadioptric cameras (e.g.
cameras and mirror systems)
17
Omnidirectional cameras
Advantages
•
Wide vision field
•
One-shot image
•
High speed
•
Vertical Lines
•
Rotational Invariance
Disadvantages
• Low Resolution
• Distortions
• Low readability
• Customisable field of view
• Customisable resolution
18
An omnidirectional camera view
Omnidirectional
image
Panoramic Cylinder
(from Centre for Machine Perception, Praha)
19
When is a catadioptric camera equivalent
– up to distortion – to a perspective one?
h
Suppose we are given
a catadioptric image
For what kinds of mirrors
can the image be warped by
h into a perspective image?
20
Review:
The projection induced by a camera
f
The projection induced
by a camera is the
function from space
to the image plane, e.g.
21
Review:
The projection induced by a camera
The projection induced
by a camera is the
function from space
to the image plane, e.g.
f-1(p)
The least restrictive
assumption that can be
made about any camera
model is that the inverse
image of a point is a
line in space
22
Review:
The projection induced by a camera
For many cameras,
all such lines do not
necessarily intersect
in a single point
23
Some optics: Caustics
For many cameras,
all such lines do not
necessarily intersect
in a single point
Their envelope is
called a (dia-)caustic
and represents a
locus of viewpoints
24
Review: Central projections
If all the lines intersect
in a single point, then
the system has a single
effective viewpoint and
it is a central projection
25
Review: Central projections
If all the lines intersect
in a single point, then
the system has a single
effective viewpoint and
it is a central projection
If a central projection
takes any line in space
to a line in the plane,
then it must be a
perspective projection
26
When is a catadioptric camera equivalent
– up to distortion – to a perspective one?
If the projection induced
by a catadioptric camera is
at most a scene independent
distortion of a perspective
projection, then it must at
least be a central projection
g
27
When is a catadioptric camera equivalent
– up to distortion – to a perspective one?
If the projection induced
by a catadioptric camera is
at most a scene independent
distortion of a perspective
projection, then it must at
least be a central projection
The lines in space along
which the image is
constant intersect in a
single effective viewpoint
28
When is a catadioptric camera equivalent
– up to distortion – to a perspective one?
Question:
Which combinations of
mirrors and cameras
give rise to a system
with a single effective
viewpoint?
29
Central catadioptric solutions
parabolic mirror &
orthographic camera
hyperbolic mirror &
perspective camera
elliptic mirror &
perspective camera
Theorem [Simon Baker & Shree Nayar, CVPR 1998]:
A catadioptic camera has a single effective viewpoint
if and only if the mirror’s cross-section is a conic
section
30
Omni-Vision for Mobile Robots
•
•
•
•
•
Map matching
Image-based localization
Observation of Optical Flow
Biomimetic Behaviours
Integration of Omni-vision with other sensors:
– Sonar
– Laser range finder
•
•
•
Outdoor Navigation
SLAM (Simultaneous Localization And Mapping)
Environment reconstruction & 3D mapping
•
Miscellanea
E. Menegatti - Omni Vision
31
Navigation/Localization Tricks
•
•
•
•
Invariance of Azimuth
Rotational Invariance
Vertical Lines mapped in radial lines
Robustness to occlusion
E. Menegatti - Omni Vision
32
Invariance of Azimuth
The azimuth of the object is maintained by the sensor
33
Rotational Invariance
Initial Position
Image Counter-Rotated
P1
Robot Rotated by 90°
P2
P5
P4
P3
34
Vertical Lines  radial lines
Original Image
Edge Detection
+
Hough Transform
E. Menegatti - Omni Vision
New Design to stretch
vertical lines
35
Robustness to occlusion
•
•
Thanks to the wide FOV, usually occluding objects do not change much
the image
Several similarity measures have been proved to be robust to occlusion
•
Extreme case presented by Jogan & Leonardis
Matjaž Jogan, Aleš Leonardis
“Robust localization using an omnidirectional
appearance-based
subspace model of environment”
environment”
Robotics and Autonomous Systems 45 (2003) 51–72
36
Applications
•
•
•
•
•
Map matching
Image-based localization
Observation of Optical Flow
Outdoor Navigation
SLAM (Simultaneous Localization And Mapping)
• Environment reconstruction & 3D mapping
37
Map matching - 1
• Yagi used the vertical
edges of the objects to
find position of the robot
on a map
• Edges tracking
Y. Yagi, Y. Nishizawa, M. Yachida,
Map-Based Navigation for A Mobile Robot with Omnidirectional Image Sensor COPIS,
IEEE Trans. Robotics and Automation,pp.634-648,Vol.11,No.5,1995.10
38
Map matching - 2
• Menegatti et al. used the
Chromatic Transitions of
Interest to perform scan
matching
• Monte-Carlo Localization
Algorithm
• Almost the same
approach used with Laser
range Finders
E. Menegatti, A. Pretto,
Pretto, A. Scarpa,
Scarpa, E. Pagello
Omnidirectional vision scan matching for robot localization in dynamic environments
IEEE Transactions on Robotics, Vol.
Vol. 22, Iss. 3 June 2006 pages 523- 535
39
Image-based navigation - 1
•
Ishiguro and Menegatti:
–
–
–
–
–
–
FFT magnitude for position
FFT phase for heading
Self-organization of the memory
Image-based Localisation
Hierarchical Localization
Image-Based Monte Carlo Localisation
Emanuele Menegatti, Takashi Maeda, Hiroshi Ishiguro,
``Hierarchycal
``Hierarchycal Image-based Memory for Robot Navigation,''
Navigation,''
Robotics and Autonomous Systems, Elsevier - 2004
Emanuele Menegatti, M. Zoccarato, E. Pagello, H.Ishiguro,
``Image-based
``Image-based Monte-Carlo Localisation with Omnidirectional images''
Robotics and Autonomous Systems, Elsevier - 2004
40
Image-based navigation - 2
•
Kröse et al:
– Used Principal Component Analysis to
extract linear feature
– Dataset described in term of eigenimages
– Probabilistic localization
B. Kröse, N. Vlassis, R. Bunschoten, and Y. Motomura.
“A probabilistic model for appareance-based
robot localization”
localization”
Image and Vision Comp, vol. 19(6):pp. 381–391, April 2001..
41
PCA Explained
f " µ + Bc = µ + b1c1 + b2c 2 + ...
•f is a vector, e.g., an image
•B is a set of basis vectors, the
“principal components”
•c are the coefficients
•Principal Components Analysis
discovers the basis vectors one at
a time, in the order that minimizes
the difference between the
reconstructed and the original
vectors F, up to rank(F)
42
Image-based navigation - 3
•
Gross et al:
– Used slices of the
panoramic cylinder
– Slices confronted via
colour histograms
– Hybrid map: topological
map aumented with
metric information
T. Wilhelm, H.-J. Böhme,
Böhme, and H.-M. Gross.
“A multi-modal system for tracking and analyzing faces on a mobile robot”
robot”
Robotics and Autonomous Systems, 48:31–
48:31–40, August 2004.
43
Observation of Optical Flow
•
Ishiguro used:
– Foci of Expansion (FOE) to
estimate relative positions
– No encoder info
•
Svoboda used:
– Optical flow to discriminate
translation and rotations
Tomáš Svoboda, Tomáš Pajdla, and Václav Hlavác.
“Motion estimation using central panoramic cameras”
IEEE Int. Conf. on Intelligent Vehicles, 1998.
Hiroshi Ishiguro, Kenji Ueda and Saburo Tsuji,
``Omnidirectional Visual Information for Navigating a Mobile Robot'',
Robot'', IEEE Int. Conf. on Robotics and
44
Automation (ICRA-93), pp. 799-804, 1993.
Outdoor Navigation - 1
• Omnidirectional Vision for
Road Following with NN:
– Road classification
– Steering angle
Z. Zhu, S.
S. Yang, G.
G. Xu,
Xu, X.
X.Lin, Dingji Shi
"Fast road classification and orientation estimation using omni-view images and neural
networks,"
IEEE Transaction on Image Processing, Vol. 7, No.8, August 1998, pp. 1182-1197.
45
Outdoor Navigation - 2

Image-based navigation:
–
–
–
Topological navigation
Histogram matching
Different localisation
accuracies
Paul Blaer and Peter Allen
“Topological Mobile Robot Localization Using Fast Vision Techniques”
Proceedings of the 2002 IEEE International Conference on Robotics & Automation
2002
46
Outdoor Navigation - 3

Image-based navigation:
– Dimension reduction with PCA
– Histogram matching
José-Joel Gonzalez-Barbosa and Simon Lacroix
Rover localization in natural environments by indexing panoramic images
Proceedings of the 2002 IEEE International Conference on Robotics & Automation 2002
47
SLAM
Michael Kaess and Frank Dellaert,
Visual SLAM with a Multi-Camera Rig,
Georgia Tech Technical Report GIT-GVU-06-06, 2006
Thomas Lemaire, Simon Lacroix.
Long Term SLAM with panoramic vision.
Submitted to Journal of Fields Robotics special issue on
"SLAM in the Fields".
48
Environment Reconstruction
C. Geyer, K. Daniilidis,
Mirrors in Motion: Epipolar geometry and motion estimation,
Proc. Inter. Conf. on Computer Vision, October, 2003, Nice,
France.
H.Bakstein
H.Bakstein,, T.Pajdla
T.Pajdla.. Rendering Novel Views from a Set of
Omnidirectional Mosaic Images. Workshop on
Omnidirectional Vision and Camera Networks 2003, CD ROM,
IEEE June 2003.
49
References
WWW:
The page of omnidirectional vision
http://www.cis.upenn.edu/~kostas/omni.html
ICCV03 Course on Omnidirectional Vision
http://www.cis.upenn.edu/~kostas/omni/iccv03.html
Book:
R. Benosman & S.B. Kang (Eds.)
Panoramic Vision
Springer 2001
50
References
Special issues:
K. Daniilidis& N. Papanikolopoulos
The Big Picture
IEEE Robotics & Automation Magazine Dec. 2004
Hiroshi Ishiguro and Ryad Benosman
Special issue on omnidirectional vision and its applications Machine Vision and
Applications (2003) Vol 14
Yasushi Yagi and Katsushi Ikeuchi
Special Issue on Omni-Directional Research in Japan
International Journal of Computer Vision Vol. 58, Num. 3, Springer July 2004
Peter Sturm, Tomas Svoboda and Seth Teller
Special issue on Omnidirectional Vision and Camera Networks
Computer Vision and Image Understanding Vol. 103, Iss. 3, Sept. 2006
51