Optimizing Content-Preserving
Projections for Wide-Angle Images
ACM SIGGRAPH 2009
Robert Carroll
(University of California, Berkeley)
Maneesh Agrawal (University of California, Berkeley)
Aseem Agarwala
(Adobe Systems, Inc.)
Outline
•
•
•
•
•
Introduction
Wide-angle projection
Approach
Results
Future work
Introduction
Perspective
Mercator
Stereographic
Paper result
The space of wide-angle projections
• Wide-angle projections should maintain the
following properties:
– Shape constancy
– Orientation constancy
• There is no wide-angle projection that can
simultaneously preserve all of these
properties
Perspective projection
• The viewing sphere is projected onto a
tangent plane through lines emanating from
the center of the sphere.
– Orientation constancy
– Not conformal projection
– Robust for fields of view less than about 40°
– Field of view approaches 180° the stretching
becomes infinite
Mercator projection
The Mercator projection is a cylindrical projection that
is designed to maintain conformality
• Conformal projection
• It can handle a complete 360 horizontal field of view,
but stretches to infinity as the vertical field of view
approaches 180.
• useful for panoramic images
with large horizontal fields of
view
Stereographic projection
The viewing sphere is projected onto a tangent plane
through lines emanating from the pole opposite the
point of tangency.
• Conformal projection
• Like perspective projection, stereographic projection
stretches objects toward the periphery
Approach
Load
Image
Select
lines
Crop result
image
Approach
Select lines
• Click on the two endpoints of the linear
structure to specify the constrain
– general line constraint
endpoints
– fixed orientation line constraint
(modify the general line constraint)
Drawn line
Line in the scene
endpoints
Approach
• The general line constraint
– Keep linear structures in the scene from bending
• The fixed orientation line constraint
– Let linear structures map to straight lines at a
user-specified orientation in output images
(user can choose vertical or horizontal)
Approach
• Given these line constraints our algorithm
computes a mapping from the viewing sphere
to the image plane.
Mathematical setup
• Notations
Viewing sphere maps to plane
longitude : 𝜆
latitude : 𝜙
𝑣
𝑢
vector form: 𝐮 = 𝑢, 𝑣
𝝀 = 𝜆, 𝜙
Mapping function:
𝑢 𝜆, 𝜙
𝑣 𝜆, 𝜙
,
vector form: 𝐮 𝝀
Mathematical setup
• Local properties of this mapping 𝐮 𝝀
– Differential vector 𝝀
𝐡=
𝜕𝑢
𝜕𝜙
𝜕𝑣
𝜕𝜙
,𝐤=
𝜕𝑢
1
𝜕𝜆
𝜕𝑣 cos 𝜙
𝜕𝜆
• Conformal mapping
spherical coordinates are non-Euclidean
equal steps in 𝜆 travel different distances on
the sphere depending on 𝜙
(1)
𝜋
𝜋
cos
− sin
2
2 = 𝟎 −𝟏 𝐤
𝐡=
𝜋
𝜋
𝟏
𝟎
sin
cos
2
2
(2)
𝜕𝑢
𝜕𝜙
(3)
=
𝜕𝑣 1
−
𝜕𝜆 cos 𝜙
,
𝜕𝑣
𝜕𝜙
=
𝜕𝑣 1
𝜕𝜆 cos 𝜙
Cauchy-Riemann equations for mapping a sphere to a plane [Hilbert and Cohn-Vossen 1952; Snyder 1987]
Mathematical setup
• Discretize the mapping
– In our case, we can’t derive a close-form solution
– We discretize the mapping by sampling a uniform
grid in (𝜆, 𝜙) indexed by integers (𝑖, j)
Quad
𝑉: entire set of vertices
(𝑖, j) that fall in the
field of view of the
input image
Conformality
𝜕𝑢
𝜕𝜙
=−
𝜕𝑣 1
𝜕𝜆 cos 𝜙
,
𝜕𝑣
𝜕𝜙
=
𝜕𝑣 1
𝜕𝜆 cos 𝜙
(3)
• We form conformality constraints on the mesh
by discretizing the Cauchy-Riemann equations
(3), giving
𝑢𝑖,𝑗+1 − 𝑢𝑖,𝑗 = −(𝑣𝑖+1,𝑗 − 𝑣𝑖,𝑗 )/ cos 𝜙𝑖,𝑗
(4)
𝑣𝑖,𝑗+1 − 𝑣𝑖,𝑗 = −(𝑢𝑖+1,𝑗 − 𝑢𝑖,𝑗 )/ cos 𝜙𝑖,𝑗
(5)
Conformality
• All quads on the viewing sphere are not equal
in size
– We weight the constraints by cos 𝜙
𝐸𝑐 =
+
(𝑖,𝑗)∈𝑉
𝑤𝑖,𝑗 2 ( 𝑣𝑖+1,𝑗 − 𝑣𝑖,𝑗 + cos 𝜙𝑖,𝑗 (𝑢𝑖,𝑗+1 − 𝑢𝑖,𝑗 ))2
(𝑖,𝑗)∈𝑉
(6)
𝑤𝑖,𝑗 2 ( 𝑢𝑖+1,𝑗 − 𝑢𝑖,𝑗 + cos 𝜙𝑖,𝑗 (𝑣𝑖,𝑗+1 − 𝑣𝑖,𝑗 ))2
Straight lines
• We define
𝐿: the set of all line constraints marked by user
𝐿𝑓 : orientation line constrain
points lie on a line
(line is on the viewing sphere)
𝑚𝑎𝑝𝑝𝑖𝑛𝑔
points are collinear
on the image plane
• Virtual vertex
– Midpoint of line-quad intersection
– We define a virtual vertex
as a bilinear interpolation
of the surrounding vertices.
Sphere
Virtual vertex
Straight lines
• We compute the position of a virtual vertex on
the sphere, and its bilinear interpolation
coefficients (a, b, c, d), as shown in Figure
𝑢𝑖+1,𝑗+1
𝑢𝑖+1,𝑗
𝑢𝑖,𝑗+1
𝑢𝑖,𝑗
on which we place our line constraints.
Straight lines
•
: two line endpoints
• For the rest of this section we drop the superscript 𝑙
and assume the u variables correspond to virtual
vertices for a particular line
Straight lines
𝑇
• Distance 𝐮 − 𝐮𝑠𝑡𝑎𝑟𝑡 𝐧(𝐮𝑠𝑡𝑎𝑟𝑡, 𝐮𝑒𝑛𝑑 )
(7)
• We therefore define the line energy for a constrained line 𝑙 as
(8)
This energy function is non-linear, so we simplify the line energy in two ways,
each of which can be solved linearly
Straight lines
• We can express the energy function as
another way :
Equation (10) is the normalized length of the projection of (𝐮
− 𝐮𝑠𝑡𝑎𝑟𝑡 ) onto (𝐮𝑒𝑛𝑑 − 𝐮𝑠𝑡𝑎𝑟𝑡 )
Straight lines
(8)
• Two ways to simplify the line energy
– By fixing the normal vector in equation (8)、(9)
Smoothness
(13)
(14)
Spatially-varying constraint weighting
• Line endpoint weights :
• Salience weights :
• Face detection weights :
– face detection algorithm of Viola and Jones [2004],
as implemented in OpenCV [Bradski and Kaehler2008]
• Total weight
Total energy and Optimization
• Total energy function
𝐸 = 𝑤𝑐 2 𝐸𝑐 + 𝑤𝑠 2 𝐸𝑠 +𝑤𝑙 2
𝐸𝑙𝑜 +
𝐸𝑙𝑑
(16)
• The quadratic energy function at each
iteration of our algorithm results in a sparse
linear system
Ax = 0
• PARDISO sparse direct solver
Results
Future work
• Developing a completely automatic system
that identifies salient linear structures using
line detection algorithms
• Improved by using a more sophisticated
salience measure