Shapelets Correlated with Surface
Normals Produce Surfaces
Peter Kovesi
School of Computer Science & Software Engineering
The University of Western Australia
Shape from Shading and Shape from Texture attempt to
• Calculate the surface normal direction at each point in
the image, and then
• Infer the surface shape from the surface normals.
Surface normals are usually calculated in terms of slant and tilt
N
slant s
eye
surface
Slant is the angle between the viewing
direction and the surface normal
Tilt is the angle of the projection of the surface normal on a
plane perpendicular to the viewing direction
projection of surface normal
tilt t
N
slant s
viewer
plane perpendicular
to viewing direction
Problem:
Tilt often has an ambiguity of p
Problem:
Tilt often has an ambiguity of p
Problem:
Regularization schemes are often used to deal with noise.
However, they typically involve the minimization of an
Energy Term that has mixed units
2
ˆ
ˆ
E   f x  f x  f xx   I  nˆ  s  ...
2
surface gradient error
(highly nonlinear with surface angle)
2
bending energy
shading error
Problem:
Regularization schemes are often used to deal with noise.
However, they typically involve the minimization of an
Energy Term that has mixed units
2
ˆ
ˆ
E   f x  f x  f xx   I  nˆ  s  ...
2
surface gradient error
(highly nonlinear with surface angle)
2
bending energy
shading error
• The weighting parameters have to act as unit conversion operators.
• The appropriate values will be dependent on scale.
• This can be problematic if the regularization is being applied within a
multi-scale framework.
• Discontinuities have to be automatically detected and treated specially.
Alternative approach:
Project the surface gradients onto a set of integrable
basis functions and reconstruct the surface from these.
Frankot and Chellappa [1988] first introduced this
approach using Fourier basis functions.
Features of work presented here:
• Localized, non-orthogonal basis functions are used.
• Correlations with basis functions are performed with
respect to slant and tilt rather than in terms of gradient in
x and y.
• Ambiguities of p in tilt can be considered.
• Low quality reconstructions are possible with slant
information alone.
Decomposing and Reconstructing Signals
with Basis Functions
signal
band-pass filters/wavelets
filtered at
scale 2
filtered at
scale 4
filtered at
scale 6
Reconstruction obtained by summing
band-passed signals over 6 scales
We only have the gradient of the surface…
We only have the gradient of the surface…
Differentiation is linear
surface gradient
f  g f  g
surface

band-pass filter
We only have the gradient of the surface…
Differentiation is linear
surface gradient
f  g f  g
surface

band-pass filter
Integral of band-pass filter
(shapelet gradient)
A band-passed version of the signal can be obtained
from a correlation performed between the gradient of
the signal and integral of the band-pass filter.
Choice of Shapelet
• The gradient of shapelet function must satisfy the
admissibility condition of zero mean.
• Must ensure preservation of phase information in the
signal.
• The bank of shapelet filters should provide a uniform
coverage of the signal spectrum so that it is faithfully
reconstructed.
These requirements are satisfied by a Gaussian
Note that correlating the gradient of one shapelet filter
with the gradient of the signal will correspond to
extracting a band of frequencies from the signal
gradient.
Note that correlating the gradient of one shapelet filter
with the gradient of the signal will correspond to
extracting a band of frequencies from the signal
gradient.
The signal gradient spectrum differs from the original
signal spectrum in that its amplitudes are scaled by the
frequency.
Note that correlating the gradient of one shapelet filter
with the gradient of the signal will correspond to
extracting a band of frequencies from the signal
gradient.
The signal gradient spectrum differs from the original
signal spectrum in that its amplitudes are scaled by the
frequency.
The sum of the shapelet gradient transfer functions
should form a curve that is inversely proportional to
frequency to counteract the scaling of the gradient
spectrum. The net result will then be an even coverage
of the original signal spectrum.
Shapelet bank formed by
five Gaussians with height
proportional to scale.
Transfer functions of the
gradients of the shapelets
above.
The sum forms a curve that
is approximately inversely
proportional to frequency.
A bank of shapelets will not form
an orthogonal basis set
This does not matter if we have a strongly redundant basis
set. (Achieved by using a continuous wavelet transform.)
A non-orthogonal basis set can be considered to be the sum
of several orthogonal basis sets.
- We get accurate reconstruction up to a scale factor
• The scaling of the reconstruction due to the redundancy
of the basis set can be determined and corrected if
required.
• Overcomplete basis sets have the advantage that small
changes in local features result in correspondingly small
changes in the basis coefficients.
• Overcompleteness also provides robustness to noise.
In Summary:
• Correlate the gradients/normals of the surface with the
gradients/normals of a set of basis functions.
• These correlation results correspond to bandpassed
versions of the original signal.
• Reconstruct surface by summing correlation results.
Perform correlation in terms of slant and tilt separately
- slant and tilt are calculated by different means.
- slant and tilt have differing degrees of uncertainty.
Correlation of Slant
Slant lies in the range 0 - p/2 which only provides
gradient magnitude information.
  tan(s )
The slant correlation at scale i is formed by
gradient of shapelet at scale i
Ci   f  si
gradient of surface
With no tilt information this correlation measure
matches positive and negative gradients equally.
(No distinction between concave or convex shape)

Correlation of Tilt
• If gradient magnitudes match and tilts match
correlation is +ve
• If gradient magnitudes match and tilts differ by p
correlation is -ve
• If tilts differ by p/2 correlation is 0
Use the cosine of the tilt difference as a correlation measure
Cti  cos(t f  t si )
tilt of surface
tilt of shapelet at scale i
Overall Correlation
Reconstruction
Ci  Ci  Cti
R  Ci
i

synthetic test surface
needle diagram of
surface normals
surface normals
reconstruction over 4 scales
reconstruction over 2 scales
reconstruction over 6 scales
Range data from OSU database
Shapelet reconstruction using the
gradients of the range data
Slant and tilt with Gaussian noise of 0.3 radians standard deviation
Shapelet reconstruction
Frankot Chellappa reconstruction
Terzopoulos multigrid reconstruction
Allowing for Tilt Ambiguity of p
Ct  cos (t f t s )
2
When tilts are aligned
value is 1
When tilts are orthogonal value is 0
When tilts are opposite value is 1
… results in rectification of the band-pass components
of the surface
Reconstruction with a tilt ambiguity of p
bumps surface
reconstruction of
ramps surface
reconstruction
Reconstruction with slant data alone
Slant-only reconstruction of bumps
Slant-only reconstruction of ramps
Sinc function
Slant-only reconstruction of sinc
Slant-only reconstruction of face
…after segmentation and negation
Real data: Surface normals from texture
Shapelet reconstruction
Shapelet reconstruction
Sand ridges:
Slant set to p/4 at all points
Tilt set to 0 for white regions, p for black regions
Shape from Occluding Contours
Surface normals at occluding
contours are perpendicular to
viewing direction.
This is a powerful cue to shape.
Manually marked contours
A crude approximation of the surface normals in the scene…
At all points slant is set to 0, except along contours where
slant is set to p/2, and tilt is set to be perpendicular to contour.
This gradient field is not integrable.
Shapelet reconstruction
Manually drawn contours
Conclusions
• Reconstructions from surface normals using shapelets is
simple and the process is robust to noise.
• The use of basis functions implicitly imposes a continuity
constraint in the reconstruction.
• Being able to treat slant and tilt separately permits data with
tilt ambiguity, or no tilt data, to be considered.
• Much can be made from limited amounts of surface normal
data. This allows some some aspects of structure to be
deduced from line drawings.