Seminars reporter :Shape From Shading
By: ZHAN Tao (ID: 01130447) & QIAN Kun (ID: 02341735)
140.429 Image Processing
Seminars reporter : Shape From Shading
By: ZHAN Tao (ID: 01130447)
QIAN Kun (ID: 02341735)
KEY WOEDS:
Shape from shading, Integration Methodology, Fast Marching Method, Digital
image matching, the mathematical model, Geometric Model, Row Integration,
Synthetic Image, Lunar Image,
ABSTRACT:
Shape from shading (SfS) is a fundamental problem in Computer Vision. This report
briefly gives the ideas of the questions: What is SfS? What is the theory of SfS? What
is the history of SfS? How to process the SFS? Finally, use some simple examples to
specify the SFS.
WHAT IS SFS?
Shape from shading (SFS) uses the pattern of shading in a single image to infer the
shape of the surface in view. A typical example of shape from shading is astronomy,
where the technique is used to reconstruct the surface of a planet from photographs
acquired by a spacecraft.
The reason that shape can be reconstructed from shading is the link between image
intensity and surface slope. The radiance at an image point can be calculated with the
surface normal, the direction of the illumination (pointing towards the light source)
and the albedo of the surface, which is typical of the surface's material. After
calculating the radiance for each point we get the reflectance map of the image.
The parameter of the reflectance map might be unknown. In this case we have to
estimate the albedo and illuminant direction. Albedo and illuminant can be computed,
by assuming we are looking at a lambertian surface, with help of the averages of the
image brightness and its derivatives. From the reflection map and by assuming local
surface smoothness, we can estimate local surface normal, which can be integrated to
give local surface shape.
THE THEORY OF SFS
Smooth variations in the brightness or shading of objects in an image are often an
important cue in human vision for estimating the shape of depicted objects. This
project is concerned with modelling this process and setting up prototype systems for
automatically recovering surface shape from image shading: the results can
potentially be applied to the computer interpretation of satellite, medical and SAR
images, as well as automated visual inspection of industrial parts.
Seminars reporter :Shape From Shading
By: ZHAN Tao (ID: 01130447) & QIAN Kun (ID: 02341735)
140.429 Image Processing
Reconstructing shape from shading can be reduced to solving a first-order, nonlinear
partial differential equation. Challenges in solving this problem are: to reduce the
amount of prior information required, such as knowledge of the light source direction;
to develop a robust, general-purpose solver; and to develop massively parallel
techniques. Our longer-term aim is to fuse information from a shape-from-shading
solver with information from other "shape-from" modules, in particular shape from
stereo. We have selected this goal, as the capabilities of the differing modules are
often complementary. For example, shape from stereo works best on images
exhibiting edges and textures, whereas shape from shading performs optimally on
images exhibiting smooth brightness variation.
HISTORY OF SFS
Shape-from-Shading (SfS) is one of the fundamental problems in Computer Vision.
First introduced by Horn in the 1970s, its goal is to solve the image irradiance
equation, which relates the reflectance map to image intensity. Due to difficulties in
solving the equation, most of the works in the field add simplifying assumptions to
the equation. Of particular importance is the common assumption that scene points
are projected orthographically during the photographic process.
Since the first SfS technique was developed, many different approached have
emerged. Each algorithm works well for certain images, but performs poorly for
others.
TYPE OF PROCESSION
There are many types to process the SFS now, as we research, the common
methodologies for processes the SFS are: Digital image matching, The Integration
Methodology, Fast Marching Method
The Integration Methodology
The main concern of this work is the integration of the dense depth map obtained
Seminars reporter :Shape From Shading
By: ZHAN Tao (ID: 01130447) & QIAN Kun (ID: 02341735)
140.429 Image Processing
from the shape from shading with a sparse depth data from stereo for the
reconstruction of 3D visible surfaces with accurate metric measurements. This
integration has two advantages. First, it helps in resolving, to some extent, the
ambiguity of the 3D visible surface discontinuities produced by shape from shading
due to highly textured regions. Second, it compensates for the missing metric
information in the shape from shading by employing the data obtained from stereo,
which work better at the highly textured regions. On the other hand, the sparse depth
data from stereo does not contain all the depth information about the surface, so it
cannot be used directly to represent the visible surfaces accurately. The integration
process is based on propagating the error divergences between the available depth
data from stereo and the shape from shading throughout the remaining measurements
where only shape from shading data is available. This can be done in three steps; first,
the error difference in the depth measurements between the data sets is calculated.
Second, we fit a surface to that error difference. Finally, the resultant surface is used
to correct the shape from shading data. The surface fitting process, which is cast as a
function approximation, is carried out in this paper using neural networks.
Fast Marching Method
Sethian first introduces the Fast Marching Method in 1996 [12]. It applies to a front
propagating normal to itself with a speed F = F (i, j) . The main idea of the method is
to methodically construct the solution using only upwind values. Let the solution
surface T ( i, j) be the time at which the curve crosses the point (i ,j) then it satisfies
|▽T| F=1 , namely, the Eikonal equation. The physical meaning is that the gradient of
arrival time surface is inversely proportional to the speed of the front. For an upwind
scheme, the approximation to the gradient can be written as
where the difference operator notation, for example,
is
employed. Rouy and Tourin [2] introduce an iterative algorithm to solve equation by
referring to a grid point and its four neighbors. The method is explicitly stated in [5].
The upwind difference structure allows the information to propagate in one way only,
meaning, from smaller values of T to larger values. It builds the solution outward
from the smallest T value and then steps away from the boundary conditions in an
upwind direction. To solve this non-linear system, an optimal ordering of the grid
points is produced based on the A* search. Starting with an initial position for the
front, the method systematically marches the front outwards one grid point at a time.
The update procedure for this method can be organized as follows:
1. Label the boundary value points as Known.
2. Label as Trial all points that are one grid point away.
3. Label as Far all other grid points.
4. Begin loop: Let A be the Trial point with the smallest T value.
5. Add the point A to Known; remove it from Trial.
6. Label as Trial all neighbors of A that are not Known. If the neighbor is in Far,
remove, and add to the set Trial.
7. Recalculate the values of T at all Trial neighbors of A by solving the quadratic
equation.
8. Return to Step 4.
Seminars reporter :Shape From Shading
By: ZHAN Tao (ID: 01130447) & QIAN Kun (ID: 02341735)
140.429 Image Processing
Followed we introduce the emphasis model: Digital image matching mathematical
model.
Digital image matching
Digital image matching is an automatic method for the reconstruction of object
surface information from the gray values of digital images. For the automatic
derivation of object heights it is usually necessary to estimate the positions of
conjugate points in all images. A necessary prerequisite to obtain a reliable solution
when performing digital image matching is the presence of sufficient, non-periodic
image texture. However, not all images fulfill this demand. Furthermore, if only a
single image of the object is available, digital image matching cannot be applied. It is
necessary to investigate surface reconstruction method, which allow for the estimation
of object surface parameters from images with poor texture and from single images.
Shape from shading (SFS) is one of these methods, that it directly relates image gray
values to inclination of the corresponding surface patch relative to the direction of
illumination and direction to the imaging sensor. Shape from shading has been
investigate in different research area: Computer Vision, Photogrammetry and
Astrogeology.
In Computer Vision SFS has been developed for surface reconstruction in close range
images (e.g. Zhang et al., 1999;Tsai and Shah, 1994; Lee and Kuo 1993) and in
Photogrammetry, SFS has been studied for DTM reconstruction or refinement
(Heipke et al. 2000 ). In the field of Asterogeology, SFS has been developed for the
geometric This paper is concerned with the implementation of a SFS technique for
automatically generating a digital terrain model(DTM) using a single digitized aerial
photograph of a terrain area with low signal(information) content.
The mathematical model.
The mathematical model for the SFS is established based on the fact that the pixel’s
gray level variation in image space are proportional to the shading intensity variations
of the illumination intensity and the direction of the incident light with respect to the
local surface orientation as well as the direction of the incident light direction and the
terrain albedo. In classical SFS the reconstruction of surface slopes is ambiguous,
because there is only one gray value observation per surface element while surface
orientation has two degree of freedom. A solution, however, can be obtained by
introducing various surface constraints (Zhang et al., 1999).
Another possibility to overcome this ambiguity is the introduction of a radiometric
and a geometric surface model.
Radiometric Model
The radiometric model creates the relation between the image gray values and the
corresponding suface patches. The image gray values are influenced by the radiance
and wavelength of the incident illumination, atmospheric effects, surface reflectance
properties and sensor characteristics. The basic equation of SFS is derived by
assuming a constant albedo ( A ) and lambertian reflectance properties for the whole
surface. In this case, the gray value G (x ',y ') at the position x', y ' in image space
only depends on the angle between illumination direction S
Seminars reporter :Shape From Shading
By: ZHAN Tao (ID: 01130447) & QIAN Kun (ID: 02341735)
140.429 Image Processing
and the surface normal vector V (fig1):
In the lambertian model the surface looks equally bright from every viewing direction.
Another law that is good description of the light scattering behavior of law-albedo
surfaces is the lommel-seeliger law. In this model the radiance observed at a sensor
comes from light scattered by all particles in the medium lying within the field of
view of the sensor:
Geometric Model
In SFS, the geometric model describes a piecewise smooth surface. The geometric
model consists of a DTM grid. The grid nodes is defined by Xk, Y l, Z k l .The mesh size
of the grid depends on the roughness of the terrain. A height Z i at an arbitrary
position, dX dY is interpolated from the neighboring grid heights, e.g. by bilinear
interpolation :
Where:
Z1, Z2, Z3, Z4 : heights of the neighboring grid
Then,
Seminars reporter :Shape From Shading
By: ZHAN Tao (ID: 01130447) & QIAN Kun (ID: 02341735)
140.429 Image Processing
The observed gray value (eq. (1), (2)) become a function of the Mesh height, Z k l and
the albedo .
Observation Equation
If the light source direction, the interior and exterior orientation are assumed to be
known quantities, the Z k l and A are treated as unknowns. Subsequently, each object
surface element is projected in to image space using the collinearity equations of
photogrammetry and initial values of grid nodes height , Z k l.
Next, the observed gray value, (Z k l) is resample for resulting position. The
corresponding observation equation read:
One such equation can be formulated for every object surface element projected into
every available image. The unknowns are then computed in an iterative least squares
adjustment.
Implementation from digital image match
The implementation SFS algorithm is tested with the simulated and the real data. In
this research work, the lamebrain model is utilized for modeling the terrain
reflectivity property. It is very difficult to choose good test image for SFS algorithms.
A good test image must match the assumptions of the algorithms, e.g. lamebrain
reflectance model, constant albedo value. It is not difficult to satisfy these
assumptions for synthetic image. In real image, there will be errors to the extent that
these assumptions are not matched.
In this research, simulate data was generated using a predefined bilinear surface and
for real data an aerial photograph of a smooth hilly terrain with low information
content was chosen.
Surface Reconstruction Using Synthetic Image
Synthetic image in scale of 1:40000 was generated by a raytracing algorithm, using
the synthetic DTM, together with a constant value for the surface albedo.
The exterior orientation of the image and light source position were considered as
known values. The unknown mesh heights were reconstructed in a least squares
adjustment according to eq.(6). The RMSE for the reconstructed surface was
estimated to be equal to 3cm.
Seminars reporter :Shape From Shading
By: ZHAN Tao (ID: 01130447) & QIAN Kun (ID: 02341735)
140.429 Image Processing
Surface Reconstruction Using Real Image
One black and white aerial image with an image scale of approximately 1:40000 of
poorly texture area in Iran was used(fig.2). The image was digitized using
photogrammetric scanner with a pixel size of 14µm, resulting in a ground sample
resolution of about 0.56m. The interior and exterior orientation were determined using
digital stereo plotter. The illumination direction was calculated from known time of
the image acquisition and geographical coordinates of surface area. Themaximum
height difference within the chosen test area is about 50m.
For this area a DTM with a mesh size of 3m was measured analytically(fig.3).Then
average albedo value was estimate from the digital image by considering the
measured DTM.
The RMSE for the reconstructed surface for this image was estimated to be equal to
4m.
It should also be mentioned that our implemented SFS model for the real data was
accompanied by a preprocessing stage by which the influential noise was significantly
reduced. The results show the gray shade values are also significantly influenced by
the other factors such as the non-uniform terrain albedo, atmosphere, etc. which have
Seminars reporter :Shape From Shading
By: ZHAN Tao (ID: 01130447) & QIAN Kun (ID: 02341735)
140.429 Image Processing
not been included into our SFS functional model.
Shape from shading has been investigated using two different data: synthetic and real
data. DTM heights were determined from aerial images of poor texture. The RMSE
for the reconstructed surface for the simulated data and the real data were estimated to
be equal to 3cm and 4m respectively. The result for the real image shows that the
lambertian reflection does not sufficiently describe surface reflectance properties.
Surfaces with unknown and varying albedo must be considered. This can be
formulated within the frame of multi image shape from shading.
There are several possible directions for future research:
· Reflectance models used in SFS method are too simplistic. Recently, more
sophistical model has been proposed(e.g.Clark, 1992). This not only includes
more accurate model for lambertian but also includes replacing the assumption of
orthographic projection with perspective projection which is a more realistic model of
cameras in the world.
· Another direction to improve the results of SFS is the combination of shading with
some other techniques, such as: image matching, shape from shadow and etc. or use
the results of SFS to improve the results of other techniques.
SOME SIMPLE EXAMPLES OF SFS
Shape-from-shading (also known as photoclinometry) is a method for determining the
shape of a surface from its image. For a surface of constant albedo, the brightness at a
point (x,y) in the image is related to the gradients (p,q) by the following expression
i(x,y) = a R[p(x,y),q(x,y)]
where R is the reflectance map, p = dz/dx and q = dz/dy are the partial derivatives of
the surface in the x- and y-directions, and a is a constant that depends on the albedo,
the gain of the imaging system and other factors. The above expression also assumes
that any additive offsets, e.g., due to atmospheric scattering, have been removed.
A variety of methods have been developed for inverting the above equation (see Horn
1990). The next section describes a simple method that provides satisfactory results in
many planetary imaging scenarios. It is based on some early ideas described by Horn
(1977).
Row Integration
The reflectance map depends on the position of the light source, the observer, and the
type of surface material (Horn 1981). It can be thought of as a lookup table that gives
the brightness as a function of the gradients. For Lumberton surfaces the brightness is
proportional to the cosine of the angle between the vector that is normal
(perpendicular) to the surface and the vector in the direction of the light source. As
noted by Pentland (1988), if the angle between the vector in the direction of the light
source and the vector in the direction of the observer are more than 30 degrees apart
and the surface is not too rough the reflectance map can be approximated by a linear
relationship. If the image is rotated so that the vector that points to the sun is in the xz plane, it can be shown that
i(x,y) ~ a [sin(s) p(x,y) + cos(s)]
where s is the zenith angle of the sun. The constant scale factor a is difficult to
determine directly without ground truth (i.e., ground targets with known albedo and
slope). However, since in most images the gradients are more-or-less uniformly
distributed in all directions, the expected value of the gradient in the x-direction E[p]
Seminars reporter :Shape From Shading
By: ZHAN Tao (ID: 01130447) & QIAN Kun (ID: 02341735)
140.429 Image Processing
~ 0 and so the average image brightness
E[i] ~ a cos(s).
This then allows us to estimate the scale factor
a = E[i] / cos(s).
The elevation map z(x,y) can be obtained iteratively, row-by-row as
z(x,y) = z(x-1,y) + [i(x,y) - a cos(s)] / a sin(s)
where z(0,y) are the boundary values. If the boundary values z(0,y) are unknown, we
can minimize the mean-squared elevation difference between rows by subtracting the
average row elevation from the elevations in the row.
Synthetic Image Example
The image below (left) is of that of a conical surface where brightness represents
height. The image below (right) is a shaded rendition of the cone viewed from below
(azimuth = 180 deg., zenith = 70 deg.) using a Lambertian reflectance map with the
sun to the left (azimuth = 270 deg., zenith angle = 45 deg.).
The figure below (left) shows a synthetic image generated with the sun directly
overhead (zenith = 0 deg.). The image below (right) is the elevation surface estimated
from that image. The poor performance of shape-from-shading algorithms at high sun
angles (low zenith angles) is like that experienced by image analysts under similar
conditions.
As the zenith angle increases the performance improves. The figure below (left)
shows a synthetic image generated with the sun at an azimuth angle of 270 deg. and a
zenith angle of 30 deg. The image below (right) is the elevation surface estimated
from that image.
As the zenith angle continues to increase the true shape of the surface begins to
emerge. The figure below (left) shows a synthetic image generated with the sun at an
azimuth angle of 270 deg. and a zenith angle of 60 deg. The image below (right) is the
elevation surface estimated from that image.
Seminars reporter :Shape From Shading
By: ZHAN Tao (ID: 01130447) & QIAN Kun (ID: 02341735)
140.429 Image Processing
Comparing perspective views of the original and estimated elevation surface from the
last example shows the method capable of recovering the general shape of the surface.
The slight tilt in the surface in the direction of the sun is a common artifact of this
kind of algorithm.
Lunar Image Example
The image below (right) is a portion of frame LUE50467 acquired by the Clementine
spacecraft in its 218-th orbit around the moon. The image has been rotated so that the
sun is to the left. The middle image is the elevation surface computed by the shapefrom-shading algorithm. One method of validating this result is to compute a shaded
rendition from the elevation surface under the same illumination and viewing
conditions as the original image. The shaded rendition (right) compares favorably
with the original image (left).
Once the elevations are known one can generate simulated perspective views of the
Seminars reporter :Shape From Shading
By: ZHAN Tao (ID: 01130447) & QIAN Kun (ID: 02341735)
140.429 Image Processing
scene. By generating two views that are within a few degrees of one another, 3-D
renditions in the form of synthetic stereo images can be created. The image below is a
synthetic stereo pair generated from directly overhead. It is rendered anaglyphically
with the left image encoded in red and the right image in blue as well as side-by-side
for those that do not have colored glasses.
To view the images in 3-D, look away from picture and focus at a point in the
distance. Then shift your gaze to the picture. You should see three images.
Concentrate (but don't focus) on the middle image and try to bring different parts of
the view into correspondence. This will take some practice. If you have difficulty
seeing 3-D, try moving back from the screen.
The image below is an oblique stereo pair. A parallel projection algorithm is used
(Foley and Van Dam 1983) that simulates how the scene would appear at a distance
as seen through a zoom lens. The observer is located at a azimuth of 180 deg. and a
zenith angle of 45 deg.
Seminars reporter :Shape From Shading
By: ZHAN Tao (ID: 01130447) & QIAN Kun (ID: 02341735)
140.429 Image Processing
CONCLUSION
When stereo imagery does exist, shape-from-shading provides an alternative method
for extracting terrain data. Although its use has been limited primarily to constantalbedo planetary mapping applications (i.e., where the surface is covered more-or-less
by the same material) new algorithms currently under development will extend to the
general case in terrestrial imaging applications where the albedo is not constant.
Seminars reporter :Shape From Shading
By: ZHAN Tao (ID: 01130447) & QIAN Kun (ID: 02341735)
REFERENCE
Davis, P.A., Soderblom, L.A., 1984.Modeling crater
topography and albedo from monoscopic Viking Orbiter
images. Journal of Geophysical Research 89 (B11), 9449-9457.
Brooks, M.J., Horn, B.K.P., 1985. Shape and Source from
Shading, Proc. Jnt'l Joint Conf. Artificial Intelligence, pp. 932936.
Hashemi, L., 2001. Automatic DTM generation using Shape
from Shading. M.Sc. Thesis, Department of Geomatic
Engineering, University of Tehran, Iran.
Heipke, C., 1992. Integration of Digital Image Matching and
Multi Image Shape from Shading. Informatik aktuell, Springer,
Berlin, pp. 367-374.
Heipke, C., Piechullek, C., 1996. DTM Refinement Using
Multi Image Shape from Shading. InArchPhRs, (31) , B3/III ,
pp. 644-651.
Figure 2: Digitized aerial image
Figure 3: Perspective view of
the reference DTM
140.429 Image Processing