Modern Texture Synthesis
Rachel Heck
heckr@cs.wisc.edu
Abstract
Texture synthesis is a method that takes in a small patch of texture
and produces a new texture patch or block that looks as if it were
created using the same methods as the example texture. In this
paper, I explore two modern texture synthesis techniques presented
in two recent Siggraph papers: “Fast Texture Synthesis Using
Tree-Structured Vector Quantization” and “Image Quilting for
Texture Syntheis and Transfer.” The advantages and
disadvantages of the two techniques are shown through clear
examples generated with implementations of the two techniques.
1. Introduction
Texture is one of the traits of everyday objects that give them their unique look. In
computer graphics, it is often necessary to add texture to digital objects in order to make
them look real. While manually painting a texture on a surface or object is a definite
option, it requires great artistic skill as well as a long period of time spent in front of a
computer. Texture mapping opens up a new option for texturing digital objects. By
mapping a 2D image to an object, you can easily produce detail without the arduous task
of texture painting. Unfortunately, texture mapping with a single small piece of example
texture can easily create unnecessary seams and distortion.
This motivation led to the development of texture synthesis algorithms. A texture
synthesis algorithm is one which takes in an example texture patch as input and produces
a new patch of texture of a desired size which appears as if it could have been produced
using the same technique as the original texture.
The first significant texture synthesis paper was published in Siggraph in 1995
[HEEGER]. In this paper, “Pyramid-Based Texture Analysis/Synthesis,” statistics are
used to coerce the pyramid of a random image to look like a pyramid generated by an
input texture patch. This technique works well on simple stochastic texture patches.
Even better, this technique is able to produce 3D textures, which do not create distortions
when mapped to a digital object. Later algorithms, in particular the ones presented in
[EFROS99] and [PORTILLA], focused on creating texture patches for more structured
and complex texture samples.
In this paper, I will describe and analyze the results of two recent texture synthesis
techniques. The first, “Fast Texture Synthesis Using Tree-Structured Vector
Quantization,” uses Markov Random Fields to generate tileable, stochastic textures one
1
pixel at a time [WEI]. The second, “Image Quilting for Texture Synthesis and Transfer,”
takes a different approach to texture synthesis by using texture blocks as the scale of
processing instead of individual pixels [EFROS01]. In the next section, I will describe
the two algorithms in detail and then conclude with a comparison and discussion of the
results and algorithms.
2. The Algorithms
In this section, the algorithms that are being compared will be explained in detail. Each
explanation focuses on the texture synthesis process described in one particular paper.
For details on the applications to which the original authors applied their techniques and
further algorithm improvements discussed by the authors, please see the original papers.
2.1 Fast Texture Synthesis Using Tree-Structured Vector
Quantization
In 2000, Wei and Levoy published “Fast Texture Synthesis Using Tree-Structured Vector
Quantization” at Siggraph [WEI]. In this paper, they describe a texture synthesis
algorithm based on Markov Random Fields. The algorithm produces high-quality
stochastic textures by computing the value of each pixel in the synthesized image through
a deterministic searching process in the input image. In the second half of the paper, they
explain how tree-structured vector quantization (TSVQ) can be used in a search method
to help speed up their algorithm. While they claim that the tree-structured vector
quantization searching technique greatly improves the processing time, the overall texture
synthesis algorithm does not change if using the simple exhaustive search method
described in the first half of their paper. Thus, I will not place emphasis on the TSVQ
method, as my purpose is to describe the basic texture synthesis algorithm.
The great strength of Wei and Levoy’s algorithm is that it can be run in a single
resolution or at multiple
Wei and Levoy’s 2000 Algorithm -- Single Resolution:
resolutions using image
pyramids which can improve
(1) Produce a random white noise image of the desired
the results of the final texture
size.
as well as speed up the texture
synthesis generation time. In
(2) Match the random image’s histogram to the input
image’s histogram.
addition, their algorithm is
guaranteed to produce tileable
(3) Replace each pixel in the random image in raster
results even when the original
scan order with a pixel from the input image that best
texture is not.
2.1.1 Single Resolution
At the heart of Wei and
Levoy’s texture synthesis
matches it. To compute the best match, use the SSD of
the neighborhood around the pixel.
Figure 1: A quick summary of Wei and Levoy’s Algorithm in a
single resolution [WEI].
2
technique is their single resolution algorithm. This algorithm is composed of a few
simple steps (see Figure 1 for a quick summary). First, the algorithm creates a random
white noise image of the size m x n, where m and n are the dimensions of the texture
patch to be generated. Since a true random white noise image will not have the same
color distribution as the input image, the random image must be modified so that its
histogram matches that of the input image (see Figure 2). This is done as described in
Heeger and Bergen’s paper “Pyramid-Based Texture Analysis/Synthesis” (see Figure 3
for the algorithm) [HEEGER]. Finally, each pixel in the random image is replaced with a
pixel from the input image which best matches it. The pixel in the input image with the
lowest Sum-of-Squared Differences (SSD) in an S x S neighborhood when compared
with the pixel in question from the random image and its neighbors is said to be the best
matching pixel, where S is specified by the user.
Figure 3: Heeger and Bergen’s Method for Histogram
Matching [HEEGER]
There are two things to pay special attention to when
computing the SSD of the pixels’ neighborhoods. First,
when computing the value for any particular pixel in
Figure 2: For Wei and Levoy’s texture synthesis algorithm, (a) the picture on the left is an example
input image. (b) The picture in the middle is an example of a white noise image generated for step 1 of
the single resolution algorithm. (c) The image on the right is the random image coerced to match
histograms with the input image (step 2 of single resolution algorithm).
Histogram Matching
(1) Read in the image that is to be coerced
(image1) and the image to match to (image2).
(2) Compute the histograms for the two images.
(3) For each image, compute the cummulative
distribution function (cdf) which maps pixel
values to the interval [0,1] evenly.
(4) Compute the inverse cummulative
distribution function (icdf) for image2 which
maps the interval [0,1] back to a pixel value.
the random image, only some of the
pixels in its neighborhood have been
given definite values. In other
(5) Compute the value of each pixel in image1
words, when computing the value
by applying image1’s cdf to pixel currently at
for the pixel at (x,y) in the random
that location and then applying image2’s icdf to
image, only pixels in raster scan
the result to get the final pixel value.
order before the one at (x,y) have
been computed already. The part of
a pixel's neighborhood that has already been computed is called its casual neighborhood.
Only the casual neighborhood should be used to compute the amount of match for a pair
of pixels, with one exception that will be explained next.
3
Second, the neighborhood around a pixel on the edge of an image must be computed in a
slightly different manner. For the random image, the edges are treated toroidally (i.e. a
pixel on the left edge of the image is a neighbor of the pixel in the same row of the image
but on the right edge; the corresponding fact is true of the top and bottom edges). By
treating the edges in the random image toroidally, you guarantee that the generated
texture will be tileable. When the toroid rule is used to find the value for a neighbor, the
casual neighborhood rule is ignored. In this way, the random pixels on the right and
bottom edges of the image are used to compute the final synthesized image thus adding a
bit of randomness to the result.
For the input image, edges must be treated differently. An input image is not guaranteed
to be tileable. So, if the image where to be treated toroidally, discontinuities in the
generated texture could arise. To solve this problem, only pixels with a completely
interior neighborhood are considered for placement into the generated image.
2.1.2 Multiple Resolutions
Wei and Levoy make only slight modifications to their single resolution algorithm in
order to make it work in
Wei and Levoy’s 2000 Algorithm – Multiple Resolutions:
multiple resolutions (see
Figure 4 for a quick
(1) Do steps 1 and 2 of the single resolution algorithm.
summary). First, the
random image for the
(2) Compute the Gaussian pyramid of the input image
multiple resolution version
and the random image.
is computed in the same
way as it was in the single
(3) Starting at the lowest resolution level of the pyramids,
resolution version. The
do step 3 of the single resolution algorithm on each level
next step is to compute the
of the pyramids using multi-resolutional neighborhoods
Gaussian pyramid for each
to compute a pixels match.
of the images (i.e. the
histogram-matched
random image as well as
the input image). Finally, for each level of the pyramid starting at the lowest resolution,
the new level is computed by matching neighborhoods as in the single resolution
algorithm. The one difference is that neighborhoods are computed in a slightly different
manner. The neighborhood for pixel (x,y) at level L of the random image pyramid is its
casual, toroid S x S neighborhood as well as the non-casual, toroid N x N neighborhood
of pixel (x/2,y/2) at level L+1, where N < S. The same is true for the input image’s
pyramid except that its neighborhoods must again be completely interior.
Figure 4: A quick summary of Wei and Levoy’s Algorithm in
multiple resolutions [WEI].
2.2 Image Quilting for Texture Synthesis and Transfer
Until 2001, most texture synthesis algorithms computed the value of each pixel in the
synthesized texture individually. However, in 2001, Efros and Freeman published their
paper “Image Quilting for Texture Synthesis and Transfer” [EFROS01]. In this paper,
4
they note that most pixels in a synthesized texture do not have a choice about their final
pixel value. Using this observation, Efros and Freeman present an algorithm based on
blocks of texture rather than individual pixels. By using blocks, the texture synthesis
process becomes easier and faster whiles still producing excellent results for both
stochastic and structured textures.
In the second half of Efros and Freeman’s paper, they describe a way to apply their image
quilting algorithm for texture transfer by using the local intensity properties of an
example image to create a new image that looks like the original but is “rendered” in the
style of an input texture. Once again, I will focus here on the basic image quilting
technique. For details on texture transfer, see Efros and Freeman’s paper [EFROS01].
Efros and Freeman’s Image Quilting algorithm consists of three steps (see Figure 5 for a
quick summary). First, a block of size S x S, where S is specified by the user, is chosen
at random from the input image. This block is placed in the upper left corner of the
synthesized image. Next, blocks are placed in raster scan order one-by-one until the
synthesized image is complete. Each block overlaps the one to its left and the one above
it by some number of pixels. In practice, Efros and Freeman found that an overlap region
of S/6 pixels worked well. This overlap region is used to determine which block of the
input image to place in the synthesized
Efros and Freeman’s 2001 Algorithm:
image. For each valid block in the input
image, the Sum-of-Squared Differences
(1) Place a random block of user specified size
(SSD) in the overlap region is computed
S x S from the input image into the new image.
between the synthesized image and the
input image block. One of the blocks
(2) For every other block, randomly choose a
whose SSD is below a particular threshold
block from the input image whose SSD in the
is chosen at random which adds a certain
overlap region is below a threshold.
amount of uncertainty to the process.
(3) Before placing each chosen block, compute
Again, in practice, Efros and Freeman
the minimum cost path through the overlap
found that this threshold did not have to
region and use this as the boundary between
be specified by the user. They used a
the blocks.
threshold that was equal to 110% of the
best error.
Figure 5: A quick summary of Efros and Freeman’s
Image Quilting Algorithm [EFROS01].
These first two steps alone turn out to
produce decent results, but Efros and Freeman found that by allowing the newly placed
block to have jagged edges, better textures could be synthesized. The final step in the
Image Quilting algorithm is to compute the minimum cost path through the error surface
at the overlap region. Then, as this path describes the best natural boundary between the
already placed blocks of the synthesized image and the new block from the input image,
this path can be used as the boundary between the blocks in the final image.
3. Comparison and Results
Figure 6: Some results from the algorithms described in [WEI] and [EFROS01]. The top image is the
input texture. The second row shows results from Wei and Levoy’s single resolution algorithm. The first
image was produced with S = 5 and the second was
5 produced with S = 9. The third row shows results
from Wei and Levoy’s multiple resolution algorithm. The first image was produced with S = 5 and N = 3.
The second image was produced with S = 7 and N = 5. The bottom image was produced with Efros and
Levoy’s image quilting algorithm.
Figure 7: Some results from the algorithms described in [WEI] and [EFROS01]. The top image is the
input texture. The second row shows results from Wei and Levoy’s single resolution algorithm. The first
image was produced with S = 5 and the second was produced with S = 9. The third row shows results
from Wei and Levoy’s multiple resolution algorithm. The first image was produced with S = 5 and N = 3.
The second image was produced with S = 7 and N = 5. The bottom image was produced with Efros and
Levoy’s image quilting algorithm.
Figures 6 and 7 show several results for stochastic textures. Observe that the results from
6
Wei and Levoy’s single resolution algorithm with a 9x9 neighborhood, Wei and Levoy’s
multiple resolution algorithm with a 7x7 neighborhood, and Efros and Freeman’s image
quilting algorithm are very similar. Wei and Levoy’s single resolution algorithm with a
9x9 neighborhood takes considerably longer than the other two algorithms. Moreover,
while being able to use a 7x7 neighborhood in Wei and Levoy’s multiple resolution
algorithm improves the speed of the algorithm, the image quilting algorithm is still
several times faster. It appears from these results that Efros and Freeman’s observation
that most pixels in a synthesized texture do not have a choice about their final color was a
well-founded observation. By using patches, image quilting is able to produce very
similar results for stochastic textures in a small percentage of the time without
propagating errors made during processing (see the results from the single resolution
algorithm with a 5x5 neighborhood). Additionally, image quilting can produce very
convincing structured textures while Wei and Levoy’s algorithms cannot (see Figure 8).
Figure 8: Some results from the algorithms described in [WEI] and [EFROS01]. The first image is a
structured input texture patch for the texture synthesis algorithms. The second image is the texture
generated by Wei and Levoy’s single resolution algorithm using S = 9. The third picture is the
texture generated by Wei and Levoy’s multiple resolution algorithm with S = 7 and N = 5. The last
image is the texture generated by Efros and Freeman’s image quilting algorithm.
There are a few disadvantages to the image quilting technique. First, image quilting does
not produce tileable results. Second, by observation, the results of the image quilting
algorithm on stochastic textures are more “regular” than the ones produced by Wei and
Levoy’s algorithm.
4. Discussion
From the above observations, it seems that image quilting reliably produces excellent
results in a short amount of time. These properties make image quilting a better choice
over Wei and Levoy’s algorithm under most circumstances.
Ideally, image quilting could be used to produce tileable results just as Wei and Levoy’s
algorithm does. Unfortunately, as the algorithm is described in Efros and Freeman’s
paper, it cannot. Once possible extension for image quilting would be to use pyramids, as
done in Wei and Levoy’s algorithm, to produce tileable textures. By running the image
quilting algorithm on lower levels of the pyramid, you can estimate what is on the edges
of the image and use this information to produce tileable results in higher resolutions.
7
For now, Wei and Levoy’s algorithm is still better to use when trying to synthesize
tileable or highly randomized stochastic textures.
5. Works Cited
[EFROS99] A. A. Efros and T. K. Leung. “Texture Synthesis By Non-Parametric
Sampling.” In International Conference on Computer Vision, pages 1033-1038,
Corfu, Greece, September 1999.
[EFROS01] Alexei A. Efros and William T. Freeman. “Image Quilting for Texture
Synthesis and Transfer.” In Siggraph 01, pages 341-346, 2001.
[HEEGER] David J. Heeger and James R. Bergen. “Pyramid-Based Texture
Analysis/Synthesis.” In Siggraph 95, pages 229-238, 1995.
[PORTILLA] J. Portilla and E. P. Simoncelli. “A Parametric Texture Model Based on
Joint Statistics of Complex Wavelet Coefficients.” International Journal of
Computer Vision, 40(1):49-71, December 2000.
[WEI] Li-Yi Wei and Marc Levoy. Fast Texture Synthesis Using Tree-Structured Vector
Quantization. In Siggraph 00, pages 479-488 2000.
8