International Journal of Engineering Trends and Technology (IJETT) – Volume 4 Issue 6- June 2013
Image Pruning Using Belief Propagation in
Image Completion
N.Nanthini#1, Dr. T.Meyyappan, M.Sc., M.phil. M.B.A.,(M.Tech)., Ph.D*2,
#
Department of Computer Science and Engineering ,Alagappa University ,Karaikudi,Tamilnadu ,India
Abstract— An interactive image completion method is proposed
based on Belief Propagation (BP). Blocked area or area with loss
of information in a target image is completed with BP combined
with texture synthesis in an interactive way. The target image is
decomposed by BP into levels of Intrinsic Mode Functions (IMF)
images, while the user is allowed to indicate the structural image
edge to recover in the unknown image regions. For each level of
IMF image, first the target image patches along user-specified
curves in the unknown region are completed. Then the remaining
target image patches are prioritized to complete according to
image gradient feature. The target image patches are completed
based on the combination frequency feature values from BP and
the texture synthesis. A fast algorithm for filling unknown
regions in an image using the strategy of exemplar matching.
Unlike the original exemplar-based method using exhaustive
search, we decompose exemplars into the frequency coefficients
and select fewer coefficients which are the most significant to
evaluate the matching score. Moreover, the evaluation of searcharrays runs in parallel maps well on the modern graphics
hardware with Graphics Processing Units (GPU). The
functionality of the approach has been demonstrated by
experimental results on real photograph
This concept reflects the fact that images frequently
contain collections of objects each of which can be the basis
for a region. In a sophisticated image processing system it
should be possible to apply specific image processing
operations to selected regions. Thus one part of an image
(region) might be processed to suppress motion blur while
another part might be processed to improve color rendition.
The amplitudes of a given image will almost always be either
real numbers or integer numbers.
The latter is usually a result of a quantization process
that converts a continuous range (say, between 0 and 100%) to
a discrete number of levels. In certain image-forming
processes, however, the signal may involve photon counting
which implies that the amplitude would be inherently
quantized.
In other image forming procedures, such as magnetic
resonance imaging, the direct physical measurement yields a
complex number in the form of a real magnitude and a real
phase.
Keywords: Belief Propagation, Markov Random Field, Label
Image completion is one of the key areas in image
processing. It is an important photo-editing task which
involves synthetically filling a hole in the image such that the
image still appears natural. State-of-the-art image completion
methods work by searching for patches in the image that fit
well in the “hole” region.
Pruning, Message Scheduling.
I.INTRODUCTION
Image processing is the methodology to convert an
image into digital form and perform some operations on it, in
order to get an enhanced image or to extract some useful data
from it. It is a type of signal privilege in which input is an
image, like video frame or photograph and the output might
be either image or characteristics associated with that same.
Usually image processing system includes treating images as
two dimensional signals while applying already set signal
processing methods to them.
Our key insight is that the image patches remain
natural under a variety of transformations (such as scale,
rotation and brightness change), and it is important to exploit
this. We propose and investigate the use of different
optimization methods to search for the best patches and their
respective transformations for producing consistent, improved
completions. Experiments on a number of challenging
problem instances demonstrate that our methods outperform
state-of-the-art techniques.
We will restrict ourselves to two-dimensional (2D)
image processing although most of the concepts and
techniques that are to be described can be extended easily to
three or more dimensions. We begin with certain basic
definitions.
ISSN: 2231-5381
http://www.ijettjournal.org
Page 2651
International Journal of Engineering Trends and Technology (IJETT) – Volume 4 Issue 6- June 2013
II. EXISTING METHODS
There have been three main approaches so far, for dealing
with the image completion problem. Below are the existing
methods for the image completion process.
A. Statistical - Based Methods:
These methods are mainly used for the case of texture
synthesis. Typically, what these methods do is that, given an
input texture, they try to describe it by extracting some
statistics through the use of compact parametric statistical
models .Then, in order to synthesize a new texture, these
methods typically start with an output image containing pure
noise, and keep perturbing that image until its statistics match
the estimated statistics of the input texture. [10].
Besides the synthesis of still images, parametric statistical
models have been also proposed for the case of image
sequences [12]. However, the main drawback of all methods
that are based on parametric statistical models is that, as
already mentioned, they are applicable only to the problem of
texture synthesis [11], and not to the general problem of
image completion.
But even in the restricted case of texture synthesis, they can
synthesize only textures which are highly stochastic and
usually fail to do so for textures containing structure as well.
Nevertheless, in cases where parametric models are applicable,
they allow greater flexibility with Respect to the modification
of texture properties [15].
(a)
(b)
(c)
differential equation (PDE) [2], which is typically non-linear
and of high order.
The main disadvantage of almost all PDE based methods is
that they are mostly suitable for image inpainting situations.
This term usually refer to the case where the missing part of
the image consists of thin, elongated regions[19],[17].
Furthermore, PDE-based methods implicitly assume that the
content of the missing region is smooth and non-textured [16].
For this reason, when these methods are applied to images
where the missing regions are large and textured, they usually
over smooth the image and introduce blurring artifacts.
C. Exemplar-Based Methods:
These methods try to fill the unknown region simply by
copying content from the observed part of the image [18]. All
exemplar-based techniques for texture synthesis that have
appeared until now, were either pixel-based [4],[5], or patchbased [6],[9],[10], meaning that the final texture was
synthesized one pixel, or one patch at a time by simply
copying pixels or patches from the observed image
respectively.
Recent exemplar-based methods also place emphasis on the
order by which the image synthesis proceeds, usually using a
confidence map for this purpose. There are two main
handicaps of related existing techniques.
Exemplar based methods for texture synthesis has been also
used for the case of video. E.g Schodl et al. [9] are able to
synthesize new video textures simply by rearranging the
recorded frames of an input video, while the texture synthesis
method of Kwatra et al. [2],[3] that has been mentioned above
applies to image sequences as well.
First, the confidence map is computed based on heuristics and
ad hoc principles that may not apply in the general case and
second, once an observed patch has been assigned to a
missing block of pixels[1], that block cannot change its
assigned patch thereafter. This last fact reveals the greediness
of these techniques, which may lead to visual inconsistencies.
D. Gradient-Based Filling:
Fig 1 a .Original image b. Image with missing region
c. Completion using image inpainting.
Image inpainting methods, when applied to large or
textured missing regions, very often over smooth the image
and introduce blurring artifacts.
B. PDE - Based Methods:
These methods, on the other hand, try to fill the missing
region of an image through a diffusion process, by smoothly
propagating information from the boundary towards the
interior of the missing region. According to these techniques,
the diffusion process is simulated by solving a partial
ISSN: 2231-5381
For computing DCT coefficients on image blocks with
unknown pixels, if the unknown pixels are filled with average
color of known pixels[5], the DCT coefficients do not reflect
the texture or the structural information in the block very well.
For example, if the missing region ­ is located at the center of
an image block with progressive color change from left to
right, filling ­ with average color is not a good approximation.
For a smooth image, the gradient at pixels will be
approximately equal to zero. Based on this observation, we
developed a gradient-based filling method to determine the
unknown pixels before computing DCT.In detail, for each
http://www.ijettjournal.org
Page 2652
International Journal of Engineering Trends and Technology (IJETT) – Volume 4 Issue 6- June 2013
unknown pixel pi;j , letting the discrete gradient at this pixel
be zero will lead to linear equations k linear equations with k >
l which is actually an over determined linear system[23].
Note that, the gradient filling may generate smoother images
at those unknown pixels when the known pixels are highly
textured. This will be overcome by a revision of the
Criminisiet’s method [1]. We find more than one matched
patches (actually 0:1% of the toal exemplars), among which
the one with the highest SSD score on the known pixels is
finally selected.
Furthermore, using m-dominant DCT coefficients in selection
will also reduce the effect led by the pixels filled in this step.
mark the linear structure in the target region firstly and then
the image patch is copied along the prescribed direction.This
model generates good image but it needs user’s interaction.
Criminisi proposed an exemplar-based image completion
model which determined the filling order under the constraint
of PDE-based data item. The exemplar which is along a linear
structure has a higher filling priority, and the geometrical
property in the completed image is preserved. Another crossisophote diffused PDE is used in the completion model which
has a better structure preserving property.
E. Texture Synthesis:
In-painting, the technique of modifying an image in
an undetectable form, is as ancient as art itself. The goals and
applications of in-painting are numerous, from the restoration
of damaged paintings and photographs to the
removal/replacement of selected objects. In this paper, we
introduce a novel algorithm for digital in-painting of still
images that attempts to replicate the basic techniques used by
professional restorators. After the user selects the regions to
be restored, the algorithm automatically fills-in these regions
with information surrounding them. The fill-in is done in such
a way that isophote lines arriving at the regions’ boundaries
are completed inside. In contrast with previous approaches,
the technique here introduced does not require the user to
specify where the novel information comes from.
Texture synthesis is another kind of image
completion. It regenerates texture patterns in the target region.
It aims at restoring texture pattern properly and seamlessly.
The texture synthesis model uses variable neighborhood
searching to preserve the entire texture pattern. Graphcut
approach is used to reduce seams between texture patches, and
it generates ω-tile set to avoid highly repetitive patterns.
A non-parametric method based on Markov random
field theory is introduced to preserve the local structure and
produce a wide variety of textures. The contrast enhancement
is also used to generate image with a good visual perception.
Vector quantization is used to get a fast texture synthesis
model. The non-parametric model is extended to exemplarbased to accelerate the computing speed. All texture synthesis
approaches process image based on the global information,
and the entire texture pattern in image is resorted. They deal
nothing with the local geometrical property, so they cannot
preserve the linear structures well in image.
Texture synthesis is not suitable for geometrical
image completion. Efros proposed the exemplar-based texture
synthesis model. Bornard introduced this model into the
geometrical natural image completion, and Perez proved that
used in geometrical image completion this kind of model
could generate a good result. In exemplar-based image
completion model, the basic unit of synthesis is a filling patch
(exemplar) more than a single pixel.
An exemplar is compared and copied during the
completion procedure. Harrison determines the filling order
by “textureness” of pixel, and the pixels which are highly
constrained by neighborhood pixels have higher filling
priorities. Although this intent is good, strong linear structures
cannot be properly preserved. The method proposed by Drori
“iteratively approximates the unknown region and fills in the
image by adaptive fragments”. It uses an inverse alpha matte
as the confidence map to determine the traversal order.
This method is time-consuming and it cannot often
generate good result. Jian et al. introduced an image
completion model with structure propagation. User should
ISSN: 2231-5381
F. Image In-painting:
This is automatically done (and in a fast way),
thereby allowing to simultaneously fill-in numerous regions
containing completely different structures and surrounding
backgrounds. In addition, no limitations are imposed on the
topology of the region to be in-painted. Applications of this
technique include the restoration of old photographs and
damaged film; removal of superimposed text like dates,
subtitles, or publicity; and the removal of entire objects from
the image like microphones or wires in special effects.
The modification of images in a way that is nondetectable for an observer who does not know the original
image is a practice as old as artistic creation itself. Medieval
artwork started to be restored as early as the Renaissance, the
motives being often as much to bring medieval pictures “up to
date” as to fill in any gaps. This practice is called retouching
or in-painting. The object of in-painting is to reconstitute the
missing or damaged portions of the work, in order to make it
more legible and to restore its unity.
The need to retouch the image in an unobtrusive way
extended naturally from paintings to photography and film.
The purposes remain the same: to revert deterioration (e.g.,
cracks in photographs or scratches and dust spots in film), or
to add or remove elements. Digital techniques are starting to
be a widespread way of performing in-painting, ranging from
attempts to fully automatic detection and removal of scratches,
all the way to software tools that allow a sophisticated but
mostly manual process.
http://www.ijettjournal.org
Page 2653
International Journal of Engineering Trends and Technology (IJETT) – Volume 4 Issue 6- June 2013
Although a number of techniques exist for the semiautomatic detection of image defects (mainly in films),
addressing this is out of the scope of this paper. Moreover,
since the in-painting algorithm here presented can be used not
just to restore damaged photographs but also to remove
undesired objects and writings on the image, the regions to be
in-painted must be marked by the user, since they depend on
his/her subjective selection. Here we are concerned on how to
“fill-in” the regions to be in-painted, once they have been
selected.
G. Disadvantage:






The main drawback of all methods that are based on
parametric statistical models is that they are
applicable only to the problem of texture synthesis,
and not to the general problem of image completion
The main disadvantage of almost all PDE based
methods is that they are mostly suitable for image
inpainting situations. This term usually refer to the
case where the missing part of the image consists of
thin, elongated regions.
Most of the exiting methods in literature take a long
time to retouch one image, which is far from
practical using in an interactive image processing and
editing.
It does not give satisfactory result in the regions wish
large unknown areas and highly textured region.
Lose linear structure and composite texture
All these approaches are extremely slow due to the
high computational complexity.
III. PROPOSED METHODS
This operator can be used for image manipulations
including: image retargeting and object removal. The operator
can be easily integrated with various saliency measures, as
well as user input, to guide the resizing process.
In addition, we define a data structure for multi-size
images that support continuous resizing ability in real time we
present a representation of multi-size images that encodes, for
an image of size (m×n), an entire range of retargeting sizes
from a×b to m×n and allow the user to retarget an image
continuously in real time.
A.Proposed Algorithm:
1)
BP (Belief Propagation) algorithm: Loopy belief
propagation (BP) is an effective solution for assigning labels
to the nodes of a graphical model such as the Markov random
field (MRF), but it requires high memory, bandwidth, and
computational costs. Furthermore, the iterative, pixel-wise,
and sequential operations of BP make it difficult to parallelize
the computation. In this paper, we propose two techniques to
ISSN: 2231-5381
address these issues. The first technique is a new message
passing scheme named tile-based BP that reduces the memory
and bandwidth to a fraction of the ordinary BP algorithms
without performance degradation by splitting the MRF into
many tiles and only storing the messages across the
neighbouring tiles.
The tile-wise processing also enables data reuse and
pipeline, resulting in efficient hardware implementation. The
second technique is an O(L) fast message construction
algorithm that exploits the properties of robust functions for
parallelization. We apply these two techniques to a very largescale integration circuit for stereo matching that generates
high-resolution disparity maps in near real-time. We also
implement the proposed schemes on graphics processing unit
(GPU) which is four-time faster than standard BP on GPU.
The main advantage is that we now can hopefully
apply belief propagation (i.e a state-of-the-art optimization
method) to our energy function. The reason is the intolerable
computational cost of BP, caused by the huge number of
existing labels. Motivated by this fact, one other major
contribution of this work is the proposal of a novel MRF
optimization scheme, called Priority-BP that can deal exactly
with this type of problems, and carries two significant
extensions over standard BP: one of them, called dynamic
label pruning, is based on the key idea of drastically reducing
the number of labels.
However, instead of this happening beforehand
(which will almost surely lead to throwing away useful labels),
pruning takes place on the fly (i.e while BP is running), with a
(possibly) different number of labels kept for each node. The
important thing to note is that only the beliefs calculated by
BP are used for that purpose. This is exactly what makes the
algorithm generic and applicable to any MRF. Furthermore,
the second extension, called priority-based message
scheduling, makes use of label pruning and allows us to
always send cheap messages between the nodes of the
graphical model.
2) MRF (Markov Random Field): Markov random
field models provide a robust and unified framework for early
vision problems such as stereo and image restoration.
Inference algorithms based on graph cuts and belief
propagation have been found to yield accurate results, but
despite recent advances are often too slow for practical use. In
this paper we present some algorithmic techniques that
substantially improve the running time of the loopy belief
propagation approach. One of the techniques reduces the
complexity of the inference algorithm to be linear rather than
quadratic in the number of possible labels for each pixel,
which is important for problems such as image restoration that
have a large label set.
http://www.ijettjournal.org
Page 2654
International Journal of Engineering Trends and Technology (IJETT) – Volume 4 Issue 6- June 2013
Another technique speeds up and reduces the memory
requirements of belief propagation on grid graphs. A third
technique is a multi-grid method that makes it possible to
obtain good results with a small fixed number of message
passing iterations, independent of the size of the input images.
Taken together these techniques speed up the standard
algorithm by several orders of magnitude. In practice we
obtain results that are as accurate as those of other global
methods (e.g., using the Middlebury stereo benchmark) while
being nearly as fast as purely local methods.
3) Pruning:
Pruning is mainly done under Priority-BP algorithm
with forward pass and backward pass by labeling the node
which to be scheduled. The actual message scheduling
mechanism as well as label pruning takes place during the
forward pass. The role of the backward pass is then just to
ensure that the other half of the messages gets transmitted as
well.
In proposed method we are using the three filters.
The filters are used to classification of wanted and unwanted
pixels of the images. We proposed three filters such as,



Sobel Filter
Prewitt Filter
Laplace Filter
b) Prewitt Filter:
The Prewitt Edge filter is use to detect edges based
applying a horizontal and vertical filter in sequence. Both
filters are applied to the image and summed to form the final
result.
c)
Laplace Filter:
Discrete Laplace operator is often used in image
processing e.g. in edge detection and motion estimation
applications. The discrete laplacian is defined as the sum of
the second derivatives Laplace operator Coordinate
expressions and calculated as sum of differences over the
nearest neighbors of the central pixel.
B. Advantages of Proposed System:
A simple image operator called seam carving is used
here that supports content-aware image resizing for both
reduction and expansion. A seam is an optimal 8-connected
path of pixels on a single image from top to bottom, or left to
right, where optimality is defined by an image energy function.
No user intervention is required by our method, which avoids
greedy patch assignments by maintaining many candidate
source patches. In this way, each missing block of pixels is
allowed to change its assigned patch many times throughout
the execution of the algorithm, and is not enforced to remain
tied to the first label that has been assigned to it during the
early stages of the completion process.
a) Sobel Filter:
The Sobel operator performs a 2-D spatial gradient
measurement on an image and so emphasizes regions of high
spatial frequency that correspond to edges. Typically it is used
to find the approximate absolute gradient magnitude at each
point in an input gray scale image.
The Sobel Edge filter is use to detect edges based
applying a horizontal and vertical filter in sequence. Both
filters are applied to the image and summed to form the final
result.
Fig: Interface of sobel filter
ISSN: 2231-5381
Our formulation applies not only to image
completion, but also to texture synthesis and image inpainting, thus providing a unified framework for all of these
tasks. To this end, a novel optimization scheme is proposed,
the “Priority-BP” algorithm, which carries 2 major
improvements over standard belief propagation: “dynamic
label pruning” and “priority based message scheduling”.
IV. CONCLUSION
In this paper we have enclosed the image completion
approach, texture separation and image in-painting. In order to
prevent the visually unpredictable results we try to avoid
greedy patch assignments, and instead pose all of these tasks
as a discrete labelling problem with a well defined global
objective function. To solve this issue, we proposed the novel
optimization scheme known as Priority-BP (Belief
Propagation) which carries two vital extensions over standard
BP.
This optimization scheme doesn’t rely on any imagespecific prior information and can thus be applied to all kinds
of images. Moreover it is generic (can be applicable to any
MRF energy) and also the investigational results on a wide
variety of images have verified the effectiveness of our
method.
http://www.ijettjournal.org
Page 2655
International Journal of Engineering Trends and Technology (IJETT) – Volume 4 Issue 6- June 2013
[18]
REFRENCES
[1]
[2]
[3]
[4]
[5]
[6]
[7]
[8]
[9]
[10]
[11]
[12]
[13]
[14]
[15]
[16]
[17]
A. Criminisi, P. P´erez, and K. Toyama, “Region filling and
object
removal by exemplar-based image inpainting,” IEEE
Transactions on Image Processing, vol. 13, pp. 1200–1212,
September 2004.
[19]
[20]
M. Bertalmio, G. Sapiro, V. Caselles, and C. Ballester, “Image
inpainting,” in ACM SIGGRAPH 2000. ACM, 2000, pp. 417–424.
[21]
M. Bertalmio, A. L. Bertozzi, and G. Sapiro, “Navier-stokes, fluid
dynamics, and image and video inpainting,” in IEEE CVPR 2001,
vol. I. IEEE, 2001, pp. 355–362.
I.Drori, D. Cohen-Or, and H. Yeshurun, “Fragment-based image
completion,” ACM Trans. Graph., vol. 22, no. 3, pp. 303–312,
2003.L.-Y. Wei and M. Levoy, “Fast texture synthesis using treestructured vector quantization,” in Proceedings of SIGGRAPH
00. New York, NY, USA: ACM Press, 2000, pp. 479–488.
P. Harrison, “A non-hierarchical procedure for re-synthesis of
complex textures,” in WSCG 2001 Conference proceedings, 2001,
pp. 190–197.
[22]
[23]
T. Chan and J. Shen, “Non-texture inpaintings by curvaturedriven diffusions,” J. Visual Comm. Image Rep., vol. 12(4), pp.
436–449, 2001.
Q. Wu and Y. Yu, “Feature matching and deformation for texture
synthesis.” ACM Trans. Graph., vol. 23, no. 3, pp. 364–367, 2004.
M. Ashikhmin, “Synthesizing natural textures,” in Symposium on
Interactive 3D Graphics, 2001, pp. 217–226.
A. Hertzmann, C. E. Jacobs, N. Oliver, B. Curless, and D. H.
Salesin, “Image analogies,” in SIGGRAPH 2001, Computer
Graphics Proceedings. ACM Press / ACM SIGGRAPH, 2001, pp.
327–340.
A. A. Efros and W. T. Freeman, “Image quilting for texture
synthesis and transfer,” in SIGGRAPH 2001, Computer Graphics
Proceedings. ACM Press / ACM SIGGRAPH, 2001, pp. 341–346.
A. A. Efros and W. T. Freeman, “Image quilting for texture
synthesis and transfer,” in SIGGRAPH 2001, Computer Graphics
Proceedings. ACM Press / ACM SIGGRAPH, 2001, pp. 341–346.
J. Jia and C.-K. Tang, “Image repairing: Robust image synthesis
by adaptive nd tensor voting,” in Proceedings of Computer Vision
and Pattern Recognition, vol. 1. Los Alamitos, CA, USA: IEEE
Computer Society, 2003, p. 643.
N. Komodakis and G. Tziritas, “Image completion using global
optimization,” in IEEE CVPR 2006, vol. I. IEEE, 2006, pp. 442–
452.
J. Sun, L. Yuan, J. Jia, and H.-Y. Shum, “Image completion with
structure propagation,” ACM Transactions on Graphics
(SIGGRAPH 2005), vol. 24, pp. 861–868, 2005.
J. Portilla and E. P. Simoncelli, “A parametric texture model
based on joint statistics of complex
D. J. Heeger and J. R. Bergen, “Pyramid-based texture
analysis/synthesis,” in SIGGRAPH, 1995, pp. 229–238.
Y. W. S. Soatto, G. Doretto, “Dynamic textures,” in Intl. Conf. on
Computer Vision, pp. ”439–446”.
W. Fitzgibbon, “Stochastic rigidity: Image registration for
nowhere-static scenes,” in ICCV, 2001, pp. 662–669.
M. Szummer and R. W. Picard, “temporal texture modeling,” in
Proc. of Int. Conference on Image Processing, vol. 3, 1996, pp.
823–826.
G. Doretto and S. Soatto, “Editable dynamic textures.” in CVPR
(2), 2003, pp. 137–142.
C. Ballester, M. Bertalm´ıo, V. Caselles, G. Sapiro, and J.
Verdera, “Filling-in by joint interpolation of vector fields and
gray levels.” IEEE Transactions on Image Processing, vol. 10, no.
8, pp. 1200–1211, 2001.
M. Bertalm´ıo, L. A. Vese, G. Sapiro, and S. Osher,
“Simultaneous structure and texture image inpainting.” in CVPR
(2), 2003, pp. 707–712.
A. A. Efros and T. K. Leung, “Texture synthesis by nonparametric sampling.” in ICCV, 1999.
ISSN: 2231-5381
http://www.ijettjournal.org
Page 2656