International Journal of Civil Engineering and Technology (IJCIET)
Volume 10, Issue 04, April 2019, pp. 106-113, Article ID: IJCIET_10_04_012
Available online at http://www.iaeme.com/ijciet/issues.asp?JType=IJCIET&VType=10&IType=04
ISSN Print: 0976-6308 and ISSN Online: 0976-6316
© IAEME Publication
Scopus Indexed
IMAGE ENHANCEMENT BY SEAM CARVING /
MSR ALGORITHM
M. K. Maaroof
Department of Computer and Mathematic
Faculty of Basic education, Babylon University, Iraq
ABSTRACT
Conventional methods such as cropping or resampling can introduce more losses
in information or distortion in resolution. In this paper we propose an improved seam
carving algorithm which incorporates multi scale retinex techniques. We propose a
MSR improvement for image enhancement. MSR is used on images under non uniform
illumination in terms of either color or lightness and has satisfactory results to achieve
color constancy and dynamic range compression. MSR is extremely sensitive to noise
speckles that cameras produce in low light areas, and it has unsatisfactory effect on
areas with normal or intensive illumination. Moreover, MSR uses a gain-offset method
for prior-to-display treatment and can lead to apparent data loss on images. This study
replaces the logarithm function in MSR with a customized sigmoid function to minimize
data loss, and adapts MSR to images by merging results from sigmoid-MSR with seam
carving on original images. Experiments show our framework, when applied to images,
can preserve areas with normal or intensive lighting and suppress noise speckles in
extreme low light areas, we have demonstrated superior performance upon the current
seam carving –multi scale retinex methods.
Keyword head: image resizing, seam carving, multi scale retinex algorithm.
Cite this Article: M. K. Maaroof, Image Enhancement by Seam Carving / Msr
Algorithm. International Journal of Civil Engineering and Technology, 10(04), 2019,
pp. 106-113
http://www.iaeme.com/IJCIET/issues.asp?JType=IJCIET&VType=10&IType=04
1. INTRODUCTION
when the rise of mobile media device such as smart phones, content aware image resizing, or
image retargeting, is fast becoming an important research area. We need to resize images and
videos to fit various device displays with different aspect ratios in a way that increasing
information and decreasing distortion. Traditional methods including resampling and cropping
had been option up until a few years ago. In these work of 2007, Avidan and Shamir proposed
a rather elegant solution called seam carving, which operates in a discrete fashion, reducing an
image’s dimension by one row or column at each step [1][2]. Later solutions by Wolf et al.
\http://www.iaeme.com/IJCIET/index.asp
106
editor@iaeme.com
Image Enhancement by Seam Carving / Msr Algorithm
[7], Simakov et al. [6] and Guo et al. [4] produce better results by employing global
optimization techniques but are generally more computationally expensive. Global solutions
are not well-suited for multi-scale media applications those that permit an image or video to be
retargeted to any size on the fly [1] because they would require processing for each change in
size. Seam carving, only requires a single preprocessing step to prepare images or video such
that real-time retargeting is possible [1]. While seam carving remains an important tool for
image retargeting, it has its limitations. High level features such as face detection were
suggested in [5] as a way to enhance the seam carving algorithm, but little work has been done
to enhance the use of low level features. The detection of such high level may fail during
practice and thus cause the entire scheme to fail. In this paper we propose a new algorithm that
which contain from interaction between the seam carving and multi scale retinex which
improves edge preservation and decreases artifacts and applying the retinx algorithm in order
to get lose less in resolution by compression and resizing of image and enhancement in RGB.
Figure 1 (a) Original image and a 40% width reduction using (b) forward-energy seam carving
and (c) MSR high resolution (d) our algorithm with 𝛼=3. Seams highlighted with MSR,
respectively
2. THEORITICAL
2.1. Seam Carving Algorithm
Seam carving works by evaluating statistically the lowest energy connected path of pixels from
either left to right or top to bottom, removing those pixels, and repeating the process to getting
the perfect image size is achieved. In order to getting the rectangular design of an image, it is
required that each path of pixels include exactly one pixel per column for horizontal seams,
one pixel per row for vertical seams. At the same procedure pixels can be added along these
seams to increase the image size. While image enlargement and reduction are both important,
seam carving applies too similar objectives for both [1] and so for the sake of brevity we center
on image reduction.
2.1.1. Seam Carving Conditions
Two primary criteria for describing the energy of a seam: backward energy and forward energy.
The backward energy criterion uses an energy map defined in [1] as
𝝏
𝝏
𝒆(𝑰; 𝒙, 𝒚) = |𝝏𝒙 𝑰| + |𝝏𝒚 𝑰|
(1)
which is the L1-norm of the image gradient. This generally works well because important
objects usually have a well-defined edge, a high gradient value along that edge. Using this map,
dynamic programming is employed to find the minimum energy path. A vertical path of pixels
of an 𝑚×𝑛 image matrix is defined as the set of locations.
http://www.iaeme.com/IJCIET/index.asp
107
editor@iaeme.com
M. K. Maaroof
𝒔 = {𝒔𝒊 = (𝒊, 𝒙𝒊 )} 𝟏≤𝒊≤𝒎
(2)
Where 𝑥𝑖 is determined by the dynamic programming stage as defined in [1]. The forwardenergy criterion as described in [2] does not use directly an energy map. Instead, it knows the
seam that minimizes the absolute differences between the pixels brought together on the
removal of the seam. Yet we can still measure the energy of individual seam pixels using the
cost map 𝑀 as defined in [2]. Since each point of this map corresponds to the minimum
cumulative energy of a seam ending at that point, we can define the individual energy of each
point along a seam as
𝒆(𝑰; 𝒔𝒊 ) = 𝑴(𝒔𝒊 ) − 𝑴(𝒔𝒊−𝟏 )
(3)
Where 𝑠𝑖 is the location of the 𝑖-th pixel in the seam 𝒔. other seam criteria can be used;
however, these are the premiere methods, having been introduced with the invention of seam
carving itself. For our purposes, it does not entirely matter which criterion is used provided it
meets the aforementioned seam requirements a path of pixels with one pixel per row or column.
High energy Pixels is a problem with seam carving arises from the use of cumulative energy
without regard to the energy of independent pixels. The cumulative energy or cost of a seam
is defined as:
E(s) = ∑i e(I; si )
(4)
Minimizing this cost using either the backward or forward energy criterion puts no
requirement on the individual pixel energies, namely, 𝑒 (𝐼; 𝑠𝑖), which may be very large. Figure
1 presents a case that clearly illustrates this problem. Seams sometimes pass through important
objects set against a high energy background even if the frame has a strong edge by retinex
because the cumulative energy would be higher through the background. In this study we
propose two modifications to the seam carving algorithm and take place the retinex for the sake
of brevity, we only detail the removal of vertical seams; the removal of horizontal seams is
done in a same matter. Seam carving will calculate paths through smooth areas. Removing
seams from these areas is un noticeable because the pixels adjacent to the seam will have
similar values and appear to be non-discrete. However, sometimes a seam may be forced
through a region with a bigger image gradient. In this case, cutting the seam will be more
noticeable because the neighbor pixels will have less-similar values. As mentioned above,
aliasing is one way to describe this effect since we are removing a sample from a part of the
image with high frequency; the result is under sampling. Anti-aliasing tries to avoid this
problem by applying a low pass filter before down sampling. This is usually done when
linearly down sampling an entire image, but seam carving only removes samples from a single
path through the image at each step. Therefore, being only concerned with aliasing along the
seam, we have implemented a simple 2-point average as shown in Figure 2. Other low-pass
filters could also be used to similar benefit, but we found that a simple average worked well.
Figure 2 Optimal seam (blue)and its neighbor seam (red middle) replaced by average (purple)
To preserve important object edges and decrease artifacts, it may seem enough to simply
threshold the individual pixel energies as defined by (1) or (3), depending on which criterion
http://www.iaeme.com/IJCIET/index.asp
108
editor@iaeme.com
Image Enhancement by Seam Carving / Msr Algorithm
is used. However, we found that this approach did not always work well. Therefore, we sought
a secondary energy map to address the main cause of artifacts. When removing vertical seams
those that run from top to bottom, he differences in pixels along the horizontal direction appear
to cause the most distortion (cf. Figure 4). Since the magnitude of the gradient includes both
horizontal and vertical directions, it still might seem like a good candidate, but it turns out that
the extra information can result in too many false positives. We consider again the silhouette
image from Figure 1 and look at the L1-norm of the image gradient (Figure 4 (a)). We would
like to preserve the person’s silhouette, but because of the high-frequency in the waves, the
magnitude of the gradient is insufficient. Based on our observations, and as demonstrated in
Figure 4, we look at the absolute pixel differences in the horizontal direction (Figure 4 (c)). As
desired, the result shows large differences around the silhouette because of the abrupt change
at the edge of the person in the horizontal direction, and small differences elsewhere. In
particular, the waves generally have higher differences in the vertical direction, but not in the
horizontal direction. Essentially, looking at absolute pixel differences is an estimate of the
partial derivative based on a 1-pixel interval. In the horizontal direction we have:
|
𝝏𝑰(𝒙,𝒚
|
𝝏𝒙
= | 𝒍𝒊𝒎
𝝏𝑰(𝒙,𝒚
|
𝝏𝒙
≅|
|
𝑰(𝒙+∆𝒙,𝒚)−𝑰(𝒙,𝒚)
|
∆𝒙
∆𝒙→𝟎
(5)
𝑰(𝒙+𝟏,𝒚)−𝑰(𝒙,𝒚)
|
𝟏
(6)
The secondary energy map for vertical seam thresholding as:
𝒆𝑻,𝑽 (𝑰; 𝒙, 𝒚) = |𝑰(𝒙 + 𝟏, 𝒚) − 𝑰(𝒙, 𝒚)|
(7)
Figure 3.a. original image b. distortion image
The secondary energy map for horizontal seam limit is defined similarly. While there are
indeed other ways to estimate the derivative of an image, we found that using works well in
identifying critical edges. In a secondary energy map, the typical thing might be to linearly
combine it with either the backward or forward energy map to form one single energy map.
The goal is to prohibit seams from passing through certain points, not merely deter the seams.
We propose threshold the secondary map defined to flag restricted points in the image.
Applying this to the silhouette image, we clearly see the outline of the person as being restricted
((Figure 4 (d)), which is exactly as desired. Threshold the Sobel-based L1-norm of the gradient
produces too many restricted points while missing some of the silhouette ((Figure 4 (b)).
Therefore, as each seam is discovered using some seam criterion, we check the secondary
energy map along the seam. If at any point the threshold is crossed, we do not remove the seam,
and flag that spot as irremovable so that future seams will avoid it as well. Although this
flagging could be done globally at each step, we only flag points along the selected seam to
facilitate the stopping criterion. Formally, this flagging can be accomplished using a binary
map 𝑇 that is updated after each discovered seam by
http://www.iaeme.com/IJCIET/index.asp
109
editor@iaeme.com
M. K. Maaroof
𝑻(𝒔𝒊 ) = {
𝟎 𝒆𝑻 (𝟏; 𝒔𝒊 ) ≤ 𝜼
𝟏 𝒆𝑻 (𝟏; 𝒔𝒊 ) > 𝜼
(8)
where 𝜂 is a threshold constant. If the threshold is not crossed, we remove the seam. Note
that the path removed from the image is also removed from the flag map 𝑇 in order to keep
them in synchronous. This process repeats until the target size is reached. The threshold value
can be selected manually as desired to guide edge preservation. However, a simple threshold
technique based on a common image denoising technique described in [3] was found to work
quite well. We define our threshold as.
𝜼 = 𝜶𝝈
(9)
where 𝜎 is the standard deviation of the secondary energy map 𝑒𝑇 and 𝛼 is some constant.
We found that 𝛼 values near 2 typically worked well, however, the threshold can be set looser
or tighter by increasing or decreasing this number, respectively. In some cases, making the
threshold too restrictive can force the algorithm to end before reaching the target size if there
are too many restricted points.
2.2. Multiscale Retinex Algorithm
It is one effective theory aiming at simulating Human Visual System (HVS) to achieve color
constancy and dynamic range compression. This study shows that with modifications, retinex
can be applicable to nighttime images even under extreme conditions. In 1986, Edwin Land [8]
proposed the last version of his retinex theory as a model for human color constancy. Several
works emerged implementing this retinex theory but with huge computational cost and
sometimes low performance in some extreme cases. What caught our attention is the
center/surround retinex, which was also brought up by Edwin Land [9] and has the
characteristics of easy implementation, fast evaluation and less parameters. Research [10, 11]
further enhanced retinex leading to what is single scale retinex (SSR) and multi scale retinex
(MSR), which is shown to be able to accomplish color rendition and dynamic variety
compression at the same time. especially on multi scale retinex, recedes a bit and only a few
works stand out. Jang [12, 13] focuses on better color correction for retinex algorithm.
Robinson et al. [14] enhance MSR by reducing halo artifacts and graying effect. Rahman et al.
[15] investigate the relationship between retinex and image compression. Jang [12] improves
the MSR with respect to weights of different scales of retinex. Jang and Wang [16, 17] both
propose to accelerate the implementation of MSR. Through experiments we see that MSR can
have an overall acceptable results on nighttime images. It supplied color constancy, dynamic
range compression and in short, better visual quality. However, one of their post processing
algorithms has to be reconsidered. Because of the logarithm function in c/s retinex and SSR or
MSR, the primary results of these retinex processes can have large range of value and are
impossible to be displayed as images directly. One approach we now commonly use is
proposed in [10] involving a gain offset method which clips those pixels with too high or too
low lightness, in which case, it is shown little information is lost. Our intuitive idea is to
eliminate the uncertainty of the value range of the result from the beginning, where we used to
take the logarithm of the ratio between two intensity values. This ratio ranges from 1/256 to
256 and the logarithm of the ratio makes itself distribute loosely across its range. We set the
higher and lower threshold of the gain offset method empirically. So we came up with an idea
to replace the logarithm function with the sigmoid function, which has a certain range of output.
This certainty is obtained by compressing the extreme pixels rather than clipping them. So
unlike the logarithm method, it doesn’t need the gain offset method which involves clipping
and leads to information loss. Combined with other methods discriminating between areas of
http://www.iaeme.com/IJCIET/index.asp
110
editor@iaeme.com
Image Enhancement by Seam Carving / Msr Algorithm
different illumination, our framework has overall better enhancement effect among nighttime
images. we first introduce the well-known MSR. The Original MSR [11] can be written as
𝑭𝒊 (𝒙, 𝒚) = ∑𝑵
𝒏=𝟏 𝑾𝒏 . { 𝒍𝒐𝒈[𝑺𝒊 (𝒙, 𝒚)] − 𝒍𝒐𝒈[𝑺𝒊 (𝒙, 𝒚) ∗ 𝑴𝒏 (𝒙, 𝒚)] }
(10)
where F is the result we get from MSR operation, the subscripts i ∈ R, G, B indicate the 3
color channels, N is the number of scales of retinex computed, and Wn are the weighting factors
of each scale. Si (x, y) is the ith channel, dimensional matrix of the input image, mark ∗ is the
convolution operator, and the Mn (x, y) are the surround functions given by
𝑴𝒏 (𝒙, 𝒚) = 𝑲𝒏 𝒆𝒙𝒑[−(𝒙𝟐 + 𝒚𝟐 )/𝝈𝟐𝒏 ]
(11)
where Kn is to insure ∫∫ Mn (x, y) dx dy = 1. Each of the expressions within the summation
in Eq.12 represents an SSR. This expression of SSR can also be written as
𝒍𝒐𝒈 [
𝑺𝒊 (𝒙,𝒚)
]
𝑺𝒊 (𝒙,𝒚)∗𝑴𝒏 (𝒙,𝒚)
(12)
which can be intuitively perceived as a comparison between the current pixel and its
weighted, surrounding pixels. σn are the standard deviations of the Gaussian distribution
determining the scales of the surrounding neighborhood taken into account. Smaller scales
provide more dynamic range compression, and larger ones provide more color constancy. The
quality can be further degraded by transmission error and poor lighting conditions in image
sequences. With experimental results we have shown that this visual quality can be improved
by using retinex which enhances the contrast in images [19].
3. EXPERIMENTALS & RESULTS
Figure 4 shows a comparison of our algorithm with the backward and forward energy seam
carving methods. The original image sizes are given for each image and for each case we
removed 300 vertical seams. For our algorithm we used the backward energy criterion for seam
selection using the forward energy criterion in our algorithm did not result in any noticeable
gain while costing more compute time. The threshold factors are given under their respective
images and effect of retinex on images in column (d). Despite the fact that the backward energy
seam carving results were generally much worse, our algorithm performed better or at least as
well as the forward energy seam carving method and increase pixie ability to carry out the
color. In particular, notice the top image in Figure 4 our algorithm was able to preserve the
slope of the mountain. The bottom image shows comparable results between our algorithm and
the forward-energy seam carving method. The results of our algorithm in Figure 5 are
comparison between the MSR and Seam carving with respect to images resizing and scope of
values for MSR and Seam carving algorithms.
4. CONCLUSIONS
Seam carving is an important image retargeting tool that can be used to fit images to various
display devices with different aspect ratios through nonlinear down sampling. It stands to be
very useful for multi size media application where many different sizes of the same image or
video are required without the need for reprocessing the image for each size request. Yet, the
algorithm is not completely robust and can be susceptible to undesirable distortion. We have
proposed a new algorithm which incorporates anti-aliasing and threshold and multi scale
retinex to treatment the distortion occurs. Overall, the results were very positive, being
comparable or better than previous seam carving implementations. While we have only
discussed using the backward and forward energy seam criteria for seam selection, our
algorithm allows implementing any optimal seam-finding method as well as augmenting
higher-level feature detectors.
http://www.iaeme.com/IJCIET/index.asp
111
editor@iaeme.com
M. K. Maaroof
(a)
(b)
(c)
(d)
Figure 4 (a) Original image and the result of removing 300 seams using (b) backward energy seam
carving, (c) forward energy seam carving and (d) seam carving-retinex algorithm.
In future work, we plan to look into devising more automatics ways of determining the
threshold factor. We believe that this will, in turn, lead to development of a stopping criteria
which could decide when further seam carving would add too much distortion and switch to
resampling or cropping.
Figure 5. Seam carving, MSR and Seam carving +MSR algorithms according to pixel energy and
image resizing
REFERENCE
[1]
[2]
[3]
[4]
[5]
S. Avidan and A. Shamir. Seam carving for content-aware image resizing. ACM
Transactions on Graphics, 2007. Vol. 26, No. 3.
S. Avidan and A. Shamir. Seam carving for media retargeting. Communications of the
ACM, 2009. Vol 52, No. 1.
D.L. Donoho and I.M. Johnstone. Ideal spatial adaptation via wavelet shrinkage. Technical
report 400. Stanford, CA: Department of Statistics, Stanford university, November 1992.
Y. Guo, F. Liu, J. Shi, Z.H. Zhou and M. Gleicher. Image retargeting using mesh
parametrization. IEEE Transactions on Multimedia, 2009. Vol. 11, No. 5.
Daw-Sen Hwang and Shao-Yi Chien. Content aware image resizing using perceptual seam
carving with human attention model. ICME, 2008.
http://www.iaeme.com/IJCIET/index.asp
112
editor@iaeme.com
Image Enhancement by Seam Carving / Msr Algorithm
[6]
[7]
[8]
[9]
[10]
[11]
[12]
[13]
[14]
[15]
[16]
[17]
[18]
[19]
D. Simakov, Y. Caspi, E. Shechtman and M. Irani. Summarizing visual data using
bidirectional similarity. In Proceedings of CVPR, 2008.
L. Wolf, M. Guttmann and D. Cohen-Or. Nonhomogeneous content-driven videoretargeting. In Proceedings of the 11th IEEE International Conference on
Computer
Vision, 2007
E. H. Land, Recent advances in retinex theory, Vision research 26 (1) (1986) 7– 21.
E. H. Land, An alternative technique for the computation of the designator in the retinex
theory of color vision, Proceedings of the National Academy of Sciences 83 (10) (1986)
3078–3080.
D. J. Jobson, Z.-u. Rahman, G. A. Woodell, Properties and performance of a
canter/surround retinex, Image Processing, IEEE Transactions on 6 (3) (1997)
451–
462.
Z.-u. Rahman, D. J. Jobson, G. A. Woodell, Multi-scale retinex for color image.
enhancement, in: Image Processing, 1996. Proceedings., International Conference on, Vol.
3, IEEE, 1996, pp. 1003–1006.
I.-S. Jang, T.-H. Lee, H.-G. Ha, Y.-H. Ha, Adaptive color enhancement based on multiscaled retinex using local contrast of the input image, in: Opt mechatronic Technologies
(ISOT), 2010 International Symposium on, IEEE, 2010, pp. 1–6.
I.-S. Jang, K.-H. Park, Y.-H. Ha, Color correction by estimation of dominant chromaticity
in multi-scaled retinex, Journal of Imaging Science and Technology 53 (5) (2009) 50502–
11.
P. Robinson, Y. Roodt, A. Nel, Adaptive multi-scale retinex algorithm for contrast
enhancement of real world scenes, in: The Proceedings of Twenty-Third Annual
Symposium of the Pattern Recognition Association of South Africa, Pretoria, South Africa,
Edited by Alta de Waal, 2012.
Z.-u. Rahman, D. J. Jobson, G. A. Woodell, Investigating the relationship between image
enhancement and image compression in the context of the multi- scale retinex, Journal of
Visual Communication and Image Representation 22 (3) (2011) 237–250.
C. Y. Jang, J. Hyun, S. Cho, H.-S. Kim, Y. H. Kim, Adaptive selection of weights in multiscale retinex using illumination and object edges.
C. Y. Jang, J. H. Lim, Y. H. Kim, a fast multi-scale retinex algorithm using dominant ssr
in weights selection, in: SoC Design Conference (ISOCC), 2012 International, IEEE, 2012,
pp. 37–40.
W. Wang, B. Li, J. Zheng, S. Xian, J. Wang, A fast multi-scale retinex algorithm for color
image enhancement, in: Wavelet Analysis and Pattern Recognition 2008. ICWAPR’08.
International Conference on, Vol. 1, IEEE, 2008, pp. 80–85.
Mayson Khazaal Maaroof, Sangita.G. Lade ,2015, H264 Video Compression Technique
with Retinex Enhancement, International Journal of Engineering Research & Technology
(IJERT)ISSN: 2278-0181 Vol. 4 Issue 05, May-2015.
http://www.iaeme.com/IJCIET/index.asp
113
editor@iaeme.com