ECE 533 Final Project Report
-Removing Blocking artifacts and Ringing artifacts in the
Compressed Image
member : Park, Byung Kwan…
(Single member)
Ⅰ. Introduction…
Nowadays transform-based compression is very popular in the still images and video
like JPEG and MPEG.
But these compressions are loss compressions and in addition they are
based on the method which divides each image into the blocks.
decoder just perform inverse transform.
In spite of these defect, usual
So there can be some artifacts.
In this project, I will
think about two artifacts, “Blocking artifact” and “Ringing artifact”
In the case of blocking artifacts, it is just direct result of independent block processing in codecs.
Many researches were done for removing the artifacts, but the paper I study proposes a method
which takes into local visibility of blocking artifact unlike other researches.
Furthermore ringing artifact is annoying phenomenon in the decoded images.
Although it was
not focused until now as much as blocking artifact, it appears along the edges like sharp
oscillation or ghost shadows in compressed images.
In this project I will try to remove these two artifacts in the highly compressed JPEG images at
the same time.
Ⅱ.
It is based on the theory of projections onto convex sets(POCS).
Approach…
As I mentioned earlier, I use the theory of POCS.
A new family of directional
smoothness constraints sets is defined based on line processing modeling of the image edge
structure.
The definition of these smoothness sets takes into account the fact that the visibility
of compression artifacts in an image is spatially varying.
To overcome the numerical difficulty
in computing the projections onto these sets, a divide-and-conquer(DAC) strategy is introduced.
Based on this strategy new smoothness sets are derived such that their projections are easier
to compute.
To use POCS method, it requires two steps.
First, the definition of the closed convex
constraints sets and second, the derivation of the projections onto these constraint sets.
Let’s think about new convex constraints sets and their projections.
Since JPEG algorithm is
based on block discrete-cosine transform(BDCT), it leads to the following constraint set :
where Fn
min
and Fn
max
are the end-points of the quantization interval that is associated with
the received quantization level of the nth BDCT coefficients (Bf) n. The constraint set is closed
and convex, and the projection of an image vector f onto CT is given by
where the nth component of F is determined by
for n = 1, 2, ….. , MN.
Note that due to the unitary nature of the BDCT transform B, its inverse
B-1 us simply its transpose.
And also smoothness constraint sets are required.
Let Q be a linear operator such that Qf
gives the difference between adjacent columns at the coding block boundaries of f.
example, for the case of N =352(the number of columns and 8*8 blocks,
This leads to another constrain set,
For
where E is a scalar upper bound that defines the size of this set.
estimated form the received compressed image data.
In practice, the quantity E is
This set is closed and convex.
Its
projection operator is derived in other papers.
The third constraint set is for the spatially varying coding artifacts.
To exploit this fact, a
visibility matrix W was introduced into the between-block-boundary variation term Qf to
characterize the local visibility of the coding artifacts.
The visibility matrix W is diagonal, and its elements are determined by the local statistical
properties of the image.
At last there can be another constraint set for ringing artifacts.
around the image edges.
Mainly ringing artifacts occur
So it is reasonable to treat these artifacts by leaving the edges and
processing around the edges.
To do this, we should detect the edges.
the horizontal, vertical and two diagonal directions is required.
can define another constraint sets.
Let
Line-processing along
Using these line-process, we
lv (i,j) denote the vertical line process at the site (i,j),
i.e.,
for i = 1,2, … , M and j = 1,2, …. ,N-1.
Then the quantity
gives the weighted variation of the entire image along the horizontal direction.
This leads to
another new constraint set
In similar way, there can be Cv(horizontal line-process), Cn(negative diagonal line-process) and
Cp(positive diagonal line-process).
Ⅲ.
Methods…
Since the projection onto the constraint sets have its numerical difficulties, the
smoothness constraint sets are introduced on subterms of directional variations.
Due to the
inherent blocking structure of BDCT compression, special attention is paid to the horizontal
/vertical smoothness sets Ch, Cv so that the resulting new smoothness sets can also remove
blocking artifacts in addition to ringing artifacts.
This is possible because of the flexibility of the
POCS approach. Here I will explain horizontal smoothness constrain set and the rest sets are
similar to the horizontal one.
Consider the horizontal variation term
This can be divided 8 subterms like
and
Since each of the smaller terms
Vhk ( f ) captures the horizontal variations of the image in part,
they can be used to define constraint sets to enforce the horizontal smoothness.
The following
constraint sets follow
This takes into the blocking artifacts automatically.
This is because the set
enforces the smoothness between block boundary columns.
So finally we get the following solution,
1) For each i = 1,2, … ,N, and for each j = 8b with b = 0,1, … ,Nk
Ch8 essentially
2) For the rest of the pixels
In the case of vertical line, it is done by same way.
solutions.
1) For
(i, j )  Dd with d = 1,2,5,6,9,10, … , compute
2) The rest of the image pixels are unchanged.
And in diagonal case, I will just give the
Ⅳ. Implementation… ( Parameters )
A. The Line Process
Directional derivatives are used to estimate the line process.
In the vertical line
process,
Then the line process
lv (i, j ) is determined using a threshold decision rule, i.e.,
And I use
where
h ,  2h
denote, respectively, the mean and the variance of
vertical block boundary sites.
manner.
And alpha is typically 0.5~2.0.
Dh f (i, j ) over the
The horizontal case is the same
At last, in the case of two diagonal directions,
B. The Visibility Weights
The blocking artifacts are limited only to the sites at the block boundaries, and ringing
artifacts are limited only within blocks inside which edges exit.
Therefore, the local activity for
these questionable sites in the above cases can be estimates from their neighboring sites.
The horizontal activity factors Wh(I,j) are determined following fashion.
1) Computable site (i,j)
: For the case the site which is inside block that contains no line process
jl, jr denote, respectively, the position of the leftmost and the rightmost column og the block.
2) Non-computable site(i,j)
① Inside block
ⅰ) No line process from (i, jl-1) to (i, jr+1) & both (i, jl-1) and (i, jr+1) are computable.
ⅱ) No line process from (i, jl-1) to (i, j) & (i,j) is computable.
No line process from (i, j) to (i, jr+1) & (i,j) is computable.
ⅲ) Otherwise
② Boundary
ⅰ) Wh (i, j ) 
Wh (i, j  1)  Wh (i, j  1)
2
for neither (i,j-1) nor (i,j+1) line process
ⅱ) If only one of the (i, j-1) and (i, j+1) is line process,
and
Wh (i, j ) equals to one of Wh (i, j  1)
Wh (i, j  1) which is not line process.
ⅲ) Otherwise
The minimum weight is used.
C. Smooth bound and
.
In the paper I studied, the method by which smooth bound is calculated.
bound we can solve following equation for
.
They suggest Newton’s method which guarantees the convergence.
Substitute

for various value.
But I just
And using this
Ⅴ.
Results…
I used my color facial image as a test image which is 256*256 pixels.
So I converted
it to gray-scale image for the further processing.
Fig 1.
Original BMP Image…
And I made a compressed image whose format is JPEG using embedded MARLAB function.
The compressed ratio is 80% and in practice original file was 192kb, compressed image was
3.2kb.
Fig 2.
80% Compressed JPEG Image…
You can see ‘Blocking artifacts’ and especially around two ears there are some ‘Ringing
artifacts’ in the compressed image, Fig 2.
Removing these artifacts was my mission.
very crucial role.
Completing this mission, detecting edge played
So I will show edge detection by 4 directional line-processing below.
Fig 3.
Line-Processing…
In Fig 3, you can see some artifacts because of block-based transform.
At last, I will show the
recovered image using POCS method.
Fig 4.
Recovered Image…
Using POCS method, I should remove ‘Blocking artifacts’ and ‘Ringing artifacts’ at the same
time.
Removing effects are not shown obviously.
So I will show original compressed image
and recovered image at the same time.
Fig 5.
Comparing original and recovered image…
Although there are some enhancements overall and more obvious around hair and around
forehead, as you see, removing these artifacts is not perfect.
incompleteness in the next section.
I will mention about these
Ⅵ.
Discussion…
Well I think my job was not perfect.
method works.
Actually I don’t know exactly how far this POCS
In addition although the paper I referred suggested 4 directional process, I
have only two directional process, horizontal and vertical direction.
At last I replaced solving
non-linear equation with just inserting various values in finding out the ramda.
When I simulated the method, I changed the ramda with various value.
In fact, there were
some changes with respect to ramda value but there were not significant change.
But there were some incompleteness in doing my job, the results are somewhat effective
removing artifacts as above picture.
Since I had to do my job by myself, I have to do many jobs by myself.
And this theme also
decided a few weeks ago because of my project mate’s abandoning.
situations,
I did my best and got some effective results.
In spite of these
If I have some extra time, I want to
solve non-linear equation in finding out ramda and process other two diagonal directions.
Ⅶ. References…
- Galatsanos,
Removal of compression artifacts using projections onto convex sets and line process
modeling
IEEE Trans. Image Processing, vol. 6, pp. 1345-1357, 1997
- S. Yang, Y. H. Hu, D. L. Tull, and T. Q. Nguyen,
Maximum likelihood parameter estimation for image ringing artifact removal
IEEE Trans. Circuits and Systems for Video Technology, vol. 11, 2001.
Ⅷ.
Appendix(MATLAB code)…
POCS.m File
figure(2); imshow(I_compr); title('Compressed JPEG image');
f = I_compr;
%%%%%%%%%%%%%%%%
%%% Main code %%%%%
[l_h, T_h, l_v, T_v, l_n, T_n, l_p, T_p] = Line_Processing(f);
%%% Need Line_Processing.m
%%%
Contain_Line.m
%%%
beta_25.m
[n_row, n_col] = size(f);
%%%%%%%%%%%%%%%%%
% w_h(i,j) for computable site(i,j) ------Eq.(46)
clc; clear all; close all;
for i = 1:n_row
for j = 1:n_col
% Loading uncompressed image
I = imread('test_original.bmp');
if ( mod(i,8)~=0 && mod(i,8) ~=1 && mod(j,8)~=0 && mod(j,8) ~=1 &&
~Contain_Line(i,j,l_v) )
[h s v] = rgb2hsv(I);
tot = 0;
I = v;
N = floor((j-1)/8);
figure(1); imshow(I); title('Original uncompressed image');
for k = N*8+1:N*8+7
tot = tot + abs( f(i,k) - f(i,k+1) );
% Making 90% compressed JPEG image
end
imwrite(I, 'test_compressed.jpg', 'quality',20);
w_h(i,j) = (1 + tot / 7)^(-1);
else
% Loading JPEG image
w_h(i,j) = 0;
I_compr = imread('test_compressed.jpg');
I_compr = ind2gray(I_compr, gray);
end
end
end
w_h(i,j) = ( w_h(i,J_l-1) + w_h(i, J_r+1) ) /2;
end
% w_h(i,j) for non-computable site(i,j)
% The second case
for i = 1:n_row
No_line = 1;
for j = 1:n_col
for k = J_l-1 : j
% In the case of site(i,j) is in the block which contains line processs
if
No_line = No_line * ~l_v(i,k);
Contain_Line(i,j,l_v)==1
end
% Considering boundary block
if ( No_line==1 && w_h(i,J_l-1)~=0 )
J_l = floor((j-1)/8)*8 + 1;
w_h(i,j) = w_h(i,J_l-1);
J_r = ceil(j/8)*8;
end
if J_l ==1
% The third case ------ Eq. (49)
J_l = J_l+1;
No_line = 1;
end
for k = j:J_r+1
if J_r == n_col
No_line = No_line * ~l_v(i,k);
J_r = J_r-1;
end
end
if ( No_line==1 && w_h(i,J_r+1)~=0 )
w_h(i,j) = w_h(i,J_l-1);
% The first case ------ Eq.(48)
end
No_line = 1;
if ( w_h(i,j) == 0)
for k = J_l-1 : J_r+1
w_h(i,j) = 1/(T_h+1);
No_line = No_line * ~l_v(i,k);
end
if ( No_line ==1 && w_h(i,J_l-1)~=0 && w_h(i,J_r+1)~=0 )
end
end
% In the case that site(i,j) is on the boundary
w_h(i,j) = 1/(T_h+1);
if ( mod(i, 8) == 1 || mod(i, 8) == 0 || mod(j,8) ==1 || mod(j, 8) == 0 )
end
if j ==1
w_h(i,j) = w_h(i,j+1);
elseif j == n_col
end
end
w_h(i,j) = w_h(i,j-1);
else
g(1:256, 1:256) = -100;
if ( l_v(i,j-1) ==0 && l_v(i,j+1)==0 )
w_h(i,j) = ( w_h(i,j-1) + w_h(i, j+1) )/2;
end
ramda = 6*10^(0);
for i = 1: n_row
if ( l_v(i,j-1) ==1 && l_v(i,j+1) ==0 )
for b = 0 : n_col/8-1
w_h(i,j) = w_h(i,j+1);
j = 8*b;
end
for k = 1:7
if ( l_v(i,j+1) ==1 && l_v(i,j-1) ==0)
g(i,j+k) = f(i, j+k) - beta_25(i,j,k,l_v,w_h,ramda)*( f(i, j+k) - f(i,j+k+1) );
w_h(i,j) = w_h(i,j-1);
end
g(i,j+k+1) = f(i, j+k+1) + beta_25(i,j,k,l_v,w_h,ramda)*( f(i,j+k) f(i,j+k+1) );
if ( l_v(i,j+1) ==1 && l_v(i,j-1) ==1)
w_h(i,j) = min( w_h(i,j+1), w_h(i,j-1) );
end
end
end
end
end
for b = 0 : n_col/8-2
j = 8*b;
k = 8;
g(i,j+k) = f(i, j+k) - beta_25(i,j,k,l_v,w_h,ramda)*( f(i, j+k) - f(i,j+k+1) );
if ( w_h(i,j) == 0 )
g(i,j+k+1) = f(i, j+k+1) + beta_25(i,j,k,l_v,w_h,ramda)*( f(i,j+k) - f(i,j+k+1) );
end
N = floor((j-1)/8);
end
for k = N*8+1:N*8+7
tot = tot + abs( f(i,k) - f(i,k+1) );
for i = 1:n_col
end
for j = 1: n_row
w_v(i,j) = (1 + tot / 7)^(-1);
if g(i,j) == -100
else
g(i,j) = f(i,j);
w_v(i,j) = 0;
end
end
end
end
end
end
figure; imshow(g)
% w_v(i,j) for non-computable site(i,j)
for i = 1:n_col
figure(7); subplot(121); imshow(f); title('Original compressed image'); subplot(122);
imshow(g); title('Recovered image by horizontal processing');
for j = 1:n_row
% In the case of site(i,j) is in the block which contains line processs
if
Contain_Line(i,j,l_h)==1
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Considering boundary block
% w_v(i,j) for computable site(i,j) ------Eq.(46)
J_l = floor((j-1)/8)*8 + 1;
for i = 1:n_col
J_r = ceil(j/8)*8;
for j = 1:n_row
if J_l ==1
if ( mod(j,8)~=0 && mod(j,8) ~=1 && mod(i,8)~=0 && mod(i,8) ~=1 &&
~Contain_Line(i,j,l_h) )
tot = 0;
J_l = J_l+1;
end
if J_r == n_row
J_r = J_r-1;
end
end
if ( No_line==1 && w_h(i,J_r+1)~=0 )
w_v(i,j) = w_v(i,J_l-1);
% The first case ------ Eq.(48)
end
No_line = 1;
if ( w_v(i,j) == 0)
for k = J_l-1 : J_r+1
w_v(i,j) = 1/(T_v+1);
No_line = No_line * ~l_h(i,k);
end
end
end
if ( No_line ==1 && w_v(i,J_l-1)~=0 && w_v(i,J_r+1)~=0 )
w_v(i,j) = ( w_v(i,J_l-1) + w_v(i, J_r+1) ) /2;
end
% In the case that site(i,j) is on the boundary
if ( mod(j, 8) == 1 || mod(j, 8) == 0 || mod(i,8) ==1 || mod(i, 8) == 0 )
% The second case
if j ==1
No_line = 1;
for k = J_l-1 : j
w_v(i,j) = w_v(i,j+1);
elseif j == n_row
No_line = No_line * ~l_h(i,k);
end
w_v(i,j) = w_v(i,j-1);
else
if ( No_line==1 && w_h(i,J_l-1)~=0)
if( l_h(i,j-1) ==0 && l_h(i,j+1)==0 )
w_v(i,j) = w_v(i,J_l-1);
w_v(i,j) = ( w_v(i,j-1) + w_v(i, j+1) )/2;
end
end
% The third case ------ Eq. (49)
if ( l_h(i,j-1) ==1 && l_h(i,j+1) == 0)
No_line = 1;
for k = j:J_r+1
No_line = No_line * ~l_h(i,k);
w_v(i,j) = w_v(i,j+1);
end
if ( l_h(i,j+1) ==1 && l_h(i,j-1) ==0 )
w_v(i,j) = w_v(i,j-1);
f(i,j+k+1) );
end
end
if ( l_h(i,j+1) ==1 && l_v(i,j-1) ==1)
end
w_v(i,j) = min( w_v(i,j+1), w_v(i,j-1) );
for b = 0 : n_row/8-2
end
j = 8*b;
end
k = 8;
end
g1(i,j+k) = f(i, j+k) - beta_25(i,j,k,l_h,w_v,ramda)*( f(i, j+k) - f(i,j+k+1) );
if ( w_v(i,j) == 0 )
g1(i,j+k+1) = f(i, j+k+1) + beta_25(i,j,k,l_h,w_v,ramda)*( f(i,j+k) - f(i,j+k+1) );
w_v(i,j) = 1/(T_v+1);
end
end
end
end
for i = 1:n_col
end
for j = 1:n_row
if g1(i,j) == -100
g1(1:256, 1:256) = -100;
g1(i,j) = f(i,j);
end
ramda = 2*10^(0);
for i = 1: n_col
end
end
for b = 0 : n_row/8-1
j = 8*b;
G = (g1+g) ./2;
for k = 1:7
figure; subplot(122);imshow(G); title('Recovered image by horizontal and vertical
g1(i,j+k) = f(i, j+k) - beta_25(i,j,k,l_h,w_v,ramda)*( f(i, j+k) - f(i,j+k+1) );
processing');
g1(i,j+k+1) = f(i, j+k+1) + beta_25(i,j,k,l_h,w_v,ramda)*( f(i,j+k) -
subplot(121);imshow(f); title('Original compressed image');
D_h_diff(:,1:n_col-1) = abs( f(:,1:n_col-1) - f(:,2:n_col) );
D_h_diff(:,n_col) = abs ( f(:, n_col) );
Conatain_line.m File
function x = Contain_Line(i,j,l)
% Calculating threshhold-----Eq.(46)
M = floor((i-1)/8); N = floor((j-1)/8);
mean_diff_h = mean(mean(D_h_diff));
[n_M, n_N] = size(l);
var_diff_h = var(var(D_h_diff));
alp = 1.7;
for i = M*8+1:M*8+8
T_h = mean_diff_h + alp*var_diff_h;
for j = N*8+1:N*8+8
if
l(i, j) ==1
% Line processing ------ Eq.(45)
x= 1; return
for i = 1:n_row
end
for j = 1:n_col
end
if abs(D_h_diff(i,j)) >= T_h
end
l_v(i,j) = 1;
else l_v(i,j) = 0;
x = 0; return
end
end
Line_Processing.m File
end
function [l_h, T_h, l_v, T_v, l_n, T_n, l_p, T_p] = Line_Processing(f)
figure; imshow(l_v); title('Vertical line processing');
[n_row, n_col] = size(f);
% Calculating Dvf'(i,j)----Eq.(44)
D_v_diff(1:n_row-1,:) = abs( f(1:n_row-1,:) - f(2:n_row,:) );
% Calculating Dhf'(i,j)----Eq.(44)
D_v_diff(n_row,:) = abs ( f(n_row,:) );
% Calculating threshhold-----Eq.(46)
% Calculating threshhold-----Eq.(46)
mean_diff_v = mean(mean(D_v_diff));
mean_diff_n = mean(mean(D_n_diff));
var_diff_v = var(var(D_v_diff));
var_diff_n = var(var(D_n_diff));
alp = 1.7;
alp = 1.7;
T_v = mean_diff_v + alp*var_diff_v;
T_n = mean_diff_n + alp*var_diff_n;
% Line processing ------ Eq.(45)
% Line processing ------ Eq.(45)
for i = 1:n_row
for i = 1:n_row
for j = 1:n_col
for j = 1:n_col
if abs(D_v_diff(i,j)) >= T_v
if abs(D_n_diff(i,j)) >= T_n
l_h(i,j) = 1;
l_n(i,j) = 1;
else l_h(i,j) = 0;
else l_n(i,j) = 0;
end
end
end
end
end
end
figure; imshow(l_h); title('Horizontal line processing');
figure; imshow(l_n); title('-45 degree digonal direction line processing');
% Calculating Dnf'(i,j)----Eq.(44)
% Calculating Dhf'(i,j)----Eq.(44)
D_n_diff(1:n_row-1,1:n_col-1) = abs( f(1:n_row-1,1:n_col-1) - f(2:n_row,2:n_col) );
D_p_diff(1:n_row-1,2:n_col) = abs( f(1:n_row-1,2:n_col) - f(2:n_row,1:n_col-1) );
D_n_diff(n_row, 1:n_col) = abs( f(n_row,1:n_col) );
D_p_diff(1:n_row,1) = abs( f(1:n_row,1) );
D_n_diff(1:n_row, n_col) = abs( f(1:n_row, n_col) );
D_p_diff(n_row,1:n_col) = abs( f(n_row,1:n_col) );
% Calculating threshhold-----Eq.(46)
mean_diff_p = mean(mean(D_p_diff));
var_diff_p = var(var(D_p_diff));
alp = 1.7;
T_p = mean_diff_p + alp*var_diff_p;
% Line processing ------ Eq.(45)
for i = 1:n_row
for j = 1:n_col
if abs(D_p_diff(i,j)) >= T_p
l_p(i,j) = 1;
else l_p(i,j) = 0;
end
end
end
figure; imshow(l_p); title('45 degree digonal direction line processing');
beta_25.m File
function x = beta_25(i,j,k,l,w, ramda)
if l(i,j+k) == 1
x = 0; return
else
x = ramda*(w(i,j+k)^2)/(1+2*ramda*(w(i,j+k)^2)); return
end