A Study of Spatial Color Interpolation
Algorithms for Single-Detector Digital Cameras
Introduction
Usage of digital cameras is spreading widely as they are easy image input devices. The increasing
popularity of digital cameras has provided motivation to improve all elements of the digital photography
signal chain. To lower cost, digital color cameras typically use a single image detector. Color imaging
with a single detector requires the use of a Color Filter Array (CFA) which covers the detector array. In
this arrangement each pixel in the detector samples the intensity of just one of the many-color
separations. The recovery of full-color images from a CFA-based detector requires a method of
calculating values of the other color separations at each pixel. These methods are commonly referred as
color interpolation or color demosaicing algorithms.
In a single-detector camera, varying intensities of light are measured at an rectangular grid of image
sensors. To construct a color image, a CFA must be placed between the lens and the sensors. A CFA
typically has one color filter element for each sensor. Obviously, the color interpolation algorithms
depend on the CFA configuration. Many different CFA configurations have been proposed. One of the
most popular is the Bayer pattern[1], which uses the three additive primary colors, red, green and blue
(RGB), for the filter elements. Since our goal in this study is to examine color interpolation algorithms,
we don't pay special attention to CFA patterns. Instead we exclusively work with the Bayer pattern in
RGB color space, as shown in Figure.(1) below. The selection of Bayer pattern is mainly due to its
popularity.
Figure 1
In Section II, we will describe the color interpolation algorithms included in this study. Section III gives
software implementation of some of these algorithms in Matlab. We then compare the performance of
these algorithms in Section IV.
Notice even though our study is specifically targeted at Bayer pattern with RGB color space, many
algorithms studies here can be extended to different pattern with different color space easily.
In our effort to include as many algorithms as possible, we have tried the most extensive search within
our resource limitation. But by no means this inclusion is complete and we apologize if some other work
in this area should have been included.
Algorithm Description
In this section we will describe a number of color interpolation algorithms used in digital cameras as
reported in liberatures. To describe these algorithms, we classify them into two distinct groups, nonadaptive algorithms and adaptive algorithms.
As their names suggest, non-adaptive algorithms denote those algorithms that perform interpolation in a
fixed pattern for every pixel (within a group). While adaptive algorithms imply that those algorithms can
detect local spatial features present in the pixel neighborhood, then make effective choices as to which
predictor to use for that neighborhood. In other words, adaptive algorithms pocesses some intelligence
and therefore are more sophisticated in general.
There are many ways to describe these algorithms. Graphical description, kernel convolution, arithmetic
equations or description via examples are all useful means to explain how the algorithm works. In our
presentation of each algorithm, we will try to adopt any one of these means or combinations of those
means, whichever makes the description as easily understandable as possible. As a result, different
algorithms may be subject to different description means.
Before we jump into the detailed description, we want to mention briefly some other algorithms that are
not included here in detail. This includes Median Filtering Method [28], some other linear estimator
methods in [29][31][33], Pattern Classification Method[32], and a new Gradient based method
described in a newly approved patent[30]. We would like include them all in the future.
Nearest Neighbor Replication

In this interpolation method[2][3][4][5], each interpolated output pixel is assigned the value of
the nearest pixel in the input image. The nearest neighbor can be any one of the upper, lower, left
and right pixels.

An example is illustrated below in Figure 2 for a 3x3 block in green plane. Here we assume the
left neighboring pixel value is used to fill the missing ones.
Figure 2
Bilinear Interpolation

For convenience, figure 1 is duplicated here

Interpolation of green pixels : the average of the upper, lower, left and right pixel values is
assigned as the G value of the interpolated pixel. For example : G8 = (G3+G7+G9+G13) / 4

Interpolation of red/blue pixels :
o Interpolation of a red/blue pixel at a green position : the average of two adjacent pixel
values in corresponding color is assigned to the interpolated pixel. For example : B7 =
(B6+B8) / 2 ; R7 = (R2+R12) / 2
o Interpolation of a red/blue pixel at a blue/red position : the average of four adjacent
diagonal pixel values is assigned to the interpolated pixel. For example : R8 =
(R2+R4+R12+R14) / 4 ; B12 = (B6+B8+B16+B18) / 4

For reference, refer to [3][4][5][6][7][8][9][10]
Cubic Convolution Interpolation

We will rely on system kernel convolution to describe this interpolation scheme. In general,
interpolation can be expressed as :
where h is the interpolation kernel and c(x,y) is the data sample itself at a pixel ( xk,yl ). Apparently, in
this formation, h, the interpolation kernel, is the main part of the interpolation algorithm.

Cubic convolution is a third-degree interpolation using 16 adjacent pixels and some typical cubic
convolution kernels are :
o
one-variable cubic interpolation :
and the parameter "a" has been reported as -1 in [22], 0.5 in [23] and 0.75 in [24].
o
two-variable cubic interpolation [25]:
where "b" and "c" are two parameters and (b,c) = (0.33,0.33), (b,c) = (1.5,-0.25) have
been reported in [26].
o
Cubic B-Spline interpolation [27] :
Cubic B-spline is the three-degree B-spline
Smooth Hue Transition Interpolation

One of the key objection of the bilinear interpolation is that the hues of adjacent pixels change
abuptly and in an unnatural manner. The Bayer CFA pattern, on the other hand, can be thought of
as consisting of a luminance channel (the relatively numerous green pixels) and a chrominance
channel (the relative sparse red and blue pixels). A scheme can be created to interpolate these
channels differently.

For convenience, figure 1 is duplicated here

Interpolation of green pixels : same as in bilinear interpolation. Also note interpolation of green
pixels has to be done before interpolations of red/blue pixels.

Interpolation of red/blue pixels :
o
o
o
o

The idea here is to try to impose a smooth transition in hue from pixel to pixel.
To do so, define blue "hue value" as: B/G. And red "hue value" can be analogously
defined.
Considering the interpolation of blue pixel values: there are three different cases of blue
pixel value interpolations.
 estimating blue pixel value at the green position and the adjacent blue pixels are
on left and right : e.g. B7 = G7 / 2 * (B6 / G6 + B8 / G8)
 estimating blue pixel value at the green position and the adjacent blue pixels are
on top and bottom : e.g. B13 = G13 / 2 * (B8 / G8 + B18 / G18)
 estimating blue pixel value at the red position : e.g. B12 = G12 / 4 * (B6 / G6 +
B8 / G8 + B16 / G16 + B18 / G18)
Interpolation of red pixel values can be carried out analogously
For reference, refer to [3][11][12]
Smooth Hue Transition Interpolation in Logarithmic Exposure
Space

Again, for convenience, figure 1 is duplicated here

Interpolation of green pixels : same as in bilinear interpolation. Also note interpolation of green
pixels has to be done before interpolations of red/blue pixels.

Key step : After the green pixel interpolations are performed, transfer the original pixel data
together with the interpolated green pixel values from linear exposure space into logarithmic
exposure space. Let's denote these log values ad Glog, Blog and Rlog, respectively.

Interpolation of red/blue pixels in log space :
o
o
o
In logarithmic space, define blue "hue value" as : Blog- Glog . And red "hue value" can be
analogously defined.
Considering the interpolation of blue pixel values : there are three different cases of blue
pixel value interpolations.
 estimating blue pixel value at the green position and the adjacent blue pixels are
on left and right : e.g. B7log = G7log + (B6log - G6log + B8log - G8log) / 2
 estimating blue pixel value at the green position and the adjacent blue pixels are
on top and bottom : e.g. B13log = G13log + (B8log - G8log + B18log - G18log) / 2
 estimating blue pixel value at the red postion : e.g. B12log = G12log + (B6log G6log + B8log - G8log + B16log - G16log + B18log - G18log) / 4
Interpolation of red pixel values can be carried out analogously

Finally, transfer the all pixel values back into linear exposure space

For reference, refer to [3][11][13]
Edge Sensing Interpolation Algorithm I

From our description of non-adaptive algorithms, it can be seen that most of the color
interpolation is done by averaging neighboring pixels indiscriminately. This causes an artifact -the "zipper effect" in the interpolated image. To combat with this artifact, it is natural to derive
an algorithm that can detect local spatial features present in the pixel neighborhood and then
makes effective choices as to which predictor to use that neighborhood. The result is a reduction
or elimination of "zipper-type" artifacts. And algorithms that involve this kind of "intelligent"
detection and decision process are referred as adaptive color interpolation algorithms.

Human visual systems are sensitive to edges present in the images and non-adaptive color
interpolation algorithms often fail around edges since they are not able to detect "edges". To deal
with edges, let's first look at an edge sensing algorithm[14][3][15].

Interpolation of green pixels :
o
o
o
o

First, define two gradients, one in horizontal direction, the other in vertical direction, for
each blue/red position. For instance, consider B8 : define two gradients
as
, where | . | denotes absolute value
Define some threshold value T
The algorithm then can be described as follows :
The choice of T depends on the images and can have defferent optimum values from
different neighborhoods. A particular choice of T is
[4]. In this case, the
algorithm becomes :
Interpolation of red/blue pixels : same as in Smooth hue transition interpolation in logarithmic
exposure space. Other methods may be used as well.
Edge Sensing Interpolation Algorithm II

A slightly different edge sensing interpolation algorithm is described in [17].

Interpolation of green pixels: same as in edge sensing interpolation algorithm I except that
horizontal and vertical gradients are defined differently. Refer to figure 6 below, to estimate G5
at B5, we define gradients as :
figure 6
and the actual algorithm follows as in the edge sensing interpolation algorithm I.

Interpolation of red/blue pixels : similar to bilinear interpolation algorithm except that color
difference is interpolated instead of color itself.
Linear Interpolation with Laplacian second-order correction
terms I

This algorithm [18] is designed for optimizing performance of images with horizontal and
vertical edges. To illustrate the algorithm, we'll utilize real examples to make description a little
bit easier to understand.

Interpolation of green pixels : refer to figure 6, duplicated below for convenience. We'll consider
estimating the green value at a blue pixel. Interpolation done at a red pixel can be carried out
similarly. Let's find out G5 at B5.
figure 7

o
Define horizontal and vertical gradients as :
o
The algorithm is :
Interpolation of red/blue pixels : refer to figure 7 below. Depending on the positions, we have
three cases.
o
case 1 : find red/blue values at a green pixel where nearest neighbors are in the same
column.
For instance, to estimate R4 at G4, use predictor
R4 = (R1 + R7) / 2 + (G4 - G1 + G4 - G7) / 4
o
case 2 : find red/blue values at a green pixel where nearest neightbors are in the same
row.
For instance, to estimate R2 at G2, use predictor
R2 = (R1 + R3) / 2 + (G2 - G1 + G2 - G3) / 4
o
case 3 : find red/blue values at a blue/red pixel. For instance, estimate R5 at B5.
Define two diagonal (negative and positive) directions as follows :
And the actual algorithm is described as :
Linear Interpolation with Laplacian second-order correction
terms II

An extension to previous interpolation method with Laplacian second-order correction terms is
described in [19][20].

Interpolation of green pixels : again, we will borrow the figure from previous algorithm
descriptions. For convenience, it's duplicated below. We'll consider estimating the green value at
a blue pixel. Interpolation done at a red pixel can be carried out similarly. Let's find out G5 at B5.
o
Define eleven color correction terms with corresponding predictors as follows :
CC1 = B5 - B3 + B5 - B7; P1 = ( G4 + G6 ) / 2 + CC1 / 4;
CC2 = B5 - B1 + B5 - B9; P2 = ( G2 + G8 ) / 2 + CC2 / 4;
CC3 = B5 - B1 + B5 - B7; P3 = ( G2 + G6 ) / 2 + CC3 / 4;
CC4 = B5 - B7 + B5 - B9; P4 = ( G6 + G8 ) / 2 + CC4 / 4;
CC5 = B5 - B9 + B5 - B3; P5 = ( G4 + G8 ) / 2 + CC5 / 4;
CC6 = B5 - B3 + B5 - B1; P6 = ( G2 + G4 ) / 2 + CC6 / 4;
CC7 = B5 - B1; P7 = G2 + CC7 / 2;
CC8 = B5 - B7; P8 = G6 + CC8 / 2;
CC9 = B5 - B9; P9 = G8 + CC9 / 2;
CC10 = B5 - B3; P10 = G4 + CC10 / 2;
CC11 = B5 - B1 + B5 - B3 + B5 - B7 + B5 - B9; P11 = ( G2 + G4 + G6 + G8) / 4 + CC11 / 8;
notice CC1 - CC6 are Laplacian second-order derivative operators and CC7 - CC10 are
Laplacian first-order derivative operators.
o
The algorithm is :
If min{CC1, CC2 } < a given Threshold (e.g. 69 in 8-bit log space)
G5 = the corresponding predictor; (e.g. if CC1 < CC2 and CC1 <
Threshold, then G5 = P1)
Else if min{CC3,CC4,CC5,CC6} < a given Threshold (e.g. 170 in 8-bit log space)
G5 = the corresponding predictor;
Else if min{CC7,CC8,CC9,CC10} < a given Threshold (e.g. 250 in 8-bit log
space)
G5 = the corresponding predictor;
Else
G5 = P11;
End

Interpolation of red/blue pixels : refer to figure 7 below. Depending on the positions, we have
three cases.
figure 7
o
case 1 : find red/blue values at a green pixel where nearest neighbors are in the same
column. For instance, to estimate R4 at G4 :
 Define three color corrections terms and their corresponding predictors as follows:
CC1 = G4 - G1 + G4 - G7; P1 = ( R1 + R7 ) / 2 + CC1 / 2;
CC2 = G4 - G1; P2 = R1 + CC2;
CC3 = G4 - G7; P3 = R7 + CC3;
The algorithm is:
If CC1 < a given Threshold (e.g. 69 in 8-bit log space)
R4 = P1;
Else if CC2 < CC3
R4 = P2;
Else
R4 = P3;
End
o
case 2 : find red/blue values at a green pixel where nearest neightbors are in the same
row. For instance, to estimate R2 at G2:
 Define three color corrections terms and their corresponding predictors as follows:
CC1 = G2 - G1 + G2 - G7; P1 = ( R1 + R3 ) / 2 + CC1 / 2;
CC2 = G2 - G1; P2 = R1 + CC2;
CC3 = G2 - G7; P3 = R3 + CC3;

The algorithm is :
If CC1 < a given Threshold (e.g. 69 in 8-bit log space)
R2 = P1;
Else if CC2 < CC3
R2 = P2;
Else
R2 = P3;
End
o
case 3 : find red/blue values at a blue/red pixel. For instance, estimate R5 at B5 :
 As the green pixel interpolation scheme above, we define eleven color correction
terms and their corresponding predictors as follows :
CC1 = G5 - G4 + G5 - G6; P1 = ( R4 + R6 ) / 2 + CC1 / 2;
CC2 = G5 - G2 + G5 - G8; P2 = ( R2 + R8 ) / 2 + CC2 / 2;
CC3 = G5 - G2 + G5 - G6; P3 = ( R2 + R6 ) / 2 + CC3 / 2;
CC4 = G5 - G6 + G5 - G8; P4 = ( R6 + R8 ) / 2 + CC4 / 2;
CC5 = G5 - G4 + G5 - G8; P5 = ( R4 + R8 ) / 2 + CC5 / 2;
CC6 = G5 - G2 + G5 - G4; P6 = ( R2 + R4 ) / 2 + CC6 / 2;
CC7 = G5 - G2; P7 = R2 + CC7;
CC8 = G5 - G6; P8 = R6 + CC8;
CC9 = G5 - G8; P9 = R8 + CC9;
CC10 = G5 - G4; P10 = R4 + CC10;
CC11 = G5 - G2 + G5 - G4 + G5 - G6 + G5 - G8;
P11 = ( R2 + R4 + R6 + R8 ) / 4 + CC11 / 4;
The use of these color correction terms and predictors is the same as in the green pixel
interpolation schemes.
Interpolation using a Threshold-based variable number of
gradients

This algorithm is described in [21].

Basic procedures :
1. a set of gradients is determined from the color values in the 5x5 neighborhood centered at
the pixel under consideration. Each gradient corresponds to a different direction.
2. for each set of gradients, a threshold value is determined and the threshold is used to
select a subset of gradients. Low-valued gradients indicate pixels having similar color
values whereas high-valued gradients would be expected in regions of the image where
there are many fine details or sharp edges.
3. the subset of gradients is used to locate regions of pixels that are most like the pixel under
consideration. The pixels in these regions are then weighted and summed to determine the
average difference between the color of the actual measured center pixel value and the
missing color.

Interpolation of the green,red/blue value at the blue/red pixel : consider figure 8 below, we want
to estimate G13 and B13 at R13.
figure 8
o
figure 9
Form eight gradients as follows :
Gradient N = |G8 - G18| + |R3 - R13| + |B7 - B17| / 2 + |B9 - B19| / 2 + |G2 - G12| / 2 + |G4 - G14| / 2 ;
Gradient E = |G14 - G12| + |R15 - R13| + |B9 - B7| / 2 + |B19 - B17| / 2 + |G10 - G8| / 2 + |G20 - G18| / 2;
Gradient S = |G18 - G8| + |R23 - R13| + |B19 - B9| / 2 + |B17 - B7| / 2 + |G24 - G14| / 2 + |G22 - G12| / 2;
Gradient W = |G12 - G14| + |R11 - R13| + |B17 - B19| / 2 + |B7 - B9| / 2 + |G16 - G18| / 2 + |G6 - G8| / 2;
Gradient NE = |B9 - B17| + |R5 - R13| + |G8 - G12| / 2 + |G14 - G18| / 2 + |G4 - G8| / 2 + |G10 - G14| / 2;
Gradient SE = |B19 - B7| + |B25 - R13| + |G14 - G8| / 2 + |G18 - G12| / 2 + |G20 - G14| / 2 + |G24 - G18| / 2;
Gradient NW = |B7 - B19| + |R1 - R13| + |G12 - G18| / 2 + |G8 - G14| / 2 + |G6 - G12| / 2 + |G2 - G8| / 2;
Gradient SW = |B17 - B9| + |R21 - R13| + |G18 - G14| / 2 + |G12 - G8| / 2 + |G22 - G18| / 2 + |G16 - G12| / 2;
We show a numerical examples here too :
o
Gradient
N
E
S
W
NE
Value
12
13
7
8
4
table 1
SE
NW
SW
7
12
14
Determine a threshold and select a subset of gradients :
the threshold is determined by T = k1*Min + k2 * (Max - Min), where Min is the
minimum gradient value and Max is the maximum gradient value. k1 and k2 are
determined expirically as 1.5 and 0.5, respectively.
Here k1*Min accounts for the case in which the gradients are all very similar, so that we
wish to inclue all of them by setting a threshold that exceeds them. k2 * (Max - Min)
accounts for the case in which there is a significant difference between the maximum and
minimum gradient values.
consider the numerical example illustrated in table 1, Min = 4, Max = 14 and T = 11
consequently. Therefore the selected subset of gradients consists of those with value less
than 11:
Gradient subset = {S, W, NE, SE}
o

Locate the pixels in the regions corresponding to the subset of gradients and to use those
pixels to determine a color difference between the center pixel color and the color to be
recovered.
 Determine the average green, blue and red values in the gradient subset regions :
continue with the above example, we have :
G
B
R
S
G18
( B17 + B19 ) / 2
( R13 + R23 ) / 2
W
G12
( B7 + B17 ) / 2
( R11 + R13 ) / 2
NE
( G4 + G8 + G10 + G14 ) / 4
B9
( R13 + R5 ) / 2
SE
( G14 + G18 + G20 + G24 ) / 4
B19
( R13 + R25 ) / 2

Form the average color values in the gradient subset regions to get Gsum, Bsum and
Rsum
Continue with the example, we have :
Gsum = G18 + G12 + ( G4 + G8 + G10 + G14 ) / 4 + ( G14 + G18 + G20 G24 ) / 4;
Bsum = ( B17 + B19 ) / 2 + ( B7 + B17 ) / 2 + B9 + B19;
Rsum = ( R13 + R23 ) / 2 + (R11 + R13 ) / 2 + ( R13 + R5 ) / 2 + ( R13 + R25 ) / 2;

Find the normalized color difference by dividing the difference of two sums by
the number of gradients in the threshold subset, and add this normalized color
difference to the pixel value under consideration to form the other two missing
color components.
For example, G13 = R13 + ( Gsum - Rsum ) / 4; B13 = R13 + ( Bsum - Rsum ) / 4;
Interpolation of the blue/red value at the green pixel : consider figure 9 below, we want to
estimate R13 and B13 at G13.
o
Form eight gradients as follows :
Gradient N = |G3 - G13| + |B8 - B18| + |G7 - G17| / 2 + |G9 - G19| / 2 + |R2 - R12| / 2 + |R4 R14| / 2 ;
Gradient E = |R14 - R12| + |G15 - G13| + |G9 - G7| / 2 + |G19 - G17| / 2 + |B10 - B8| / 2 + |B20 B18| / 2;
Gradient S = |B18 - B8| + |G23 - G13| + |G19 - G9| / 2 + |G17 - G7| / 2 + |R24 - R14| / 2 + |R22 R12| / 2;
Gradient W = |R12 - R14| + |G11 - G13| + |G17 - G19| / 2 + |G7 - G9| / 2 + |B16 - B18| / 2 + |B6 B8| / 2;
Gradient NE = |G9 - G17| + |G5 - G13| + |R4 - R12| + |B10 - B18|;
Gradient SE = |G19 - G7| + |G25 - G13| + |B20 - B8| + |R24 - R12|;
Gradient NW = |G7 - G19| + |G1 - G13| + |B6 - B18| + |R2 - R14|;
Gradient SW = |G17 - G9| + |G21 - G13| +|R22 - R14| + |B16 - B8|;
o
As described earily, determine a threshold and select a subset of gradients. Again, T =
k1*Min + k2 * (Max - Min), where k1 = 1.5 and k2 = 0.5
Again, Locate pixels in the selected regions and use those pixels to determine a color difference
between the center pixel color and the color to be recovered, finally add this color difference to
produce an estimate for the missing color value.
Pattern Recognition Interpolation

The edge sensing algorithm can be generalized into a general pattern recognition approach to
luminance interpolation [16][3].

Again, like edge sensing algorithm, pattern recognition interpolation pays special attention to
luminance channels. So we emphasize on the interpolation of green pixels first.

Interpolation of green pixels :
o First note that each pixel that needs to be interpolated for green color is surrounded by 4
green pixels, refer to duplicated figure 1 below.
o
Let H refer to a green pixel with a code value greater than the average of all four pixels in
neighborhood, and let L refer to a green pixel with a code value less than the average.
Then these four green pixels can form four unique patterns (up to rotation) :
o
For convenience, (a) and (d) are called edge patterns, (b) is called a stripe pattern and (c)
is called a corner pattern.
Therefore for each pixel that is to be interpolated, first determine its pattern, then
depending on its pattern, different predictor is used :
 edge pattern : refer to figure above, G12 = median{G7,G11,G13,G17}, where
median{G7,G11,G13,G17} is defined as the average of the second ( B )and the
third (C) largest values in {G7,G11,G13,G17}. i.e, if we sort {G7,G11,G13,G17}
as A>B>C>D, then G12 = median{G7,G11,G13,G17} = ( B + C ) / 2
 stripe pattern : a larger pixel neighborhood as described in figure 3 is needed
o
Figure 3
and
where M = SUM(G) / 4 , S = SUM(X) / 8 and 'clip'
function constrains the prediction to lie between the values B and C. If the
prediction is greater than B, it is set o B; if the prediction is less than C, it is set to
C

corner pattern : again, a larger pixel neighborhood is needed, as described in
figure 4.
figure 4
and just as in the stripe pattern case,
L's} and S = SUM(X) / 4
, where M = median{H's,
Interpolation of red/blue pixels : can use any method described before, for example, bilinear
interpolation or smooth hue transition interpolation
Pattern Matching Interpolation

People have observed that there's a high degree of correlation between the RGB channels in
natural scenes.

The idea here is to exploit the correlations between the green, red and blue components, and
determine the missing components from the pixel contexts. Since green is sampled with the
highest sampling frequency, the reproduction of the green components is much more accurate
than the reproduction of the red or blue components. For this reason, this algorithm specifically
targets at the interpolations of chrominance channels.

Interpolation of green pixels : use any method described before.

Interpolation of red/blue pixels :
o
o
estimate red/blue values at green pixels : use any method described before. For instance,
use Bilinear Interpolation, i.e., use the average of two adjacent color samples.
estimate blue values at red pixels :
 First let's define "pattern" p(x,y) for a red pixel at position (x,y), refer to figure 5,
as a four-dimensional vector or a green pattern : P(x,y) = [G(x-1,y); G(x+1,y);
G(x,y-1); G(x,y+1)], where G(x,y) is the green value at position (x,y).
 Next compare the green pattern p(x,y) with its four neighboring green patterns and
measure their similarities by :
where || . || denotes the l-1 norm of a vector


Sort these delta values and let = the ith largest value, therefore
Define some threshold value T
 The actual algorithm is as follows : if
< T, then we have high confidence that
the image is smooth around (x,y), so the blue value is the average of all four adjacent diagonal pixel
values. else the desired blue value
=

Let's refer to a real example now. Consider figure 5, we want to estimate the blue
value at R15 :
figure 5
the procedures are :
1. P15 = [G14; G16; G9; G21];
P8 = [G7; G9; G2; G14];
P10 = [G9; G11; G4; G16];
P20= [G19; G21; G14; G26];
P22 = [G21; G23; G16; G28]
2.
3. sort them. Suppose the result is :
4.
o

estimate red values at blue pixels can be carried out analogously
For reference, refer to [10]
Algorithm Comparison
In this section we will try to compare different color interpolation algorithms introduced and
implemented in previous sections. In doing so, we compare these algorithms from three aspects. First,
we will compare them using subjective measures, we will try to identify the major difference among
these algorithms with visual observations. Secondly, we will switch to objective quality measure, in our
case, we choose Mean Squared Error (MSE) between the original test image and the reconstructed
image. MSE is chosen for its simplicity and popularity. Finally, we compare their computational cost,
which is vital for implementation in practice.
o
o
o
o
o
o
o
Nearest Neighbor Replication Interpolation : significant errors occur throughout the
image, very evident blocking effect, very bad for sharp edges. Unacceptable for still
imaging systems. In video applications, many of these problems might be averaged away.
Bilinear Interpolation : much improved performance over Nearest Neighbor Replication
Interpolation; some blurring due to averaging adjacent pixels; introduce the so-called
"zipper" effect
Smooth Hue Algorithms : less blur compared to Bilinear algorithm as seen in the 'BMW'
image. On the other hand, significant 'hue' interpolation error is imposed as seen in the
image "flowers". This is due to the division (or subtraction in Log space) in R/B channel
interpolation where large G difference between adjacent pixels will cause huge R/B error
during interpolation.
Edge Sensing Algorithms : as their names suggest, indeed sharper images can be seen.
Since R/B interpolation uses the same algorithm as in Smooth Hue Transition algorithms,
significant "hue" error can be seen in the image "flowers". Not very suitable for colored
small objects.
Linear Interpolation with Laplacian 2nd order color correction terms I : gives one of
the best quality among all algorithms. Especially good for edges.
Interpolation using Threshold-based Variable Number of Gradients : appears to give
the best result among all algorithms. Sharp edges as well.
Pattern Recognition : fairly good results, but edges aren't dealt as well as variable
number gradients method or Laplacian color correction method
Objective Quality Measure

To compare these algorithms objectively, we adopt Mean Square Error (MSE) as our quality
metrics. The MSE between the original image Io(x,y) and the interpolated image Ir(x,y) for each
color channel is defined as :
where m x n is the size of the image

The MSE values for each algorithm that are implemented in Section III are depicted below for
three color channels separately. We normalized the MSE value to the minimum since we are only
interested in the relative performance among these algorithms.

From these plots, we conclude that based on objective measure using MSE :
o
o
o
Nearest Neighbor Replication Interpolation gives significant larger mean squared error
than other algorithms and therefore the interpolated image is expected to show the worst
artifacts, which can be verified in previous subsection.
In terms of Green (or Luminance) channel interpolations, Variable Number Gradients
Method gives the best performance in both examples. Laplacian Color Correction I
method also gives impressive results. And as far as the rest algorithms are concerned,
their MSE values don't show any major difference.
In terms of Red/Blue (or Chrominance) channel interpolations, there's no obvious
superior algorithm that can stand out alone. Variable Number Gradients Method gives
slightly better results.
Computational Cost

Here we will compare the computational cost associated with each of the algorithms
implemented. The way we compute the cost is to count the flops occurred in Matlab when
processing some input images. For those non-adaptive algorithms, the cost is fixed regardless of
the content of the input image as long as the image size is fixed. While for adaptive algorithms,
the computational cost is scene dependent since those algorithms involve comparisons or
decision making steps where branching can occur. This variation on computational cost due to
image content is, however, not evident for two examples we included here.
Since the computational cost is solely determined from the software point of view, it is of course
related to the actual implementation of the algorithm. Efficient program reduces the cost. In our
study no special effort is denoted to deal with this implementation-dependent feature of the cost
and none of the algorithm code is tuned intentionally to increase its efficiency, nor do we
decrease it on purpose. We assume the behavior of our implementation observes the average in
general.

Observations :
o
o
o
o
o
Nearest Neighbor Replication Interpolation doesn't involve any arithmetic operation and
therefore doesn't impose any computational cost
Excluding Nearest Neighbor Replication algorithm, Bilinear interpolation imposes the
least amount of computation among the other eight interpolation schemes
On the other hand, interpolation using threshold-based variable number gradients
method has a significant computational cost penalty compared to others, and it's very
impractical for real-time digital video applications.
Linear interpolation with Laplacian second-order color correction terms I also imposes
a large computational cost compared to other algorithms but with a much less penalty
compared to variable number gradients method.
The rest algorithms have similar computational cost
Reference
[1].Bayer,Bryce E.
[2].Ozawa,N.
"Color imaging array"
U.S. Patent 3,971,065
"Chrominance signal interpolation device for a color camera"
U.S. Patent 4,716,455
[3].Adams, James E. "Interactions between color plane interpolation and other image processing
functions in electronic photography" Proceedings of SPIE Vol. 2416 P.144-151
[4].Sakamoto,Tadashi. et.al., "Software pixel interpolation for digital still cameras suitable for a 32-bit
MCU" IEEE Transactions on Consumer Electronics. Vol.44, No.4, P.1342-1352, November 1998
[5].Zen,Hidemori. et.al., "A New digital signal processor for progressive scan CCD"
Transactions on Consumer Electronics. Vol.44, No.2, P.289-295, May 1998
IEEE
[6].Cok,David R. "Single-chip electronic color camera with color-dependent birefringent optical
spatial frequency filter and red and blue signal interpolating circuit" U.S. Patent 4,605,956
[7].Parulski,Kenneth A. "Color Filters and Processing Alternatives for one-chip cameras"
Transactions on Electron Devices. Vol.ED-32, NO.8, August 1985
[8].Chan,Wen-Hsin. et.al.,
Vol.2654. P.164-171
"A Mega-Pixel resolution PC Digital Still Camera"
IEEE
Proceedings of SPIE
[9].Tsai, Y.Tim. et.al., "Optimized Image Processing Algorithms for a Single Sensor Camera" 1997
IEEE Pacific Rim Conference on Communications, Computers and Signal Processing, Vol.2, P.10101013
[10].Wu,XiaoLin. et.al., "Color Restoration from Digital Camera Data by Pattern Matching"
Proceedings of SPIE Vol.3018 P.12-17
[11].Cok,David R. "Signal processing method and apparatus for producing interpolated chrominance
values in a sampled color image signal" U.S. Patent 4,642,678
[12].Parulski,Kenneth A. et.al.,
SPIE Vol.1448 P.45-58
"A High-Performance Digital Color Video Camera"
Proceedings of
[13].Weldy,John A. "Optimized design for a single-sensor color electronic camera system"
Proceedings of SPIE Vol.1071 P.300-307
[14].Hibbard,Robert H. "Apparatus and method for adaptively interpolating a full color image utilizing
luminance gradients" U.S. Patent 5,382,976
[15]Adams, James E. et.al.,
[16]Cok,David R.
4,630,307
"Color Processing in Digital Cameras"
Eastman Kodak Company
"Signal processing method and apparatus for sampled image signals"
U.S. Patent
[17]Laroche,Claude A. "Apparatus and method for adaptively interpolating a full color image utilizing
chrominance gradients" U.S. Patent 5,373,322
[18]Hamilton, Jr. et.al.,
U.S.Patent 5,629,734
"Adaptive color plane interpolation in single sensor color electronic camera"
[19]Adams,James E. et.al.,
U.S.Patent 5,506,619
"Adaptive color plane interpolation in single color electronic camera"
[20]Adams,James E. et.al., "Design of practical color filter array interpolation algorithms for digital
cameras" Proceedings of SPIE Vol.3028 P.117-125
[21]Chang, Ed. et.al., "Color Filter Array Recovery Using a Threshold-based Variable Number of
Gradients" to be published in Proceedings of SPIE, January, 1999
[22]Rifman, S.S. et.al., "Evaluation of Digital Correction Techniques for ERTS Images - Final
Report" Report 20634-6003-TU-00 TRW Systems, California, July 1974
[23]Keys, Robert.G. et.al., "Cubic Convolution Interpolation for Digital Image Processing"
Transactions on Acoustic, Speech and Signal Processing, Vol ASSP-29, P1153-1160, 1981
IEEE
[24]Simons, K.W. "Digital Image Reconstructon and Resampling for Geometric Manipulation"
Proceedings of IEEE Symp. on Machine Processing of Remotely Sensed Data P.3A-1--3A-11,1975
[25]Mitchell, Don P. et.al., "Reconstruction Filters in Computer Graphics"
(SIGGRAPH'88 Proceedings), Vol.22, No.4, P.221-228, August 1988
Computer Graphics,
[26]Schreiber, William F. et.al., "Transformation between Continuous and Discrete Representations of
Image: A Perceptual Approach" IEEE Transactions on Pattern Analysis and Machine Intelligence
Vol.PAMI-7 No.2, P.178-186 March 1985
[27]Hou, Hsieh S. et.al., "Cubic Splines for Image Interpolation and Digital Filtering" IEEE
Transactions on Acoustic, Speech and Signal Processing Vol.ASSP-26 P.508-517, 1987
[28]Freeman,William T. "Method and apparatus for reconstructing missing color samples"
Patent 4,663,655
U.S.
[29]Wober,Munib A. et.al., "Method and apparatus for recovering image data through the use of a
color test pattern" U.S.Patent 5,475,769
[30]Matraszek,Tmoasz A. et.al., "Gradient based method for providing values for unknown pixels in
a digital image" U.S.Patent 5,875,040
[31]Brainard,D.H. et.al., "Bayesian method for Reconstructing Color Images from Trichromatic
Samples" IS&T 47th Annual Conference P.375-380, 1994
[32]Shimizu,Eiichi et.al., "The Digital Camera Using New Compression and Interpolation
Algorithm" IS&T 49th Annual Conference P.268-27, 1996
[33].Wu,Chi-hsin et.al., "Reconstruction of Color Images from a Single-Chip CCD Sensor Based on
Markov Random Field Models" Proceedings of SPIE. Vol.2564 P.282-288