Color Image Processing
in the block DCT Space
Jayanta Mukhopadhyay
Department of Computer Science & Engineering
Indian Institute of Technology, Kharagpur, 721302, India
jay@cse.iitkgp.ernet.in
1
What is COLOR?
Selective emission/reflectance of different wavelengths
2
Color Spectrum
Illumination
Reflectance
•Spectrum: Intensity as a function of wavelength.
3
Color Stimuli
Illumination
Reflectance
•The colour of an object: is the product of the spectrum of
the incident light with the light absorption and/or reflection
properties of the object.
4
What is Perceived Color?
• The response generated by a stimulus in the cones
gives the perceived color
• Three responses
5
Human color perception
• For human eye
• – Approximately 65% of all cone are
sensitive to red light
• – 33% are sensitive to green light
• – 2% are sensitive to blue light
• But blue cones are the most sensitive.
6
Tri-stimulus Values
•
Integration over wavelength
X = ∫C(λ)x(λ) dλ = Σ C(λ)x(λ)
Y = ∫C(λ)y(λ) dλ = Σ C(λ)y(λ)
Z = ∫C(λ)z(λ) dλ = Σ C(λ)z(λ)
Real colors span a subset of the XYZ space.
• Two different stimuli can have same XYZ values.
–Metameris
• Additive color mixtures modeled by addition in XYZ
space.
•
7
Amounts of three primaries needed
to match all wavelengths of the
spectrum
The curves represented by the cone’s
reception are not simple peaks. They are,
instead, quite complex curves.
They even go negative!
RGB is not capable of reproducing every
single color we can see.
8
Perceived Color Features
• Intensity
– Sum of the spectrum
– Energy under the spectrum
• Hue
– Mean wavelength of the spectrum
– What wavelength sensation is
dominant?
• Saturation
– Standard deviation of the spectrum
– How much achromatic/gray
component?
• Chrominance – Hue and saturation
9
Limitation of Tri-Stimulus Model
• No physical feel as to how colors are
arranged.
• How do brightness change?
• How does hue change?
• Subtractive like paint cannot be
modeled by XYZ space.
10
CIE XYZ Space
•
Intensity (I) – X+Y+Z
• Chrominance (x,y) - (X/I, Y/I)
–Chromaticity chart
– Projection on a plane with normal (1,1,1)
–Reduction of dimension
–Similar to 3D to 2D in geometry
– Each vector from (0,0,0) is an isochrominance line
– Each vector maps to a point in the
chromaticity chart
11
RGB-to-XYZ Space
12
CIE Chromaticity Chart
• Shows all the visible colors
• Achromatic Colors are at
(0.33,0.33)
– Called white point
• The saturated colors at the
boundary
– Spectral Colors
13
Chromaticity Chart: Hue
• All colors on straight line
from white point to a
boundary has the same
spectral hue
– Dominant wavelength
14
Chromaticity Chart: Saturation
• Purity (Saturation)
– How far shifted towards
the spectral color
– Ratio of a/b
– Purity =1 implies spectral
color with maximum
saturation
15
Color Reproducibility
• Only a subset of the 3D
CIE XYZ space called 3D
color gamut
• Projection of the 3D
color gamut
–Triangle
–2D color gamut
Large if using more saturated
primaries
• Cannot describe
brightness range
reproducibility
16
Standard Color Gamut
17
Color spaces
• RGB (CIE), RnGnBn (TV – NTSC)
• XYZ (CIE)
• UVW (UCS de la CIE), U*V*W* (UCS modified
by the CIE)
• YUV, YIQ, YCbCr
• HSV, HLS, IHS
• Munsel colour space (cylindrical representation)
• CIELuv
• CIELab
18
RGB-to-YCbCr
19
Color Enhancement
20
Color Processing in the
Compressed Domain



Computation with reduced storage.
Avoid overhead of inverse and
forward transform..
Exploit spectral factorization for
improving the quality of result and
speed of computation.
21
Basic Approaches
• Modify the DC coefficient for increasing brightness.
Aghaglzadeh and Ersoy (1992), Opt.Engg.
• Modify AC coefficients for increasing contrast.
Tang, Peli and Acton (2003), IEEE SPL
• A combination of both.
S. Lee (2007), IEEE CSVT
• Preserve also colors by processing DCT of chromatic
components.
22
Different methods
• Multi-Contrast Enhancement with Dynamic
Range Compression (S. Lee (2007), IEEE CSVT)
Modification of DC coefficients and AC coefficients
(following similar strategy of multi-contrast
enhancement).
Normalized DC coefficients (x) are modified as follows:
23
Proposed Approach
• Adjust background illumination.
Use DC coefficients of the Y component.
• Preserve Local Contrast.
Scale AC coefficients of the Y component appropriately.
• Preserve Colors.
Preserve Color Vectors in the DCT domain.
DCT coefficients of Cb and Cr components.
24
Contrast : Definition
Let μ and σ denote the mean and standard deviation of
an image. Contrast ζ of an image is defined here as:

 

.
Weber Law:   L
L
where  L is the difference in luminance between a
stimulus and its surround, and L is the luminance of the
surround
25
Theorem on Contrast Preservation in
the DCT Domain
Let d be the scale factor for the DC coefficient and a
a be the scale factor for the AC coefficients of a DCT
block Y. The processed DCT block Ye is given by:
Ye (i, j )
 d Y (i , j ), i  j 0
 { aY (i , j ), otherwise
The contrast of the processed image then becomes a / d
times of the contrast of the original image.
In this algorithm
d = a =  for preservation of the contrast.
26
Preservation of Colours in the DCT
Domain
Let U and V be the DCT coefficients of the Cb and Cr
components, respectively. If the luminance component Y
of an image is uniformly scaled by a factor  , the colors
of the processed image with Ye , Ue and Ve are preserved
by the following operations:
U (i , j )
128)) 128, i  j 0
N
U ( i , j ), otherwise
N ( (
U e (i, j )  {
V (i , j )
128)) 128, i  j 0
N
V ( i , j ), otherwise
N ( (
Ve (i, j )  {
27
Enhancement by Scaling Coefficients
• Find the scale factor by mapping the DC
coefficient with a monotonically increasing
function.
• Apply scaling to all other coefficients in all the
components.
• For blocks having greater details, apply block
decomposition and re-composition strategy.
28
Mapping functions for adjusting the
local background illumination

(TW)
f (Y (0, 0))
Y (0, 0)
(DRC)
Lee, CSVT’07
Mitra and Yu , CVGIP’87
(SF)
De, TENCON’89
29
Monotonic Mapping Functions
30
Scaling only DC coefficients
31
Scaling both DC and AC
coefficients
32
Preservation of Contrast and
Color
original
33
Enhancement of Blocks with more details
Apply CES
on smaller
blocks
Block
Decompos.
8x8 block
Smaller DCT
blocks
Block
Composition
Enhanced Block
34
Removal of Blocking Artifacts
original
35
Some Results
original
MCEDRC
AR
TW-CES-BLK
MCE
MSR
36
Enhancement near Edges
AR
TW-CES-BLK
MCE
DRC-CES-BLK
MCEDRC
SF-CES-BLK
37
Some Results
original
MCEDRC
AR
TW-CES-BLK
MCE
MSR
38
Enhancement near edges
AR
TW-CES-BLK
MCE
DRC-CES-BLK
MCEDRC
SF-CES-BLK
39
Some Results
original
MCEDRC
AR
TW-CES-BLK
MCE
MSR
40
Enhancement near edges
AR
TW-CES-BLK
MCE
DRC-CES-BLK
MCEDRC
SF-CES-BLK
41
Metrics for Comparison
QM 
4 xy2 x y
 x2   y2 
 
xy
JPEG Quality Metric (JPQM)
CM    2    2  0.3  2    2
Wang and Bovic (SPL, 2002)
Wang and Bovic (ICIP,2002)
Susstrunk and Winkler (SPIE, 2004)
  R G
 RG 
 
B
 2 
42
Approaches under consideration
• Alpha Rooting (AR) :
Aghaglzadeh and Ersoy (1992), Opt.Engg.
• Multi-Contrast Enhancement (MCE):
Tang, Peli and Acton (2003), IEEE SPL
• Multi-Contrast Enhancement with Dynamic Range
Compression (MCEDRC):
S. Lee (2007), IEEE CSVT
• Contrast Enhancement by Scaling (CES):
Proposed work
• Multi-Scale Retinex (MSR) (a reference spatial
domain technique):
Jobson, Rahman and Woodell (1997), IEEE IP
43
Average Performance Measures
Techniques
JPQM CEF
Y- CbQM QM
CrQM
AR
MCE
8.58
7.00
0.97
0.94
0.80 0.67
0.76 0.67
0.67
0.67
MCEDRC
7.92
0.97
0.86 0.67
0.67
TW-CES-BLK
7.79
1.50
0.90 0.82
0.81
DRC-CESBLK
SF-CES-BLK
8.16
1.18
0.86 0.76
0.76
8.13
1.25
0.89 0.78
0.77
44
Computational Complexities
Techniques
Per Pixel Operations
AR
MCE
MCEDRC
TW-CES
DRC-CES
SF-CES
MSR
1E + 1M
2.19M+1.97A
0.03E+3.97M+2A
0.02E+4.02M+1.05A
0.05E+4M+1.08A
0.03E+4.02M+1.06A
18E+1866378M+8156703A
aE+bM+cA implies a Exponentiation, b Multiplication and c Addition operations.
45
Iterative Enhancement
Iteration no.=1
Iteration no.=3
original
Iteration no.=2
Iteration no.=4
46
Problem of Color Constancy
• Three factors of image formation:
Objects present in the scene.
Spectral Energy of Light Sources.
Spectral Sensitivity of sensors.
Spectral Response of a Sensor
Spectral Power Distribution
Surface Reflectance Spectrum
47
Same Scene Captured under
Different Illumination
Can we transfer colors from one illumination to another one?
48
Computation of Color
Constancy
• Deriving an illumination independent
representation.
- Estimation of SPD of Light Source.
E(λ)
• Color Correction
- Diagonal Correction.
<R, G, B>
49
To perform this computation with DCT coefficients.
Different Spatial Domain
Approaches
• Gray World Assumption (Buchsbaum
(1980), Gershon et al. (1988))
<R, G, B> ≡ <Ravg, Gavg, Bavg>
• White World Assumption (Land (1977))
<R, G, B> ≡ <Rmax, Gmax, Bmax>
50
Select from a set of Canonical
Illuminants
 Observe distribution of points in 2-D
Chromatic Space.
 Assign SPD of the nearest illuminant.
• Gamut Mapping Approach (Forsyth (1990),
Finlayson (1996))
- Existence of chromatic points.
• Color by Correlation (Finlayson et. al. (2001))
- Relative strength over the distribution.
• Nearest Neighbor Approach (Proposed)
- Mean and Covariance Matrix.
- Use of Mahalanobis Distance.
51
Processing in the Compressed
Domain
• Consists of non-overlapping DCT blocks
(of 8 x 8).
• Use DC coefficients of each block.
• The color space used is Y-Cb-Cr instead
of RGB.
• Chromatic Space for Statistical
Techniques is the Cb-Cr space.
52
Different Algorithms under
consideration
53
List of Illuminants
54
Images Captured at Different
Illumination
Source: http://www.cs.sfu.ca/ colour/data.
55
Performance Metrics
• Estimated SPD: E=<RE,GE,BE>
• True SPD:
T= <RT,GT,BT>
56
Average Δθ
30
25
GRW
MXW
GMAP
COR
NN
MXW-Y
20
15
10
5
0
spatial
DCT
62
Average Δrg
0.25
0.2
GRW
MXW
GMAP
COR
NN
MXW-Y
0.15
0.1
0.05
0
spatial
DCT
63
Average ΔRGB
300
250
GRW
MXW
GMAP
COR
NN
MXW-Y
200
150
100
50
0
spatial
DCT
64
Average ΔL
450
400
350
GRW
MXW
GMAP
COR
NN
MXW-Y
300
250
200
150
100
50
0
spatial
DCT
65
Time and Storage Complexities
• nl: number of illuminants.
• nc: size of the 2-D chromaticity space
• n: number of image pixels
• f: Fraction of chromaticity space covered.
• aM+bA  a number of Multiplications and
b number of Additions.
66
Time and Storage Complexities
67
Equivalent No. of Additions
per pixel (1 M= 3 A)
35
30
GRW
MXW
GMAP
COR
NN
MXW-Y
25
20
15
10
5
0
spatial
DCT
n=512, nc=32, nl=12, f=1
68
Color Correction: An Example
Image captured with
(solux-4100)
Target Ref. Image
(syl-50mr16q)
MXW-DCT-Y
COR
COR-DCT
69
Color Restoration
Original
Enhanced w/o
Color Correction
Enhanced with
Color Correction
70
Conclusion-I
• Color-constancy computation in the
compressed domain :
- requires less time and storage.
- comparable quality of results.
• Both NN and NN-DCT perform well
compared to other existing statistical
approaches.
• Color constancy computation is useful in
restoration of colors.
71
Conclusion-II
• Direct filtering in the 8x8 block DCT
space using convolution multiplication
properties.
• Approximate and exact computations
by block DCT composition and
decomposition.
• Demonstration of its applications in
removing blocking artifacts and image
enhancement.
72
Thanks
73