Advisor: Dr. Sreela Sasi
WHEN:
Performed
since the early
20th century
WHAT:
Adding color
to
monochrome
images
HOW:
WHY:
A painstaking
and subjective
manual task
Improve visual
appeal of
illustrations
Introduction
Image Colorization
2
Automation of colorization
Improve visual appeal of images
Color accuracy, finer details
Add relevant information to
images
Make images more
understandable
Introduction (contd.)
Digital Image Colorization
3
Introduction (contd.)
Applications of Image Colorization
4
Colorization Techniques
Scribble-based colorization
+
User add color scribbles
to image to be colorized
=
laborious, timeconsuming, subjective,
and painstaking manual
task.
Example-based colorization
+
=
automation by extracting
colors from sample
image
results can vary
depending on example
image chosen
Previous Research
Image Colorization
5
Image
Image
Sample
Image
New
Grayscale
Image
Texture-based
Segmentation
Texture-based
Segmentation
Feature
Extraction
Feature
Extraction
Color
Descriptors
Texture
Descriptors
Database
Texture
Descriptors
Texture
Matching
Colorization
Process
Current Research
Process Workflow
6
Image segmentation:
•Is the partitioning of an image into
homogeneous regions based on a set of
characteristics.
•Is a key element in image analysis and
computer vision.
Image Segmentation
7
Clustering:
•Is one of the methods available for image
segmentation.
•Is a process which can be used for
classifying pixels based on similarity
according to the pixel’s color or gray-level
intensity.
Image Segmentation (contd.)
8
Despite the substantial amount of research
performed to date, the design of a robust
and efficient clustering algorithm remains a
very challenging problem
Image Segmentation (contd.)
9
Color-based Image Segmentation
Composite Image
10
Color-based Image Segmentation
Composite Image with salt & pepper noise added
11
Texture-based Image Segmentation
12
Original
Image
Gabor Filters
Filtered
Image
Filtered
Image
Filtered
Image
…
Filtered
Image
…
Feature
Image
Energy Computation
Feature
Image
Feature
Image
Feature
Image
Add, mean smoothing, normalization
Feature
Image
Segmentation
Blobs
Workflow Process
Texture-Based Image Segmentation
13
Image Segmentation
Multi-Channel Filtering - Gabor Transform
14
Previous Research (contd.)
Texture-Based Segmentation
15
Image Segmentation
Normalized Sum of Gabor Responses
16
Image
Image
Sample
Image
New
Grayscale
Image
Texture-based
Segmentation
Texture-based
Segmentation
Feature
Extraction
Feature
Extraction
Color
Descriptors
Texture
Descriptors
Database
Texture
Descriptors
Texture
Matching
Colorization
Process
Current Research
Process Workflow
17
Previous Research (contd.)
Clustering and Feature Extraction
18
•The K-means algorithm has been used for
a fast and crisp “hard” segmentation.
•The Fuzzy set theory has improved this
process by allowing the concept of partial
membership, in which an image pixel can
belong to multiple clusters.
•This “soft” clustering allows for a more
precise computation of the cluster
membership, and has been used
successfully for image clustering and
segmentation.
Previous Research
19
•The Fuzzy C-means clustering (FCM)
algorithm [1] is a widely used method for
“soft” image clustering.
•However, the FCM algorithm is
computationally intensive.
•It is also very sensitive to noise because it
only iteratively compares the properties of
each individual pixel to each cluster in the
feature domain.
[1]
James C. Bezdek, Pattern Recognition with Fuzzy Objective Function Algorithms. New York: Plenum, 1981.
Previous Research (contd.)
20
Image Segmentation
Modified Fuzzy C-means Clustering
21
Previous Research (contd.)
Fuzzy C-means clustering (FCM) Algorithm
22
Step 1
Set the number c of clusters, the fuzzy parameter m, and the
stopping condition ε
Step 2
Initialize the fuzzy membership values µ
Step 3
Set the loop counter b = 0
Step 4
Calculate the cluster centroid values using (3)
Step 5
For each pixel, compute the membership values using (4) for each
cluster
Step 6
Compute the objective function A. If the value of A between
consecutive iterations < ε then stop, otherwise set b=b+1
and go to step 4
Previous Research (contd.)
FCM Pseudo-code
23
In order to improve the tolerance to noise of the
Fuzzy C-means clustering algorithm, Krinidis and
Chatzis [2] have proposed a new Robust Fuzzy
Local Information C-means Clustering (FLICM)
algorithm by introducing the novel Gki factor.
The purpose of this algorithm is to adjust the
fuzzy membership of each pixel by adding local
information from the membership of neighboring
pixels.
[2]
Stelios Krinidis and Vassilios Chatzis, "A Robust Fuzzy Local Information C-means Clustering Algorithm," Image
Processing, IEEE Transactions on, pp. 1-1, 2010.
Previous Research (contd.)
Modified Fuzzy C-means clustering with Gki factor
24
The Gki factor is obtained by using a sliding
window of predefined dimensions:
Sliding window of size 1 around the ith pixel
Previous Research (contd.)
Modified Fuzzy C-means clustering with Gki factor
25
The Gki factor is calculated by using the
following equation:
Previous Research (contd.)
Modified Fuzzy C-means clustering with Gki factor
26
This algorithm is further improved by
including both the local spatial information
from neighboring pixels and the spatial
Euclidian distance of each pixel to the
cluster’s center of gravity.
In this research, the algorithm is also
extended for clustering of color images in
the Red-Green-Blue (RGB) color space.
Current Algorithm
Modified Fuzzy C-means clustering with novel Hik factor
27
c1
X
pi
di1
di2
X
c2
c3
di3
X
Illustration of the new
Hik factor displaying the
spatial Euclidian distance
to the center of gravity
of each cluster
Current Algorithm (contd.)
28
Image
Customize
Parameters
Calculate cluster
centroid
Calculate cluster
membership values
Compute Gki
Compute Hki
Readjust membership
values
Compute objective
function
Defuzzification
and clustering
Current Algorithm (contd.)
Process Workflow
29
Current Algorithm (contd.)
Modified Fuzzy C-means Clustering
30
Simulation and Results
Synthetic Grayscale Test Image
31
Natural test image
FCM segmentation
with 5 clusters
FCM segmentation
using the modified FCM algorithm
with 5 clusters, Gki window=1 and Hik
Simulation and Results
Natural Test Image
32
Synthetic 4-color test image
with added salt and pepper noise
FCM clustering
FCM clustering
with Gki window=1 and with Hik
FCM clustering
with Gki window=5 and with Hik
Simulation and Results
Synthetic Grayscale Test Image
33
Synthetic 4-color test image
with added salt and pepper noise
FCM clustering
FCM clustering
with Gki window=1 and with Hik
FCM clustering
with Gki window=5 and with Hik
Simulation and Results
Synthetic Color Test Image
34
Image Segmentation
Clustering Demo
35
•In this research, the FCM with the Gki factor is
modified using the Hik factor, and the algorithm is
extended for the clustering of color images.
•The use of the sliding window in the Gki factor
improves the segmentation results by incorporating
local information about neighboring pixels in the
membership function of the clusters. However, this
resulted in a significant increase in the number of
calculations required for each iteration for each
pixel, and can be given by
Modified Fuzzy C-means Clustering
Summary
36
•By combining the Gki and the Hik factors, this
modified FCM algorithm considerably reduced
the number of iterations needed to achieve
convergence. The tolerance to noise of the
Fuzzy C-means algorithm is also greatly
increased, allowing for an improved capability
to obtain coherent and contiguous segments
from the original image.
Modified Fuzzy C-means Clustering
Summary (contd.)
37
•However, because of the radial nature of the
spatial Euclidean distance to the cluster’s
center of gravity, this new method may not be
as effective for images containing circular
shapes, or for images where the cluster’s
center of gravity are close to each-other.
•In this research, the FCM is extended for the
clustering of color images in the RGB color
space. The effectiveness of this algorithm may
be tested for images in other color spaces
also.
Modified Fuzzy C-means Clustering
Summary (contd.)
38
Image
Image
Sample
Image
New
Grayscale
Image
Texture-based
Segmentation
Texture-based
Segmentation
Feature
Extraction
Feature
Extraction
Color
Descriptors
Texture
Descriptors
Database
Texture
Descriptors
Texture
Matching
Colorization
Process
Current Research
Process Workflow
39
40
Sample Color Images
Image Segmentation
Normalized Sum of Gabor Responses
41
Image Segmentation
Feature Extraction
42
Blob Filtering for color and
texture extraction.
Image Segmentation
Feature Extraction (contd.)
43
Texture and Color database
Image Segmentation
Feature Extraction (contd.)
44
Image
Image
Sample
Image
New
Grayscale
Image
Texture-based
Segmentation
Texture-based
Segmentation
Feature
Extraction
Feature
Extraction
Color
Descriptors
Texture
Descriptors
Database
Texture
Descriptors
Texture
Matching
Colorization
Process
Current Research
Process Workflow
45
Grayscale Image Processing
46
Image
Image
Sample
Image
New
Grayscale
Image
Texture-based
Segmentation
Texture-based
Segmentation
Feature
Extraction
Feature
Extraction
Color
Descriptors
Texture
Descriptors
Database
Texture
Descriptors
Texture
Matching
Colorization
Process
Current Research
Process Workflow
47
• Visual descriptors are descriptions of the visual features of the
contents of images.
• They describe elementary characteristics such as the shape, color,
and texture.
• MPEG-7 is a multimedia content description standard. It was
standardized in ISO/IEC 15938 (Multimedia content description
interface).
• This description is associated with the content itself, to allow fast
and efficient searching for material that is of interest to the user.
• MPEG-7 is formally called Multimedia Content Description
Interface. Thus, it is not a standard which deals with the actual
encoding of moving pictures and audio, like MPEG-1, MPEG-2 and
MPEG-4. It uses XML to store metadata.
Previous Research
Visual descriptors
48
The Img(Rummager) application was developed in the
Automatic Control Systems & Robotics Laboratory at the
Democritus University of Thrace-Greece.
The application can execute an image search based on a
query image, either from XML-based index files, or
directly from a folder containing image files, extracting
the comparison features in real time.
http://chatzichristofis.info/?page_id=213
Previous Research
Visual descriptors
49
Previous Research (contd.)
Content-Based Image Retrieval
50
MPEG-7 EHD
Fuzzy Spatial BTDH
ADS
Previous Research (contd.)
Content-Based Image Retrieval
51
Image Descriptors used:
MPEG-7 Homogeneous Texture Descriptor:
Edge Histogram Descriptor (EHD).
CCD for Medical Radiology Images: Brightness and Texture Directionality
Histogram (BTDH)
Fuzzy rule based scalable composite descriptor (BTDH) is a compact
composite descriptor that can be used for the indexing and retrieval of
radiology medical images. This descriptor uses brightness and texture
characteristics as well as the spatial distribution of these characteristics in
one compact 1D vector. The most important characteristic of the proposed
descriptor is that its size adapts according to the storage capabilities of the
application that is using it. This characteristic renders the descriptor
appropriate for use in large medical (or gray scale) image databases.
Simulation Results (contd.)
Content-Based Image Retrieval (CBIR)
52
MPEG-7 EHD
Query image
Result
Matching
color
Fuzzy Spatial BTDH
Result
Matching
color
ADS
Result
Matching
color
Simulation Results (contd.)
Content-Based Image Retrieval (CBIR)
53
Image
Image
Sample
Image
New
Grayscale
Image
Texture-based
Segmentation
Texture-based
Segmentation
Feature
Extraction
Feature
Extraction
Color
Descriptors
Texture
Descriptors
Database
Texture
Descriptors
Texture
Matching
Colorization
Process
Current Research
Process Workflow
54
The RGB color space is defined by the three chromaticities of
the red, green, and blue additive primaries, and can produce
any chromaticity that is the triangle defined by those primary
colors.
The YCbCr color space is used in video and digital
photography systems.
• Y is the luma (luminance ) component and
• Cb and Cr are the blue-difference and red-difference
chroma components.
Simulation Results (contd.)
Image Colorization
55
Image from Wikipedia
Simulation Results (contd.)
Image Colorization
56
Simulation Results (contd.)
Colorization
57
•New and innovative method
•Automating example-based colorization
•Combines several state-of-the-art techniques
•Reasonably accurate results were obtained
•Several of the steps require custom parameters
•computationally very intensive
•Texture retrieval needs improvement
•Complex textures containing multiple colors
•Anisotropic diffusion for preserving strong edge information
•Combining these techniques in order to automatically
colorize grayscale images is a viable option
Conclusion and Future Work
58
•Images segmentation and clustering methods
computationally very intensive, Processing time for each
600x450 sample color image took 20 minutes on a quad-core
Intel 2.6 GHz processor.
•Texture retrieval methods still need to be improved for scale
and rotation invariance
•Store more complete color descriptors to accommodate
more complex textures containing multiple colors.
•Anisotropic diffusion could also be used to smooth the
Gabor response images while preserving strong edge
information.
•Testing conducted as part of this research proved that the
ability to combine these techniques in order to automatically
colorize grayscale images is a viable option.
Conclusion and Future Work (contd.)
59
[1] Anat Levin, Dani Lischinski, and Yair Weiss, "Colorization using optimization,"
ACM Transactions on Graphics, vol. 23, no. 3, p. 689–694, 2004.
[2] R. Irony, D. Cohen-Or, and D. Lischinski, "Colorization by example," in
Eurographics Symposium on Rendering, 2005, p. 277–280.
[3] Ashikhmin M., Mueller K. Welsh T., "Transferring Color to Greyscale
Images,".
[4] X., Wan L., Qu Y., Wong T., Lin S., Leung C., Heng P. Liu, "Intrinsic
colorization," ACM Trans. Graph., vol. 27, no. 5, p. 152, 2008.
[5] Malik J. Perona P., "Preattentive texture discrimination with early vision
mechanisms," J. Opt. Soc. Am. A, vol. 7, no. 5, May 1990.
[6] A. K. Jain and F. Farrokhnia, "Unsupervised texture segmentation using
Gabor filters," Pattern Recognition, vol. 24, no. 12, pp. 1167-1186, 1991.
[7] Seo Naotoshi, "Texture Segmentation using Gabor Filters," University of
Maryland, College Park, MD, Project ENEE731 , 2006.
[8] Xiaoming Hu, Xinghui Dong, Jiahua Wu, Ping Zou Junyu Dong, "Texture
Segmentation Based on Probabilistic Index Maps," in International
Conference on Education Technology and Computer, 2009, pp. 35-39.
References
60
[9]
[10]
[11]
[12]
[13]
[14]
X Muñoz, J Freixeneta, X Cufı́a, and J Martı́a, "Strategies for image segmentation
combining region and boundary information," Pattern Recognition Letters, vol.
24, no. 1-3, pp. 375-392, January 2003.
James C. Bezdek, Pattern Recognition with Fuzzy Objective Function Algorithms.
New York: Plenum, 1981.
Chuang Keh-Shih, Tzenga Hong-Long, Chen Sharon, Wu Jay, and Chen Tzong-Jer,
"Fuzzy c-means clustering with spatial information for image segmentation,"
Computerized Medical Imaging and Graphics, vol. 30, no. 1, pp. 9-15, January
2006.
Zhou Huiyu, Schaefer Gerald, Sadka Abdul H., and Celebi M. Emre, "Anisotropic
Mean Shift Based Fuzzy C-Means Segmentation of Dermoscopy Images," IEEE
Journal of Selected Topics in Signal Processing, vol. 3, no. 1, pp. 26-34, February
2009.
Stelios Krinidis and Vassilios Chatzis, "A Robust Fuzzy Local Information C-means
Clustering Algorithm," Image Processing, IEEE Transactions on, pp. 1-1, 2010.
Gauge Christophe and Sasi Sreela, "Automated Colorization of Grayscale Images
Using Texture Descriptors and a Modified Fuzzy C-Means Clustering,“ Journal of
Intelligent Learning Systems and Applications (JILSA), Vol. 4 No. 2, 2012, pp. 135143, DOI: 10.4236/jilsa.
References (contd.)
61
Questions?
62