International Journal of Engineering Trends and Technology (IJETT) – Volume 9 Number 10 - Mar 2014
Sketch4Match: Content Based Image Recognition
using Coloured Sketches
Surekha karpe#1, Sneha Shree#1, Saurabh Kurkute#1, Abuzar Shah#1, Mrs. Archana Chaugule#2
1
2
BE Student, Computer Department, DYPIET, Pune University, Maharashtra, India
Associate Prof., Computer Department, DYPIET, Pune University, Maharashtra, India
Abstract— The CBIR (Content Based Image Retrieval) systems
are one of the rising research areas in the field of Image
Processing. Apart from regular text based search methods used
in many search engines on the internet, CBIR is content based
search system. It extracts important information from the
content which is needed and the image is recognized by the
extracted content itself. The contents extracted can be shape,
colour or any specific texture.
This paper introduces the design and implementation of a
CBIR system , which is based on free hand sketch (Sketch based
Image Recognition-SBIR ). With the help of existing methods,
describe a possible way to process image to reduce its non-usable
information and to detect possible edges in each image, same
edge detection mechanism is applied to user sketch to obtain the
search object which is compared to the Database images by
morphological comparison.
II. OUR PROJECT
In this section the main structure of our system is shown.
Basic functionality of the system, functional units and
relationship among them is shown in figure.
Keywords—CBIR, SBIR, Image Processing, Edge Detection,
Morphology.
I. INTRODUCTION
In the rapidly evolving computer technology, database and
Internet, the most basic part of information searching is
mostly based upon the text search. While there are many
advanced machines to handle queries and storing vast amount
of information, the efficiency of search cannot be same. While
there are methods to search using text based contents, still it is
not efficient because most of the visual information cannot be
expressed textually, so to avoid this situation, the CBIR
technology is used. Using content which are necessary from
the image, the image search is made efficient and dependable,
CBIR allows to extract information which itself represents the
image.
We are using this techniques to build a sketch based image
recognition system which accepts user sketch and retrieves the
resultant images from database in which the user sketch object
occur. The CBIR system are very helpful in many fields e.g.
Crime Investigation, Face recognition, Agriculture etc similar
applications are given in [2][3][4]. The SBIR system is first
introduced in QBIC[5] and VisualSEEK[6] important because
mostly humans remember the visual reference of information
rather textual, also in case of two people don’t understand
each others language, in such cases the SBIR system can be
very helpful. Our project is based on the searching of images
in database using the user sketch.
ISSN: 2231-5381
Fig. 1 – The Basic structure of the system
A. Purpose of our system
Even though the measure of research in sketch-based
image retrieval increases, there is no widely used SBIR
system. Our goal is to develop a content-based associative
search engine based on existing system [1], which databases
are available for anyone looking back to freehand drawing.
The user has a drawing area, where he can draw all shapes and
moments, which are expected to occur in the given location
and with a given size. The retrieval results are grouped by
color for better clarity. Our most important task is to bridge
the information gap between the drawing and the picture,
which is helped by own pre-processing transformation
process. In our system the iteration of the utilization process is
possible, by the current results looking again, thus increasing
the precision.
http://www.ijettjournal.org
Page 525
International Journal of Engineering Trends and Technology (IJETT) – Volume 9 Number 10 - Mar 2014
III. Global Structure of Our System
As shown in Fig 2, the system have four major modules
which are User Sketch Pre-processing, Histogram
Equalization, Morphology Vector Preparation and Retrieving
Images.
Fig. 3.1– Original Image’s Edges
Image’s Edges
Fig. 4.1 – Equalised
As we can see in Fig. 3, 4, 3.1, 4.1 The edges in the
equalised image represents more information than original
image’s edges. The basic shape of the spoon is more visible in
equalised image.
Fig. 2– The Architecture of the system
1. User Sketch Pre-Processing
The user will draw free hand sketch on the given canvas
area the user sketch can be any basic object which he needs to
find in image like a triangle or a circle, user can also select a
colour of the object e.g. if user draws a brown triangle, it can
be a roof of a house or a rig of a bicycle. Next the visual
contents of user sketch is converted to mathematical version.
In this pre-processing, first the edges of the sketch are
detected, for storing the edge information like pixel value,
intensity, location etc, a matrix is prepared using morphology
techniques, and from matrix, a sketch vector is prepared
which is the extracted content from user sketch and which is
compared with the database image vectors. The results can be
clearly seen in Fig 3 and 4.
2. Histogram Equalization
In this section the pre-processing of each database image is
done while the database is being uploaded. Each image in
database is first equalised by spreading the edge pixels,
adjusting color values and then detecting possible edges
within the image. The edge detection results are compared
with normal image and its equalised image that more no of
edges are detected in the equalised image than its original
image.
Fig. 3– Original Image
ISSN: 2231-5381
Fig. 4 – Equalised Image
3. Morphological vector Preparation
In this section, the visual information is converted into
numerical format and stored into a feature vector. The vector
is prepared from the equalised image using algorithm. The
vector contains the pixel position and intensities at each pixel
location. Our project uses ARGB model information i.e. Red,
Green, Blue and Alpha. For each image in database a vector is
stored which represents the edge location and shape in an
image. This feature vector is and the user sketch vector are
compared using Morphological comparison. All feature
vectors are prepared while uploading the image database and
are pre stored in order to improve the search time.
IV. Displaying subsystem
The displaying subsystem is shown in Fig 5. The GUI have a
sketch area known as Canvas, Parameters, Image results and
Image match. User is given a canvas and a colour toolbar
from where he can draw coloured images.
Fig. 5 - The User Interface
The parameters help to rectify the results by adjusting the
position of the object in Image e.g. top, bottom, left or right.
The volume parameter indicates whether there are many such
objects in same image of different colour. The Image Result
area is where the list of images are shown in which the user
object is appearing, and the Image Match shows the exact
match of the user object found in an image. The GUI is easy
to understand and made simple in order to ease the user of
unnecessary adjustments. All the parameters are optional and
http://www.ijettjournal.org
Page 526
International Journal of Engineering Trends and Technology (IJETT) – Volume 9 Number 10 - Mar 2014
are there to improve the result in case of large result set. The
Canvas is of fixed size as variable size canvas lead to false
vector information resulting false results. user is able to load
the database location from DB menu, the result as well as user
sketch can be saved from file menu. The Search mechanism is
made such that user need not click on the Search button or
press enter key, while drawing sketch, as soon as user releases
the left mouse button and finishes sketch, further processing is
initiated. This is done to reduce the time complexity of the
search mechanism.
V. Image Database
There are many image databases are available for image
processing purpose such as flicker 160, Wang Database etc.
These databases are specially used for image processing
fields, psychological studies etc. These databases contain
simple images with decent background so as to focus on main
object. All the images in these databases are of various objects
which have basic simple shape. In our project, we are using
Microsoft Research Cambridge Object Recognition Database.
Some of the database images are shown in the Fig. 6.
VII. Conclusion
We will implement the proposed SBIR system using
methods mentioned above, we will process user sketch by
detecting edges present in user sketch, then convert the visual
edge information to numerical value and store them in a user
sketch vector using morphological vector preparation. For
pre-processing Database, we will equalise each database
image and detect edges in images. and the calculate feature
vector for each database image. both the user sketch vector
and image vectors are compared using morphological
comparison and the results are shown to user. Apart from the
present SBIR system, we will also implement the color
selection technique in user sketch which will further rectify
the results than previous system. The main difference between
the previous system and the present system is that the user
was able to draw sketch in previous system but the results
were anourmous because there is a large possibility that the
user object is definitely present in most of image even its not
clearly visible e.g. small rock. but in our system, the user have
an option to select the color of object which will result in
more precise result of images in which the object is present
with same colour.
References:
[1]
B. Sz ́ant ́o, P. Pozsegovics, Z. V ́amossy, Sz.
Sergy ́an Obuda ́ “Sketch4Match – Content-based Image
Retrieval System Using Sketches” SAMI 2011 • 9th IEEE
International Symposium on Applied Machine Intelligence
and Informatics • January 27-29, 2011 • Smolenice, Slovak
Fig. 6 - Microsoft Research Cambridge Object Recognition
Database
[2]
A.K. Jain, J.E. Lee, and R. Jin, “Sketch to photo
matching: a feature-based approach,” Proc. SPIE,
Biometric Technology for Human Identiification VII, vol.
7667, pp. 766702–766702, 2010.
VI. Testing Aspects, Implementation and Expected results
[3]
A.K. Jain, J.E. Lee, R. Jin, and N. Gregg, “GrafitiID: matching retrieval of grafiti images,” ACM MM,
We can evaluate the effectiveness of the system forming MiFor’09, pp. 1–6, 2009.
methods, and compare the different applied methods, if we
define metrics. Thus, we can determine which method works
[4]
A.K. Jain, J.E. Lee, R. Jin, and N. Gregg, “Content
effectively in what circumstances, and when not. Let be a test
based image retrieval: an application to tattoo images,”
database containing N pieces images, P length retrieval list,
IEEE International Conference on Image Processing, pp.
from which Q pieces matter as relevant results, and Z denotes
2745–2748, November 2009
the number of expected relevant hits. If we know this
information, the following metrics can be calculated.
[5]
M. Flickner, H. Sawhney, W. Niblack, J. Ashley, Q.
Hiang, B. Dom, M. Gorkani, J. Hafner, D. Lee, D. Petkovic,
precision =relevant hits (Q) / all hits (P)
D. Steele, and P. Yanker, “Query by image and video
where the precision gives information about the relative content: the QBIC system,” IEEE Computer, vol. 28, pp.
effectiveness of the system.
23–32, 2002.
recall = relevant hits (Q) / expected hits (Z)
[6]
J.R. Smith, and S.F. Chang, “VisualSEEK: a fully
automated
content-based image query system,” ACM
where the recall gives information about the absolute accuracy
Multimedia ’96, pp. 97–98, 1996.
of the system.
ISSN: 2231-5381
http://www.ijettjournal.org
Page 527