International Journal of Engineering Trends and Technology (IJETT) – Volume 19 Number 4 – Jan 2015 Review of Content Based Image Retrieval Systems Saurav Seth1,Prashant Upadhyay2,Ruchit Shroff3,Rupali Komatwar4 Department of Computer engineering1,2,3,4 Mumbai,Maharashtra,India1,2,3,4 Abstract-Content based image Retrieval (CBIR) has been an active research field since the past two decades. In contrast to a traditional system, in which the images are retrieved based on the keywords, CBIR system retrieves the images based on the visual content. In this paper, we start with the introduction to a simple CBIR system and proceed to review few of the techniques used to develop CBIR system. We also study the visual features used for feature extraction. We study the components of a conventional system. A review of a relevance feedback system and a fuzzy logic system is presented. The query performance of each technique is studied. Keywords— review, content based image retrieval, visual features, relevance feedback, fuzzy logic I. INTRODUCTION We are at the start of the age of digital information. With the ever-increasing access to the Internet, acquiring digital information has become increasingly popular in recent years [1] .The digital libraries and multimedia databases consist of many types of information which include text, audio, image and video. Content-based image retrieval (CBIR) lies at the crossroads of multiple disciplines such as databases, artificial intelligence, image processing, statistics, computer vision, high performance computing, and human – computer intelligent interaction. Several limitations are inherent in a system based on metadata. CBIR, which has a large range of uses for efficient image retrieval, reduces these problems. Text-based information about images can be easily searched using existing technology, but this requires manual description of each image in the database. In several situations, text annotation remains incomplete. It is quite rare for complete text annotation to be available because it would require describing each color, texture, shape, and object within the image. Second difficulty faced with text annotation is that a large amount of labor work is required in manual image annotation for huge image data. Instead of searching for manually annotated text-based keywords, images are retrieved based on their own visual features, such as color, texture and shape; hence researchers turned attention to content based retrieval techniques [2]. user is used to check whether the results are relevant or irrelevant. If the results are irrelevant, the feedback loop is repeated many times until the user is satisfied. A Fuzzy System uses fuzzy logic, which in contrast to crisp logic, is useful to represent uncertainty in complex problems [3]. Image data can be characterized as having a fuzzy nature due to the following [4]: 1. Descriptions of images usually involve inexact and subjective concepts. 2. Imprecision and vagueness are present in descriptions of the images and in some of the visual features. 3. Users‟ needs to retrieve images may be generally fuzzy. Fuzzy logic can minimize semantic gap between high level semantic and low level image features. [5] Fagin [6] and Orlega et al. [7] are the pioneers who successfully integrated fuzzy logic models into CBlR systems. Their proposed algorithms evaluated the fuzzy query, and also showed the effectiveness through experimental results. Medasani and Krishnapuram [8] proposed a fuzzy linguistic query in their CBlR system. The paper is organized as follows: the first section provides a brief introduction. The second section describes a general system architecture and its components followed by a review of various visual features in the third section. We review different techniques in section 4 including conventional system and its disadvantages, a relevance feedback system and a fuzzy logic system Conclusion and Future research directions are presented in the last section. Relevance feedback is a powerful technique in CBIR systems, which can improve the performance of CBIR effectively. It provides open research area to the researcher to reduce the semantic gap between low-level features and high level concepts. The basic concept behind this is that after obtaining the retrieval results, the feedback provided by the ISSN: 2231-5381 http://www.ijettjournal.org Page 178 International Journal of Engineering Trends and Technology (IJETT) – Volume 19 Number 4 – Jan 2015 INPUT IMAGE IMAGE DATABASE QUERY IMAGE FEATURE EXTRACTION SIMILARITY MATCHING FEATURE EXTRACTION IMAGE FEATURE DATABASE RETRIEVED IMAGE(S) Fig.1Model of Content based image retrieval III. II. VISUAL FEATURES CBIR COMPONENTS A. Query image This is the image inputted by the user. This image undergoes feature extraction. Finally similarity matching is used to retrieve similar images from the feature database. B. Image database This consists of all the images present in the database. Each image is subjected to the feature extraction process. This information is then stored in a feature database. C. Similarity Matching Matching images directly, that is comparing the values of the pixels of the image directly is quite often used in object recognition. Different methods have been proposed to do this and a selection of these methods is presented here and can be used in the image retrieval system. Euclidean Distance Probably the most common approach to compare images directly is the Euclidean distance. Euclidean distance is a geometrical concept which takes into consideration the co-ordinate values of the pixel points between which the distance is to be found [9]. To be able to compare images using a Euclidean distance, the images have to be of the same size which can be achieved easily with scaling algorithms. The Euclidean distance has been used successfully e.g. in optical character recognition and has been extended by different methods. D. Feature extraction Feature extraction is the process of computing numerical or alphanumerical representation of some attribute of digital images to derive the image contents. A feature is directly related to the visual characteristics of the image. ISSN: 2231-5381 A. color Color feature in content based image retrieval uses various color spaces such as RGB, XYZ, YIQ, L*a*b*, U*V*W*, YUV and HSV. The HSV color space gives the best color histogram feature, among the different color spaces [10]-[13]. HSV color space the color is presented in terms of three components: Hue (H), Saturation (S) and Value (V) and the HSV color space is based on cylinder coordinates. L*a*b* color space, L* stands for luminance, a* represents relative greenness-redness and b* represents relative bluenessyellowness It achieves device independence [14]. B. Texture Texture is an essential feature of an image when querying image databases. It depends on human visual perception. The two most commonly used features are Tamura and Gabor. In Tamura the authors propose six texture features corresponding to human visual perception: coarseness, contrast, directionality, line-likeness, regularity, and roughness [15]. They make experiments to test the significance of the feature and found the first three features to be very important. Gabor filters are a well known technique for texture analysis which was used for different works earlier. In this work we use the approach where the HSV color space (hue, saturation, value) is used. It has been proposed that Gabor filters can be used to model the responses of the human visual system [15]-[17]. C. Shape Shape from an image is quite a powerful representation as it characterizes the geometry of the object. The representation of a shape should be invariant to scale, translation and rotation. http://www.ijettjournal.org Page 179 International Journal of Engineering Trends and Technology (IJETT) – Volume 19 Number 4 – Jan 2015 The shape feature can be divided into two categories i.e. Contour based and regions based [19]. Region based includes simple geometric attributes that can be obtained by measuring properties of points belonging to the region. The properties includes area, aspect ratio etc. Typically boundary-based representations include two major steps. First, a 1D function is constructed from a 2D shape boundary parametrizing the contour. Then the constructed 1D function is used to extract a feature vector describing the shape of the object. The contour based representation has descriptors such as Fourier descriptors and CSS (Curvature Scale Space) descriptors [18][19]. IV. TECHNIQUES A. Conventional system Basically Content based image retrieval technique is a method to retrieve images that matches to the given specifications of the query image. In CBIR systems, the images stored in the database are labeled by feature vectors, which are extracted from the images by means of computer vision and digital image processing techniques. These feature vectors are obtained by the feature extraction process. In CBIR systems, the query to a database is specified by an image. The query‟s feature vector is computed and the closest items in the database, according to a similarity metric or distance defined in feature space, are returned as the answers to the query. A CBIR system is a query resolving system over image collections that use the information inherently contained in the image. The CBIR system has to be able to extract quantitative features from the images that allow the system to index the image collection and to compute a distance between images. The user interacts with the system by a querying interface, usually a web page, where the query is defined and sent to the CBIR engine. In the process the query is represented by an image provided by the users, asking the CBIR system for a list of the most similar images in the database. To resolve the query, the CBIR engine computes the image features which correspond to a point in the metric space defined by the system. Each image in database has a representative in this metric space so a distance to the query image could be computed for each image, using a similarity (or dissimilarity) function. This produces a list ordered by similarity (or dissimilarity) to the query image, which is presented to the user as the response [20]. feedback to the system, so that the system can perform well in order to reply to the original query according to the desired output. For the retrieval of an image from the database, the very first step is we extract feature vectors from images. The features can be like shape, color, texture etc. These features are stored in another database for future use. After the query image is given by the user, the features are extracted and we match those features with the one included in database image features. And if the distance between these two images is found to be small enough at an acceptable level; we consider the corresponding image in the database similar to the query. The results are based on similarity matching rather on matching images. Then user attains the opportunity to give the feedback in the form of his/her judgments expressed over the retrieval results. The relevance judgments compute the results depending on a three values. The three values are relevant, non-relevant and don‟t care. Relevant means the similar image desired to the user, non-relevant means the image is definitely not matching, and don‟t cares mean the user does not care and says anything about the image. The feedback loop stops as soon as the feedback provided by user falls under the „relevant‟ category otherwise it continues until user gets satisfied with results. One of the techniques of relevance feedback is the Bayesian framework. A Bayesian network is a representation of random variables graphically, providing an effective knowledge representation. By formulating the problems through the Bayesian belief network, we found that for our relevant image adoption problem, the Bayesian network has advantages. The Bayesian network has 3 layers, the query layer, feature index layer and relevant image layer. Query layer is the root node which represents the query example provided by the user. The feature index layer is further divided into the low level feature representations i.e. color, texture and shape, another level consists of the components of feature vectors. The third layer consists of relevant images specified by user [21]. Initial CBIR systems focused only on the visual features; however, after the popularity of these systems, the need for user-friendly interfaces became a necessity. Therefore, the CBIR field started to include efficient designs that were easy to understand and that tried to meet the needs of the user performing the search. Systems that allowed descriptive semantics in their query methods were required. In addition, systems that provided user feedback and systems that included machine learning, which may understand user satisfaction levels were required. These features have their further classifications as global features and local features. The commonly used features are color, texture, and shape. All these features are application independent [21]-[23]. B. Relevance feedback C. Fuzzy system Basically the idea of relevance feedback is to shift the load of finding the right query formulation from the user to the system. In order for correctness, the user has to provide some The fuzzy system can be combined with several other models to improve its performance. It can be combined with textual descriptions as well as with relevance feedback. ISSN: 2231-5381 Fig2. Relevance feedback system http://www.ijettjournal.org Page 180 International Journal of Engineering Trends and Technology (IJETT) – Volume 19 Number 4 – Jan 2015 1) Image retrieval using texture features: Tamura features are used to extract the texture feature of an image. These low level features are assigned fuzzy linguistic terms. For e.g.: Coarseness – very fine, fine, medium coarse, coarse, very coarse. These linguistic terms lie within the discourse [0, 1] [5]. The various semantic and syntactic rules [5] are used to obtain membership function. These are created for every linguistic term from (left to right).A function for the query is constructed which is then used for similarity matching between the query and the images in the database. This function can be defined by another fuzzy set [5]. The above system can handle queries with textual descriptions. For e.g. a query of the form “very directional ˄ very blob-like ˄ very regular” [5] would retrieve images whose linguistic terms correspond to the query‟s description. 2) Image retrieval using color feature: The L*a*b* color space can be used for feature extraction. The color space is split into triplets of L*, a*, b*. L* is given a low weightage as it does not provide any unique color. As in the texture model, each of the triplets is assigned a linguistic variable. For e.g.: a*- green, greenish, middle, reddish, red [24]. A FIS (fuzzy inference system), consisting of a rule base is used to obtain a fuzzy color histogram (FCH) [25]. This FCH is compared with all the FCH‟s of images in the database using a similarity function. The similarity function called min-max ratio is used to perform the comparison [24]. Such a system performs better as compared to a conventional system. The above system is more effective than the conventional system. For a bus image the mean of the similarity between images is 87% in contrast to 46% for a conventional system [24]. A clustering algorithm can be used to classify the images and create more appropriate linguistic variables. V. CONCLUSION We have presented a brief review of the techniques that are used in content-based image retrieval. A conventional system only captures the low level features. The use of relevance feedback (RF) provides the CBIR system with the necessary feedback which is used to improve query performance. A fuzzy system performs better than a conventional system by reducing the semantic gap. The fuzzy system assigns linguistic variables to the low level features; this helps the system to reduce the semantic gap. For future work, a fuzzy relevance feedback using both color and texture feature can be studied. [7] M. Ortega, Y. Rui, K. Cbakrabarti, K. Porkaew, S. Mehrotra, and T. S. Huang, “Supporting ranked Boolean similarity queries in MARS,” IEEE Transactions on Knowledge and Data Engineering, Vol. IO, No. 6, pp. 905925,1998. [8] S. Medasani and R. Krishnapuram, “A fuzzy approach to complex linguistic query based image retrieval,” IEEE International Conference on Fuzzy Systems, pp. 590-595, Seoul-Korea, Aug., 1999.Colour paper [9] John Eakins Margaret Graham University of Northumbria at Newcastle ,“Content-based Image Retrieval”, Report: 39 JISC Technology Applications Programme Joint Information Systems Committee October 1999. [10] X. Wan and C. C. Kuo, “Color distribution analysis and quantization for image retrieval”, In SPIEStorage and Retrieval for Image and Video Databases IV, Vol. SPIE 2670, pp- 9–16. 1996. [11] M. W. Ying and Z. HongJiang, “Benchmarking of image feature for content-based retrieval”, IEEE.Pp-253-257, 1998. [12] Z. Zhenhua, L. Wenhui and L. Bo, “An Improving Technique of Color Histogram in SegmentationbasedImage Retrieval”, 2009 Fifth International Conference on Information Assurance and Security,IEEE, pp-381-384, 2009. Signal & Image Processing : An International Journal (SIPIJ) Vol.3, No.1, February 201256 [13] E. Mathias, “Comparing the influence of color spaces and metrics in content-based image retrieval”,IEEE, pp- 371-378, 1998. [14] V. Castelli, and L. Bergman, Image Databases: Search and Retrieval of Digital Imagery, Wiley-Interscience, USA, 2002. [15] Manjunath, B., Ma, W.: Texture features for browsing and retrieval of image data. IEEE Trans on Pattern Analysis and Machine Intelligence 18 (1996) 837–842. [16] Tamura, H., Mori, S., Yamawaki, T.: Textural features corresponding to visual perception. IEEE Trans on Systems, Man and Cybernetics 8 (1978) 460–472 [17] Manjunath, B., Wu, P., Newsam, S., Shin, H.: A texture descriptor for browsing and similarity retrieval. Journal of Signal Processing: Image Communication 16 (2000) 33–43 [18] A. Del Bimbo, Visual Information Retrieval, 270 pages, Morgan Kaufmann Publishers, San Francisco, California, 1999. [19] M. Trimeche, “Shape Representations for Image Indexing and Retrieval”, Master of Science Thesis, Tampere University of Technology, May 2000. [20] A. W. M. Smeulders, M. Worring, S. Santini, A. Gupta, R. Jain; ContentBased Image Retrieval at the End of the Early Years; IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 22, No 12, Pg. 1349-1380, December 2000. [21] Jensen, F.V., Lauritzen, S.L. and Olesen, K.G. (1990): Bayesian updating in causal probabilistic networks by local computations. Computational Statistics Quarterly 4:269-282. [22] Q. Tian , N. Sebe, M.S. Lew , E. Loupias, and T. S. Huang, “Image Retrieval Using Wavelet-Based Salient Points,” J. Electronic Imaging, special issue on Storage and Retrieval of Digital Media, vol. 10, no.4 pp. 849935, 2001. [23] J. Wang, H. Zha, and R. Cipolla, “Combining Interest Points and Edges for Content-based Image Retrieval” in Proc. IEEE Int‟l Conf. Image Processing (ICIP‟ 05), pp. 1256-1259, 2005. [24] Aqeel M. Humadi, Hameed A. Younis “Application of the Fuzzy Logic in Content Based Image Retrieval using Color Feature” IJCSMC, Vol. 3, Issue. 2, February 2014, pg.170 – 180. [25] Han, J., & Ma, K. K. (2002). Fuzzy color histogram and its use in color image retrieval. IEEE Transactions on Image Processing, 11(8), 944–952. doi:10.1109/TIP.2002.801585. REFERENCES [1] Jia Li, James Z. Wang, Gio Wiederhold IRM: Integrated Region Matching for Image Retrieval, multimedia ’00 proceedings of the 8th ACM international conference on multimedia pg. 147-156 [2] kokare manesh, chatterji b. n., biswas p. k, A Survey on Current Content Based Image Retrieval Methods, IETE journal of research, 2002, vol. 48, pg. 261-271. [3]Ponce-Cruz, Pedro, Ramírez-Figueroa, Fernando D., intelligent control systems with labVIEW, 2010,XII. [4] L. Yan, and Z. Ma, Intelligent Multimedia Databases and Information Retrieval: Advancing Applications and Technologies, September - 2011. [5] Chih-Yi Chiu Hsin-Chih Lin‟ Shi-Nine Yam “A fuzzy logic CBIR System” The 12th IEEE International Conference on Fuzzy Systems, 2003 (Volume: 2) pg. 1171-1176. [6] R. Fagin, “Combining fuzzy information from multiple systems,” Journal of Computer and System Sciences, Vol. 58, No. 1, pp. 83-99, 1999. ISSN: 2231-5381 http://www.ijettjournal.org Page 181