1
2
3
 
Abstract
Increasing amounts of images have made explicit the need for more effective and efficient image indexing and search of not only the metadata associated with an image but also retrieval directly on the content of the image. The scope of the
COBWEB project (Context-Based Image Retrieval on the
WEB - EP28773) is to create a pioneer system that will provide a solution to the problem of automatically storing and retrieving images from large image databases dispersed over the Internet. The project focuses on the following issues:

Development of effective algorithms for the automatic storing of the visual features of the images that can be used as a means for retrieving similar images.

Creation of a simple, cross-platform user interface that helps the user to query the database in the most natural way.

Development of a robust communication subsystem that guarantees satisfying retrieval times.
In this paper, we will describe the architecture and implementation details of the Human Computer Interface
(HCI) and the Intelligent Search Component (ISC), which are two of the main components of the COBWEB system.
The HCI consists of a graphical interface, which enables the user to interact with the system. It provides the user with various approaches in querying the database including keywords specification and sample image submission. The ISC main tasks are to handle and translate the query criteria submitted by the user through the HCI, navigate the database and find images that satisfy those criteria and return the results to the HCI.

1
Epsilon Software SA,
377 Syngrou Ave.
175 64 Palaio Faliro Athens, Greece
Tel: +30-1-9408370, Fax: +30-1-9427719 mike13@otenet.gr
2 National Technical University of Athens,
Dept. of Electrical and Computer Engineering,
Division of Computer Science,
Zografou Campus, GR-15773,
Tel: +30- 1- 7722425, Fax:+30 - 1 - 7722534 gvard@telecom.ntua.gr
3
Senior Software Engineer, IT Consultant,
11B Kodrigktonos Str. Athens, Greece,
Tel: +30 - 1 - 8230165 ctatsio@hol.gr
Index Terms
Content-based image retrieval, image database, keywords, visual features, algorithms, cross-platform, Java, Internet, feature extaction, query, HCI, ISC, FEC
I.
I NTRODUCTION
Increasing amounts of images have made explicit the need for more effective and efficient image indexing and search of not only the metadata associated with an image
(e.g. annotations, captions, keywords) but also retrieval directly on the content of the image (visual features).
The scope of the COBWEB project (Context-Based
Image Retrieval on the WEB - EP28773) is to create a pioneer system that will provide a solution to the problem of automatically storing and retrieving images from large image databases dispersed over the Internet. The project focuses on the following issues:

Development of effective algorithms for the automatic storing of the visual features of the images that can be used as a means for retrieving similar images.

Creation of a simple, cross-platform user interface that helps the user perform queries and retrieve images in the most natural way.

Development of a robust communication subsystem that guarantees satisfying retrieval times.
In this paper, we will describe the architecture and implementation details of the Human Computer Interface
(HCI) and the Intelligent Search Component (ISC), which are two of the main components of the COBWEB system.
II.
O VERALL SYSTEM ARCHITECTURE
Figure 1 shows a simplified diagram of the overall
COBWEB system structure. It presents in a simplified manner the physical architecture of the approach and clarifies how functionality is distributed across the different elements of the system. Furthermore, it shows how those elements communicate with each other.

Output
Environment
COBWEB user COBWEB user
Internet
COBWEB
Administrator
Intelligent Search mechanism
COBWEB
Web Server
image
thumbnails preview images images visual features
COBWEB
Image Database
COBWEB internal system images keywords
Figure 1
COBWEB is implemented as an Internet / Intranetbased application consisting of three logical tiers. The user does not interact directly with the image database but through a user-friendly, graphical interface that communicates with a search engine. The search engine’s intelligence lies in its ability to navigate the image database and find images that can possibly correspond to the user’s demands. The thumbnails of those images (and full-sized previews) are available to the user interface for downloading through the COBWEB’s web server. Thus, the search engine functions as an intermediate tier between the user interface and the image database.
Figure 2 explains the purpose of each functional component of COBWEB in the context of the system’s general architecture. Interactions between the components of the system are also shown.
The Human Computer Interface (HCI) is the user’s view of the COBWEB system. The user submits keywords, sample images or sketches as criteria for his queries. The above criteria are sent directly to the Intelligent Search
Component (ISC) i.e. the search engine. The ISC uses the algorithms provided by the Feature Extraction
Component (FEC) to “translate” those sketches or images into meaningful (for the ISC) feature vectors. Then, the ISC searches into the database ( Database Component -DC ) for images that satisfy the criteria using the algorithms provided by the Feature Comparison Component (FCC) .
Specifically, the ISC compares the keywords and the visual feature vector of the sample image with the respective keywords and feature vectors of the database’s images.
After the interaction with the DC has finished, the ISC returns to the HCI a ranked list of image IDs. Finally, the
HCI fetches those images’ thumbnails (or larger, full-size preview images) from the web server and presents them to the user. thumbnails
return
(through Web
Server)
H uman
C omputer
I nterface results
(image IDs)
4
6
5 thumbnail request
1 query criteria
I ntelligent
S earch
C omponent
3 queries image database
2
F eature
E xtraction
C omponent uses algorithms uses algorithms
F eature
C omparison
C omponent
D atabase C omponent
Figure 2
At the following paragraphs we will present the architecture and implementation details of two of the main components of the COBWEB system, the Human Computer Interface
(HCI) and the Intelligent Search Component (ISC).
III.
H UMAN C OMPUTER I NTERFACE
A.
Overview
The HCI consists of a graphical interface, which enables the user to interact with the database. It is targeted to inexperienced people who do not have extensive knowledge about matters such as computers or image processing. Thus, it provides an intermediate solution between effectiveness and simplicity. At later phases of the software implementation the HCI will enable experienced users to choose to follow more sophisticated and accurate query procedures.
B.
Function / Processing
The HCI provides the user with various approaches in querying the database including: specification of keywords, submission of sample images (possibly loaded from the local file system), rough sketching of the image’s general view (with the use a simple, embedded drawing utility) and a query refinement mechanism, based on the results of the previous retrieval step.
The query procedure consists of the following steps:
1) Submission of query criteria to the ISC – The HCI transmits through the network to the ISC the keywords and the sample image the user has submitted.

2) Retrieval of images and presentation of results to the user - The HCI receives the IDs of the matching images from the ISC. Then, it requests the resulting image thumbnails from the web server and presents them to the user. Obviously, the best matches are presented first.
After that first retrieval step, the user can do one of the following:

Query refinement
The user navigates the results from the previous retrieval step, chooses the images that match best to the image he is searching for and asks for a query refinement.

Alteration of a previously retrieved image
The user selects an image from the results, uses the embedded drawing utility to alter it and make it resemble to the image he is looking for and re-queries using it as a new sample image.
C.
Implementation
A basic requirement for the HCI was to be developped as a cross-platform application. Therefore, it has been implemented as an independent Java application conforming to Sun’s Java 2 standards.
D.
Future work
One of COBWEB’s main targets is to implement the HCI as a downloadable Java applet. For the time being, the above solution is not feasible because of the following reasons:

Currently, browsers do not support the Java 2 platform.

The embedded drawing application makes the code quite lengthy for downloading.

Java 2 execution times are long, especially when dealing with complicated user intefaces.
IV.
I NTELLIGENT S EARCH C OMPONENT
A.
Overview
The Intelligent Search Component (ISC) is the search engine of the COBWEB system and is responsible for the execution of the queries to the image database. The retrieval process is triggered by the query request, submitted through the HCI. Functionally, the ISC serves as an intermediate tier between the HCI client and the database.
ISCs power lies on the use of effective and efficient mahematical algorithms, capable of extracting a number of well-defined features (visual features) from images of arbitrary size. A brief description of the nature of the algorithms integrated in the first prototype of the COBWEB system follows:

Global color histogram

Local color layout

Coarse level color distribution of homogeneous regions

Large scale color texture
Detailed description of the algorithms mentioned above is beyond the scope of this paper. For more information see references 3,4,5 and 6.
B.
Function / Processing
The ISC receives the user’s query criteria from the HCI and navigates the database to retrieve the images that satisfy the user’s needs. A detailed description of the query procedure follows:

Query reception from the HCI
The ISC receives query criteria from the HCI. As mentioned before the query criteria consist of a sample image and a set of keywords.

Extraction of visual features of sample image
The ISC uses the algorithms provided by the Feature
Extraction Component (FEC) to create a feature vector containing visual information about the sample image.

Database navigation
Initially, the ISC searches the database to find a subset of images whose keywords match the ones selected by the user. After the selection has been done, the ISC retrieves from the database the feature vectors of the images of that subset. Then, the feature vector of the sample image is compared with the respective one of each image of that subset. The above goal is accomplished with the help of the similarity algorithms contained in the Feature Comparison
Component (FCC).

Score sorting
The ISC sorts the similarity scores the FCC created for each target image of the subset.

Results aggregation
The ISC selects the images with the best scores and returns the results to the HCI, in the form of image IDs. The maximum number of returned images is subject to the end – user’s preferences.
Figure 3 is a visual representation of the procedure described before:

Figure 3
C.
Implementation
The ISC is implemented with the help of the Microsoft’s
Visual J++ environment. Microsoft’s solution has been adopted instead of Sun’s pure Java solution (Java 2) in order to ensure the communication with the FEC and FCC components, which follow the standards of Microsoft’s
Component Object Model (COM).
D.
Future work
Obviously, feature extraction and feature comparison procedures are time-consuming procedures and the ISC is considered a time-critical application. Moreover, Java is not considered the best platform for the execution of computational tasks. Therefore, it has been decided that the next version of the ISC engine will be implemented with the help of a more powerful environment / language such as
Microsoft’s Visual C++.
[1]
[2]
[3]
V.
R EFERENCES
Bach, J.R. and Fuller, C. and Gupta, A. and Hampapur, A. and
Horowitz, B. and Humphrey, R. and Jian, R. and Shu, C.F., The
Virage Image Search Engine: an Open Framework for Image
Management, Storage and Retrieval for Image and Video Databases
IV, SPIE, pp. 76-87, 1996.
Flickner, M. and Sawhney, H.S. and Niblack, W. and Ashley, J. and Huang, Q. and Dom, B. and Gorkani, M. and Hafner, J. and Lee,
D. and Petkovic, D. and Steele, D. and Yanker, P., Query by Image
Content: the QBIC System, IEEE Computer, vol.28, no. 9, pp. 23-32,
September, 1995.
Strickner, M, and Orengo, M., Similarity of Color Images, SPIE
Proc. Storage and Retrieval for Image and Video Databases III,
February 1995, pp. 381-392
[4]
[5]
[6]
[7]
[8]
[9]
Kovalev, V., and Volmer, S., Color Co-occurance Descriptors for
Querying-by-Example, IEEE Conf. Multimedia Modelling, pp. 32-
38, October 1998.
Volmer, S., Tracing Images in Large Databases by Comparison of
Wavelet Fingerprints, Proc. of the 2nd Int'l Conf. on Visual
Information Systems, pp. 163-172, December, 1997.
Pass, G., and Zabih, R., Histogram Refinement for Content-based
Image Retrieval, IEEE Workshop Applications of Computer Vision,
December 1996
Vardangalos, G., and Vlachos, M. COBWEB Deliverable D1.2:
“Revised Requirements Report”, February 1999
Vardangalos, G., and Vlachos, M. COBWEB Deliverable D1.3:
“Early Prototype Specification”, April 1999
COBWEB Web site http://cobweb.eunet.no/