View

advertisement

Effective ways for querying images by content over the Internet

M. Vlachos

1

, G. Vardangalos

2

, C. Tatsiopoulos

3

 

Abstract

Increasing amounts of images have made explicit the need for more effective and efficient image indexing and search of not only the metadata associated with an image but also retrieval directly on the content of the image. The scope of the

COBWEB project (Context-Based Image Retrieval on the

WEB - EP28773) is to create a pioneer system that will provide a solution to the problem of automatically storing and retrieving images from large image databases dispersed over the Internet. The project focuses on the following issues:

Development of effective algorithms for the automatic storing of the visual features of the images that can be

used as a means for retrieving similar images.

Creation of a simple, cross-platform user interface that helps the user to query the database in the most natural way.

Development of a robust communication subsystem that guarantees satisfying retrieval times.

In this paper, we will describe the architecture and implementation details of the Human Computer Interface

(HCI) and the Intelligent Search Component (ISC), which are two of the main components of the COBWEB system.

The HCI consists of a graphical interface, which enables the user to interact with the system. It provides the user with various approaches in querying the database including keywords specification and sample image submission. The ISC main tasks are to handle and translate the query criteria submitted by the user through the HCI, navigate the database and find images that satisfy those criteria and return the results to the HCI.

1

Epsilon Software SA,

377 Syngrou Ave.

175 64 Palaio Faliro Athens, Greece

Tel: +30-1-9408370, Fax: +30-1-9427719 mike13@otenet.gr

2

National Technical University of Athens,

Dept. of Electrical and Computer Engineering,

Division of Computer Science,

Zografou Campus, GR-15773,

Tel: +30- 1- 7722425, Fax:+30 - 1 - 7722534 gvard@telecom.ntua.gr

3

Senior Software Engineer, IT Consultant,

11B Kodrigktonos Str. Athens, Greece,

Tel: +30 - 1 - 8230165 ctatsio@hol.gr

Index Terms

Content-based image retrieval, image database, keywords, visual features, algorithms, cross-platform, Java, Internet, feature extaction, query, HCI, ISC, FEC

I.

I

NTRODUCTION

Increasing amounts of images have made explicit the need for more effective and efficient image indexing and search of not only the metadata associated with an image

(e.g. annotations, captions, keywords) but also retrieval directly on the content of the image (visual features).

The scope of the COBWEB project (Context-Based

Image Retrieval on the WEB - EP28773) is to create a pioneer system that will provide a solution to the problem of automatically storing and retrieving images from large image databases dispersed over the Internet. The project focuses on the following issues:

Development of effective algorithms for the automatic storing of the visual features of the images that can be

 used as a means for retrieving similar images.

Creation of a simple, cross-platform user interface that helps the user perform queries and retrieve images in the most natural way.

Development of a robust communication subsystem that guarantees satisfying retrieval times.

In this paper, we will describe the architecture and implementation details of the Human Computer Interface

(HCI) and the Intelligent Search Component (ISC), which are two of the main components of the COBWEB system.

II.

O

VERALL SYSTEM ARCHITECTURE

Figure 1 shows a simplified diagram of the overall

COBWEB system structure. It presents in a simplified manner the physical architecture of the approach and clarifies how functionality is distributed across the different elements of the system. Furthermore, it shows how those elements communicate with each other.

Output

Environment

COBWEB user COBWEB user

Internet

COBWEB

Administrator

Intelligent Search mechanism

COBWEB

Web Server

image

thumbnails preview images images visual features

COBWEB

Image Database

COBWEB internal system images keywords

Figure 1

COBWEB is implemented as an Internet / Intranetbased application consisting of three logical tiers. The user does not interact directly with the image database but through a user-friendly, graphical interface that communicates with a search engine. The search engine’s intelligence lies in its ability to navigate the image database and find images that can possibly correspond to the user’s demands. The thumbnails of those images (and full-sized previews) are available to the user interface for downloading through the COBWEB’s web server. Thus, the search engine functions as an intermediate tier between the user interface and the image database.

Figure 2 explains the purpose of each functional component of COBWEB in the context of the system’s general architecture. Interactions between the components of the system are also shown.

The

Human Computer Interface (HCI)

is the user’s view of the COBWEB system. The user submits keywords, sample images or sketches as criteria for his queries. The above criteria are sent directly to the

Intelligent Search

Component (ISC)

i.e. the search engine. The ISC uses the algorithms provided by the

Feature Extraction

Component (FEC)

to “translate” those sketches or images into meaningful (for the ISC) feature vectors. Then, the ISC searches into the database (

Database Component -DC

) for images that satisfy the criteria using the algorithms provided by the

Feature Comparison Component (FCC)

.

Specifically, the ISC compares the keywords and the visual feature vector of the sample image with the respective keywords and feature vectors of the database’s images.

After the interaction with the DC has finished, the ISC returns to the HCI a ranked list of image IDs. Finally, the

HCI fetches those images’ thumbnails (or larger, full-size preview images) from the web server and presents them to the user.

thumbnails

return

(through Web

Server)

H

uman

C

omputer

I

nterface

results

(image IDs)

4

6

5

thumbnail request

1

query criteria

I

ntelligent

S

earch

C

omponent

3

queries image database

2

F

eature

E

xtraction

C

omponent

uses algorithms uses algorithms

F

eature

C

omparison

C

omponent

D

atabase

C

omponent

Figure 2

At the following paragraphs we will present the architecture and implementation details of two of the main components of the COBWEB system, the Human Computer Interface

(HCI) and the Intelligent Search Component (ISC).

III.

H

UMAN

C

OMPUTER

I

NTERFACE

A.

Overview

The HCI consists of a graphical interface, which enables the user to interact with the database. It is targeted to inexperienced people who do not have extensive knowledge about matters such as computers or image processing. Thus, it provides an intermediate solution between effectiveness and simplicity. At later phases of the software implementation the HCI will enable experienced users to choose to follow more sophisticated and accurate query procedures.

B.

Function / Processing

The HCI provides the user with various approaches in querying the database including: specification of keywords, submission of sample images (possibly loaded from the local file system), rough sketching of the image’s general view (with the use a simple, embedded drawing utility) and a query refinement mechanism, based on the results of the previous retrieval step.

The query procedure consists of the following steps:

1)

Submission of query criteria to the ISC

– The HCI transmits through the network to the ISC the keywords and the sample image the user has submitted.

2)

Retrieval of images and presentation of results to the user

- The HCI receives the IDs of the matching images from the ISC. Then, it requests the resulting image thumbnails from the web server and presents them to the user. Obviously, the best matches are presented first.

After that first retrieval step, the user can do one of the following:

Query refinement

The user navigates the results from the previous retrieval step, chooses the images that match best to the image he is searching for and asks for a query refinement.

Alteration of a previously retrieved image

The user selects an image from the results, uses the embedded drawing utility to alter it and make it resemble to the image he is looking for and re-queries using it as a new sample image.

C.

Implementation

A basic requirement for the HCI was to be developped as a cross-platform application. Therefore, it has been implemented as an independent conforming to Sun’s Java 2 standards.

Java application

D.

Future work

One of COBWEB’s main targets is to implement the HCI as a downloadable Java applet. For the time being, the above solution is not feasible because of the following reasons:

Currently, browsers do not support the Java 2 platform.

The embedded drawing application makes the code quite lengthy for downloading.

Java 2 execution times are long, especially when dealing with complicated user intefaces.

IV.

I

NTELLIGENT

S

EARCH

C

OMPONENT

A.

Overview

The Intelligent Search Component (ISC) is the search engine of the COBWEB system and is responsible for the execution of the queries to the image database. The retrieval process is triggered by the query request, submitted through the HCI. Functionally, the ISC serves as an intermediate tier between the HCI client and the database.

ISCs power lies on the use of effective and efficient mahematical algorithms, capable of extracting a number of well-defined features (visual features) from images of arbitrary size. A brief description of the nature of the algorithms integrated in the first prototype of the COBWEB system follows:

Global color histogram

Local color layout

Coarse level color distribution of homogeneous regions

Large scale color texture

Detailed description of the algorithms mentioned above is beyond the scope of this paper. For more information see references 3,4,5 and 6.

B.

Function / Processing

The ISC receives the user’s query criteria from the HCI and navigates the database to retrieve the images that satisfy the user’s needs. A detailed description of the query procedure follows:

Query reception from the HCI

The ISC receives query criteria from the HCI. As mentioned before the query criteria consist of a sample image and a set of keywords.

Extraction of visual features of sample image

The ISC uses the algorithms provided by the Feature

Extraction Component (FEC) to create a feature vector containing visual information about the sample image.

Database navigation

Initially, the ISC searches the database to find a subset of images whose keywords match the ones selected by the user. After the selection has been done, the ISC retrieves from the database the feature vectors of the images of that subset. Then, the feature vector of the sample image is compared with the respective one of each image of that subset. The above goal is accomplished with the help of the similarity algorithms contained in the Feature Comparison

Component (FCC).

Score sorting

The ISC sorts the similarity scores the FCC created for each target image of the subset.

Results aggregation

The ISC selects the images with the best scores and returns the results to the HCI, in the form of image IDs. The maximum number of returned images is subject to the end – user’s preferences.

Figure 3 is a visual representation of the procedure described before:

Figure 3

C.

Implementation

The ISC is implemented with the help of the Microsoft’s

Visual J++ environment. Microsoft’s solution has been adopted instead of Sun’s pure Java solution (Java 2) in order to ensure the communication with the FEC and FCC components, which follow the standards of Microsoft’s

Component Object Model (COM).

D.

Future work

Obviously, feature extraction and feature comparison procedures are time-consuming procedures and the ISC is considered a time-critical application. Moreover, Java is not considered the best platform for the execution of computational tasks. Therefore, it has been decided that the next version of the ISC engine will be implemented with the help of a more powerful environment / language such as

Microsoft’s Visual C++.

[1]

[2]

[3]

V.

R

EFERENCES

Bach, J.R. and Fuller, C. and Gupta, A. and Hampapur, A. and

Horowitz, B. and Humphrey, R. and Jian, R. and Shu, C.F., The

Virage Image Search Engine: an Open Framework for Image

Management, Storage and Retrieval for Image and Video Databases

IV, SPIE, pp. 76-87, 1996.

Flickner, M. and Sawhney, H.S. and Niblack, W. and Ashley, J. and Huang, Q. and Dom, B. and Gorkani, M. and Hafner, J. and Lee,

D. and Petkovic, D. and Steele, D. and Yanker, P., Query by Image

Content: the QBIC System, IEEE Computer, vol.28, no. 9, pp. 23-32,

September, 1995.

Strickner, M, and Orengo, M., Similarity of Color Images, SPIE

Proc. Storage and Retrieval for Image and Video Databases III,

February 1995, pp. 381-392

[4]

[5]

[6]

[7]

[8]

[9]

Kovalev, V., and Volmer, S., Color Co-occurance Descriptors for

Querying-by-Example, IEEE Conf. Multimedia Modelling, pp. 32-

38, October 1998.

Volmer, S., Tracing Images in Large Databases by Comparison of

Wavelet Fingerprints, Proc. of the 2nd Int'l Conf. on Visual

Information Systems, pp. 163-172, December, 1997.

Pass, G., and Zabih, R., Histogram Refinement for Content-based

Image Retrieval, IEEE Workshop Applications of Computer Vision,

December 1996

Vardangalos, G., and Vlachos, M. COBWEB Deliverable D1.2:

“Revised Requirements Report”, February 1999

Vardangalos, G., and Vlachos, M. COBWEB Deliverable D1.3:

“Early Prototype Specification”, April 1999

COBWEB Web site http://cobweb.eunet.no/

Download