Introduction to Computational and Biological Vision
The program for Software-Engineering
Department of Computer Science
Ben-Gurion University of the Negev
Content-Based Image Retrieval
(CBIR)
By:
Victor Makarenkov
Michael Marcovich
Noam Shemesh
Introduction : Ideas and Goals
Content-based image retrieval, a technique which uses visual contents to search
images from large scale image databases according to users' interests, has been an
active and fast advancing research area since the 1990s. In our project we
concentrated on region-histogram features to retrieve the images according to an
example query image supplied by the user.
Let's explain the "region-histogram features".
Histogram is a measure used to describe the image. In simple words it means the
distribution of color brightness across the image. The brightness values range in
[0..255]. Region based means that the histogram measure is not taken globally for the
whole image, but locally for different image regions. This region-histogram features
were used as index of the image database.
We implemented a small CBIR system. Our goals were to check how good\bad can
the histogram measure serve us to that task. Probably the histogram won't work well
in general cases. We want to check which image classifications are better for that. We
want to see what improvements can be done on basic histogram comparison to
increase results accuracy.
General CBIR system works according to the following schema:
In our CBIR system we implemented all the parts except the one of relevance
feedback.
Course of Action
We elaborate the components of the above schema, as they are in our system.

Query formation by the user: is done using the method of example query.
Wherever the user wants to find the image he supplies the image, and the
system should return top 5 similar pictures.

Visual content description: since we using histogram of image, we transform
the file of the image to its bitmap representation. That means 2D array where
each cell contains a triple with the RGB brightness values for the colors
Red,Green,Blue.

Feature vectors: In our system, for generality purposes we assume that the
images are of fixed size 200*200 pixels. (If not our system converts them to
that size). We use local histogram values. The image is divided into N * N
square areas, and then the histogram computed in each area.Each area is of
size (200/N)*(200/N) pixels .Each image is represented with N*N length
vector where each coordinate is the histogram in the appropriate area. More
precisely: I  u1 ,..., u N *N  and ui  v0 ,...,v255  .

Similarity comparison: for a similarity comparison we used the Minkowski
distance. Minkowski distance between 2 images I and J is denoted as:
p
( F (i )  F ( j )) p while we started our research when p=2. (Euclidian
distance). Using this comparison we measured the distance for each two
images in two levels:
o Level 1: measure the Minkowski distance between each 2
corresponding histogram regions of the images. To compute the
distance in area level – to find each F(i)-F(j).
o Level 2: Summing all the powers of distances from level 1 to form a
Minkowski distance at image level with a taking a root of this
summation.

Indexing and retrieval: for all images that are in the databases the feature
vector is pre-computed and stored as index in file. When retrieval should be
made, the image with the least Minkowski (most similar images) distance
between query image and image from database is returned.
Implementation
We implemented our CBIR system in C# using Windows-Forms. The code consists of
4 classes with logic and one GUI class with the main system screen. The database is a
list of images stored under folder \database\ and the index (which is pre-computed) is
stored as a binary serialization of the feature vectors into a file. The index file is
called index.dat. Each time the system is loaded, the index is de-serialized into the
main memory, and whenever the search takes place it is ready for comparisons.
Each time the system shuts down – the index (with potentially new entries) is
serialized back into the file.
The system presents the top 5 results for the image query. Thus allowing the user to
see some possible suggestions.
At the first time the system is installed the user can change the N parameter described
above – in order to adjust the division of areas the image is divided to.
The program GUI looks like:
The user can choose the P parameter of minkowski mean and the mode of search in
the histogram areas : Regular or weighted (the closer to the center of image – the
higher similarity measure – the less the distance between images).
Experiments and results:
We experimented with our system using the following techniques:

Simple (Global) Histogram.

Complex (Areas based) Histogram.

Weighted Areas.
Our first assumption was that the results will improve as we going through these
stages. For example we thought that division of image into areas, and using complex
histogram measure will improve dramatically the results. The second assumption was
that weighted areas, i.e., giving more similarity weight to more central parts of images
that are alike. We thought it will give better results for example for passport
photographs – when the main part of the image located in its center.
Experiment process:
At the first stage, to check the system we used some "trivial" images to check if we
coded the system without bugs. We used images like: blue square at the center of
white one, or red circle at the top right corner. In this way we checked that area
weighting is working in general. Then we proceeded to the next stage.
We took sets of triples or pairs of similar (took at same place and similar illumination
and set of colors) pictures. Our goal was to show that by giving query image it will
return similar ones took in analogous way. For example a trip photo (colorful image),
party photo (dark illumination), finishing officer's course image (all the soldiers wear
same uniform).
We expect that once the image exists at the database – it should be returned first top
ranked.
Our Database is :
Results:
1. Global Histogram Measure :
For a dark image from pub we get :
Since the query image is in the database – the first result we get is the image itself.
Then we get similar picture (not the same!!) and then the yellow bear on the dark
background . On next results we will omit the same picture as a result for showing.
From a bright image we get immediately after the picture itself the following results:
The second result has little in common with the content , BUT, approximately half of
the picture has very bright illumination, so in terms of global histogram they are close.
The last example on a global histogram :
As wee see the second result after the image itself, remains with the same color
distribution and is also similar in shape. The second is less. The third is very similar,
except the part of the bright sky which is bigger than in the query image.
2. Local histogram measure:
In this experiment we divide the image into area, and then compute the similarity
between the corresponding areas, instead of global histogram comparison.
Here is illustration of image's areas:
In our Experiment with single division we make the division to 4 areas.
The immediate improvement can be seen in Barcelona pictures: The image clibs up
one place in the rank!
From the western wall picture : The second result is from the same place (after the
original image).
A similar background result:
And another result: approximately half of the image have similar colors:
The examples of party and military ceremony did not give any improvements.
3. Weighted areas: it is obvious that weighting is applicable only on divided areas of
pictures. We tested it on 4*4 divison. The more close the area to the center the more
weight on similarity measure it gets.
Here is the example for a nature picture :
Without the weighting, we got the same results on 4*4 image division.
Conclusions:
As we thought at the beginning – Histogram is quite primitive and insufficient way
for CBIR purposes. However, with certain image characteristics it may be useful, and
works well. For example on the military ceremony and the nature images.
Another important foundation we made is that one of our initial assumptions was
wrong. It is that dividing the image into many area , does not always improve the
results of retrieval. In case of too many divisions, it degrades the results. The reason
for that is that while comparing small parts, that are corresponding between the
images and are at fixed place, they can be different. But if the same picture can be
shifted, and not be found! The method is not shift invariant!
In some cases , small division (4 areas) did help. For example on the image of
Barcelona it moved a similar shifted building up 1 in rank.
The "Minkowski distance" that was changed several times during our experiments did
not make dramatic changes, but moved some further images close when P is enlarged.
May be used to tuning when similar content image exists, but is not ranked top.
Enlarging P in that case can "push" its rank higher.
References:



Dr. Fuhui Long, Dr. Hongjiang Zhang and Prof. David Dagan Feng.
Content Based Image Retrieval.
Dr. Fuhui Long, Dr. Hongjiang Zhang and Prof. David Dagan Feng
An Effective Region-Based Image Retrieval Framework
Yossi Rubner, Carlo Tomasi, and Leonidas J. Guibas
The Earth Mover's Distance as a Metric for Image Retrieval