1
1
•
Introduction to Multimedia indexing.
• Types of multimedia queries.
• Spatial access methods.
• Sequential scan method for multimedia queries.
• GEMINI approach.
• GEMINI Vs Sequential scan.
• Query By Image Content (QBIC) indexing and searching using GEMINI.
2
2
1

• Design of fast searching methods that will search a database of multimedia objects that match a query object.
• There is a huge demand for such search in fields like marketing, production, journalism and medical.
• Example for a multimedia query : in a collection of color photographs, find ones with the same color distribution as a sunset photograph.
3
3
• To compare two patterns we need to know how similar or dissimilar they are with each other.
• This distance can be quantified.
• Given two objects I and J the distance(=dissimilarity) of two objects is denoted by, D(I,J).
• For example, of two objects are two equal length time series, the distance D() can be Euclidean distance.
4
4
2

• Similarity queries can be classified in to two categories.
• Whole match queries : give a collection of N objects o1, o2, … on and a query object Q, we want to find those objects that are with in distance å from Q. here query and objects are of same type.
• Sub-pattern Match queries : Here the query is allowed to specify only part of the object. Given N data objects o1, o2,… a query (sub) object Q and a tolerance å, we want to identify the parts of data objects that match Q.
5
5
•For any query type the ideal method should fulfill the following requirements.
• It should be fast.
• Correct.
• Should require a small space overhead.
• Should be Dynamic.
6 6
3

• The strategy for searching multi-dimensional objects is, to map the objects in f-D space, and use multi-attribute access methods to cluster them and search for them.
• These access method are also known as spatial access methods.
• These methods can be classified in to three types.
1. R*-trees and rest of the R-tree family.
2. Liner quad trees
3. Grid files.
7 7
• R-tree methods seem to be most robust for higher dimensions.
• The R-tree represents a spatial object by it’s minimum bounding rectangle (MBR). Data rectangles are grouped to form parent nodes, which are recursively grouped, to form grand parent nodes and, eventually, a tree hierarchy.
• The MBR of a parent node completely contains the MBR’s of it’s children MBR’s, and are allowed to overlap.
8
8
4

• The solution for whole match query is to apply sequential scanning for each and every object O i
(1<= i <= N), we computer distance from Q and report the objects with distance D(Q, O i
) <= å.
• Sequential scanning may be slow for two reasons:
1. The distance computation might be expensive.
2. The database size N might be large.
9
9
• The GEMINI approach is based on the following two ideas.
• It is a ‘quick-and-dirty’ test to discard the vast majority of non-qualifying objects(possibly, allowing some false alarms)
• The use of spatial access methods, to achieve faster than sequential searching.
10
10
5

• Let’s consider a database of time series, such as yearly stock price movement, with one price per day. Distance function between two such series S and Q is the Euclidean distance.
D(S, Q) = (Ó i=1
365 (S[i] - Q[i]) 2 ) 1/2 where S[i] stands for the value of stock S on i th day.
11
11
• The idea behind GEMINI approach is to characterize a sequence with a single number, which will help to discard many non-qualifying sequences.
• This characteristic is called as feature.
•Let’s take average stock price over the year as the feature.
12 12
6

• If two stacks differ in their average by a large margin, it is impossible that they will be similar. The converse is not true, which is exactly the reason we may have false alarms.
• We can add some features to reduce false alarms, like standard deviation of the stock prices.
13
13
• In GEMINI method, the crucial observation is that we can guarantee that there will be no false dismissals if the distance in feature space matches or understands the distance between two objects.
• Lemma : To guarantee no false dismissals of whole match queries, the feature extraction F() should satisfy the following formula :
D feature
(F(o1), F(o2) <= D(o1, o2)
14
14
7

• Step1: Determines the distance function D() between two objects.
• Step2: Find one or more numeric feature-extractions, to provide a ‘quick-and-dirty’ test.
• Step3: Prove that the distance in feature space lowerbounds the actual distance D(), to guarantee correctness.
• Step4: Use a SAM to store and retrieve the f-D feature vectors
.
15
15
• First step involves a domain expert
• Second step requires intuition and imagination. It starts by trying to answer the ‘Feature-extracting question’ : if we are allowed to use only one numerical feature to describe each data object, what should this feature be?
16
16
8

• The GEMINI approach can solve two well known problems.
1. The dimensionality curse(Time series)
2. The ‘cross-talk’ of features(color images)
17
17
• If we take sequences of artificially generated random walks, with length n=1024, their number N varies from 50 to 400.
•From the response times for methods(GEMINI and
Sequential Scan), as a function of number of sequences N,
Clearly GEMINI outperforms the sequential scanning
.
18
18
9

•
GEMINI can be successfully applied to time series.
•The success in 1D series suggests that GEMINI is promising for 2D or highest dimensionality signals
.
19 19
• Potential applications for 2D color image search include medical, journalism and many others in art, fashion cataloging, retailing and industry.
•An example for color based query : search for images that contain tumor with texture like this one.
20 20
10

• GEMINI has also been applied for color images with in the QBIC (Query By Image Content) project of
IBM. This project studies methods to query large online image database using images content as the basis for queries.
21
21
• To follow GEMINI approach, first we need to find a distance function. For color, we compute a k-element color histogram for each item and scene, where k=256 or 64 colors
• Once these histograms are computed, one method to measure the distance between two histograms (k x 1 vectors) x and y is given by, d hist
2 ( x, y) = ( x - y) t A ( x - y) = Ó k i
Ó k j a ij
(x i
- y
I
)(x j
- y j
) where the superscript t indicates matrix transposition, and the color-tocolor similarity matrix A has entries a ij between color i and color j .
which describe the similarity
22 22
11

•
The two main problems with searching multimedia objects are -
Dimensionality curse and cross-talk.
• If we build a k-element color histogram for each item and scene, where = 256 or 64. Each component in the color histogram is the percentage of pixels that are most similar to each other.Then we have to map the object in to k-dimension space, which is not feasible.
• Since the distance feature obeys lower bounding lemma, the feature space involves cross-talk among the features
.
23
23
•
These two problems can be solved with GEMINI approach.
• We consider the average amount of red, green, and blue in a given color image as the feature. Color of the individual pixel can be described by the triplet(R, G, B).
24
24
12

•The average color vector of an image or item x = (R avg
, G avg
, B avg,
) t where, R avg
= (1/p) Ó p=1 p R(p),
G avg
= (1/p) Ó p=1 p G(p),
B avg
= (1/p) Ó p=1 p B(p).
Where p is the number of pixels in item, and R(p), G(p), B(P) are the red, green and blue components respectively of the p-th pixel
25
25
•
Given average colors x and y of two items, we define d avg as the Euclidean distance between the three-dimensional average color vectors, d avg
2 ( x, y) = ( x - y) t ( x - y) where, x and y are vectors of two items.
• given a color query, our retrieval proceeds by first filtering the set of images based on their average ( R, G, B) color, then doing a final accurate matching using their full k-element histogram .
26
13

• The GEMINI approach motivated a fast method, using the average RGB distance
• In addition to resolving the cross-talk problem, GEMINI solved the ‘dimensionality-curse’ problem at no extra cost, requiring only f=3 features, as opposed to k= 64 or 256 that d hist required.
27
•
GEMINI is useful for any setting that we can extract features from.
• Algorithms for automatic feature extraction methods exist, like the ‘Multidimensional Scaling’ (MDS) and ‘Fast Map’
.
28
14

• Modern Information Retrieval: R. Baeza-yates, B. Ribeiro-
Neto
• Image Databases: Vittorio Castelli and Lawrence D. Bergman.
• Some useful links in multimedia Indexing and Searching http://www.searchtools.com/info/multimedia-search.html
http://www.evisionglobal.com/tech/demo.html
• For further information on QBIC, http://wwwqbic.almaden.ibm.com/
29
15