Surajit Chaudhuri, Microsoft Research
Gautam Das, Microsoft Research
Vagelis Hristidis, Florida International University
Gerhard Weikum, MPI Informatik
30th VLDB Conference Toronto ,Canada,2004
Presented By
Abhishek Jamloki
CSE@UB

Realtor DB:
Table D=(TID, Price , City, Bedrooms, Bathrooms,
LivingArea, SchoolDistrict, View, Pool, Garage,
BoatDock)
 SQL query:
Select * From D
Where City=Seattle AND View=Waterfront

Consider a database table D with n tuples {t1, …, tn} over a set of m categorical attributes A = {A1, …, Am} a query Q: SELECT * FROM D
WHERE
X1=x1 AND … AND Xs=xs where each Xi is an attribute from A and xi is a value in its domain. specified attributes: X ={X1, …, Xs} unspecified attributes: Y = A – X
Let S be the answer set of Q
How to rank tuples in S and return top-k tuples to the user?


Query Reformulation
Automatic Ranking
Correlations are ignored in high dimensional spaces of IR
•
Automated Ranking function proposed based on
1) A global score of unspecified attributes
2) A conditional score (strength of correlation between specified and unspecified attributes)
Automatic estimation using workload and data analysis

•
Bayes’ Rule p ( a | b )
 p ( b | a ) p ( a ) p ( b ) p ( a , b | c )
 p ( a | c ) p ( b | a , c ) •
Product Rule
Document t , Query Q
R : Relevant document set
R = D - R : Irrelevant document set
Score ( t )
 p ( R | t p ( R | t )
)
 p ( t | R ) p ( R ) p ( t ) p ( t | R ) p ( R )
 p ( t | R ) p ( t | R ) p ( t )

 Each tuple t is treated as a document
 Partition t into two parts t(X): contains specified attributes t(Y): contains unspecified attributes
 Replace t with X and Y
 Replace R with D

 Comprehensive dependency models have unacceptable preprocessing and query processing costs
 Choose a middle ground.
 Given a query Q and a tuple t, the X (and Y) values within themselves are assumed to be independent, though dependencies between the X and Y values are allowed p ( X | C )
  x

X p ( x | C ) p ( Y | C )
  y

Y p ( y | C )




Workload W : a collection of ranking queries that have been executed on our system in the past.
Represented as a set of “tuples”, where each tuple represents a query and is a vector containing the corresponding values of the specified attributes.
We approximate R as all query “tuples” in W that also request for X (approximation is novel to this paper)
 Properties of the set of relevant tuples R can be obtained by only examining the subset of the workload that contains queries that also request for X
Substitute p(y | R) as p(y | X,W)

 p(y | W) the relative frequencies of each distinct value y in the workload
 p( y | D) relative frequencies of each distinct value y in the
 database (similar to IDF concept in IR) p(x | y,W) confidences of pair-wise association rules in the workload, that is:
(#of tuples in W that contains x, y)/total # of tuples in W
 p(x | y,D):
(#of tuples in D that contains x, y)/total # of tuples in D
 Stored as auxiliary tables in the intermediate knowledge representation layer

 p(y | w) {AttName, AttVal, Prob}
◦
B + Tree index on (AttName, AttVal)
 p(y | D) {AttName, AttVal, Prob}
◦
B + Tree index on (AttName, AttVal)
 p(x | y,W) {AttNameLeft, AttValLeft, AttNameRight, AttValRight, Prob}
◦
B + Tree index on (AttNameLeft, AttValLeft, AttNameRight, AttValRight)
 p(x | y,D) {AttNameLeft, AttValLeft, AttNameRight, AttValRight, Prob}
◦
B + Tree index on (AttNameLeft, AttValLeft, AttNameRight, AttValRight)

Preprocessing - Atomic Probabilities Module
 Computes and Indexes the Quantities
P(y | W), P(y | D), P(x | y, W) , and P(x | y, D) for All Distinct Values x and y
Execution
 Select Tuples that Satisfy the Query
 Scan and Compute Score for Each Result-Tuple
 Return TopK Tuples







Trade off between pre-processing and query processing
Pre-compute ranked lists of the tuples for all possible “atomic” queries. Then at query time, given an actual query that specifies a set of values X, we “merge” the ranked lists corresponding to each x in X to compute the final Top-K tuples.
We should be able to perform merging without scanning the entire ranked lists
Threshold algorithm can be used for this purpose
A feasible adaptation of TA should keep the number of sorted streams small
Number of sorted streams will depend on number of attributes in database

 At query time we do a TA-like merging of several ranked lists
(i.e. sorted streams)
 The required number of sorted streams depends only on the number of specified attribute values in the query and not on the total number of attributes in the database
 Such a merge operation is only made possible due to the specific functional form of our ranking function resulting from our limited independence assumptions



Index Module: takes as inputs the association rules and the database, and for every distinct value x, creates two lists Cx and Gx, each containing the tuple-ids of all data tuples that contain x, ordered in specific ways.
Conditional List Cx: consists of pairs of the form <TID, CondScore>, ordered by descending CondScore
TID: tuple-id of a tuple t that contains x
 Global List Gx: consists of pairs of the form <TID, GlobScore>, ordered by descending GlobScore, where TID is the tuple-id of a tuple t that contains x and




At query time we retrieve and multiply the scores of t in the lists Cx1,…,Cxs and in one of Gx1,…,Gxs. This requires only s +1 multiplications and results in a score2 that is proportional to the actual score.
Two kinds of efficient access operations are needed:
First, given a value x, it should be possible to perform a
GetNextTID operation on lists Cx and Gx in constant time, tuple-ids in the lists should be efficiently retrievable one-byone in order of decreasing score. This corresponds to the sorted stream access of TA.
Second, it should be possible to perform random access on the lists, that is, given a TID, the corresponding score (CondScore or GlobScore) should be retrievable in constant time.



These lists are stored as database tables –
CondList C x
{AttName, AttVal, TID, CondScore}
B + Tree index on (AttName, AttVal, CondScore)
 GlobList G x
{AttName, AttVal, TID, GlobScore}
B + Tree index on (AttName, AttVal, GlobScore)

 Space consumed by the lists is O(mn) bytes (m is the number of attributes and n the number of tuples of the database table)
 We can store only a subset of the lists at preprocessing time, at the expense of an increase in the query processing time.
 Determining which lists to retain/omit at preprocessing time done by analyzing the workload.
 Store the conditional lists Cx and the corresponding global lists Gx only for those attribute values x that occur most frequently in the workload
 Probe the intermediate knowledge representation layer at query time to compute the missing information

 The following Datasets were used:
MSR HomeAdvisor Seattle (http://houseandhome.msn.com/)
Internet Movie Database (http://www.imdb.com)
Software and Hardware:
Microsoft SQL Server2000 RDBMS
P4 2.8-GHz PC, 1 GB RAM
C#, Connected to RDBMS through DAO

 Evaluated using two ranking methods
1) Conditional
2) Global
Several hundred workload queries were collected for both the datasets and ranking algorithm trained on this workload

 For each query Q i
, generate a set H i of 30 tuples likely to contain a good mix of relevant and irrelevant tuples


Let each user mark 10 tuples in H i as most relevant to Q i
Measure how closely the 10 tuples marked by the user match the 10 tuples returned by each algorithm

 Users were given the Top-5 results of the two ranking methods for 5 queries (different from the previous survey), and were asked to choose which rankings they preferred

 Compared performance of the various implementations of the
Conditional algorithm: List Merge, its space-saving variant and Scan
 Datasets used:

 Completely automated approach for the Many-Answers
Problem which leverages data and workload statistics and correlation
 Probabilistic IR models were adapted for structured data.
 Experiments demonstrate efficiency as well as quality of the ranking system

 Many relational databases contain text columns in addition to numeric and categorical columns. Whether correlations between text and non-text data can be leveraged in a meaningful way for ranking ?
 Comprehensive quality benchmarks for database ranking need to be established