YOU MEAN WHAT I
KNOW!!!
Orthogonal Query Recommendation
What is the problem?
• When a query is poorly formulated, search engines
struggle to provide alternatives to the originally malformed query.
Why is This a Problem?
• Most techniques seek to enrich a user’s query with
semantically similar alternatives.
• Gets the user no where fast.
Graphical Depiction of Search Results
Orthogonal Queries
Consider the following graph…
Previous Work
• Cluster related queries
• (CG) Cover Graph
• Exploits click-through data
• (UF-IQF) User frequency inverse query frequency
• Based on entropy
• (SC) Search short cuts
• (SQ) Query traces as a way of finding related queries
• Queries with similar terms and results are scored higher
• Direct opposite of proposed approach
• Summary – previous methods have the pitfall of
recommending queries that are simple reformations of an
original, mal-formed query.
Solution
• “Deliberately seek out queries that are only slightly similar
to the user’s original query”
• Calculated use of orthogonality
• For diversity of results
• Use of cached queries
• For efficiency
• To react to temporary changes in users’ habits
Valuable Properties of Orthogonality
• Generalization
• Take known data and make inferences about unknown data (create
bias)
• Do not take too much stock in known data (high variance)
• Orthogonality allows us to;
• Extract properties of known data (select bias)
• Extract complementary properties of data
• Make novel generalizations about unknown data
Do opposites attract?
She said
• At least he is active.
• His games can get
annoying now and then.
• Ward break-the-fasts are
nice, but I’ve lost a few
since having ‘student
budget’ wbtf’s.
• What’s tumblr?
• I love Pinterest!
He said
• Please… no more Pinterest
•
•
•
•
recipes and home décor!
Tumblr? Oh yeah, YMCA 2nd
grade. Broke my wrist.
She calls me out when I get
lazy spiritually.
Up’s and down’s to ward
dinners; ups-food, down-setup/clean-up.
I like to watch an occasional
game.
Let’s evaluate
Standard Likert Scale
She
He
Pinterest
2
-2
Sports
-1
1
Gospel
2
2
tumblr
0
0
Ward Dinners
1
1
Results
Not opposites
Not duplicates
(2 * 2) + (-1 * 1) + (2 * 2) + (0 * 0) + (1 * 1) = 0
They are orthogonal!
Term Overlap
|𝑡𝑒𝑟𝑚𝑠 𝑝 ∩ 𝑡𝑒𝑟𝑚𝑠 𝑞 |
𝑡𝑒𝑟𝑚𝑂𝑣𝑒𝑟𝑙𝑎𝑝 𝑝, 𝑞 =
|𝑡𝑒𝑟𝑚𝑠 𝑝 ∪ 𝑡𝑒𝑟𝑚𝑠 𝑞 |
𝑡𝑒𝑟𝑚𝑠 𝑝 = 𝑏𝑜𝑜𝑘, 𝑝𝑒𝑛𝑐𝑖𝑙, 𝑛𝑜𝑡𝑒𝑏𝑜𝑜𝑘
𝑡𝑒𝑟𝑚𝑠 𝑞 = 𝑎𝑟𝑟𝑒𝑠𝑡, 𝑏𝑜𝑜𝑘, 𝑑𝑒𝑡𝑎𝑖𝑛
| 𝑏𝑜𝑜𝑘 |
𝑡𝑒𝑟𝑚𝑂𝑣𝑒𝑟𝑙𝑎𝑝 𝑝, 𝑞 =
| 𝑎𝑟𝑟𝑒𝑠𝑡, 𝑏𝑜𝑜𝑘, 𝑑𝑒𝑡𝑎𝑖𝑛, 𝑝𝑒𝑛𝑐𝑖𝑙, 𝑛𝑜𝑡𝑒𝑏𝑜𝑜𝑘 |
1
5
= = 0.20
Result Overlap
|𝑟𝑒𝑠𝑢𝑙𝑡𝑆𝑒𝑡 𝑝 ∩ 𝑟𝑒𝑠𝑢𝑙𝑡𝑆𝑒𝑡 𝑞 |
𝑟𝑒𝑠𝑢𝑙𝑡𝑂𝑣𝑒𝑟𝑙𝑎𝑝 𝑝, 𝑞 =
|𝑟𝑒𝑠𝑢𝑙𝑡𝑆𝑒𝑡 𝑝 ∪ 𝑟𝑒𝑠𝑢𝑙𝑡𝑆𝑒𝑡 𝑞 |
𝑟𝑒𝑠𝑢𝑙𝑡𝑆𝑒𝑡 𝑝 = 𝑘𝑠𝑙, 𝑎𝑚𝑎𝑧𝑜𝑛, 𝑠𝑡𝑎𝑝𝑙𝑒𝑠
𝑡𝑒𝑟𝑚𝑠 𝑞 = 𝑓𝑜𝑥𝑛𝑒𝑤𝑠, 𝑐𝑛𝑛, 𝑘𝑠𝑙
| 𝑘𝑠𝑙 |
𝑡𝑒𝑟𝑚𝑂𝑣𝑒𝑟𝑙𝑎𝑝 𝑝, 𝑞 =
| 𝑎𝑚𝑎𝑧𝑜𝑛, 𝑐𝑛𝑛, 𝑓𝑜𝑥𝑛𝑒𝑤𝑠, 𝑘𝑠𝑙, 𝑠𝑡𝑎𝑝𝑙𝑒𝑠 |
1
5
= = 0.20
High result overlap = queries composed of similar terms
Slight syntactic variants composed of the same terms
Word permutations
Incorporating highly ranked results back
into the original query
LOW TERM
OVERLAP
HIGH TERM
OVERLAP
LOW
RESULT
OVERLAP
Unrelated
Unrelated
HIGH
RESULT
OVERLAP
Orthogonal
QR
Traditional
QR
What is happening?
Issues of High Correlation
Step 1: termOverlap< 0.333 for |𝑞1|, |𝑞2| ≤
4
Hail
Syllabus
CS
653
BYU
Nebraska
Mary
Select Queries in lower left hand region
𝑞 ∈ {(𝑥, 𝑦)|𝑥𝑡𝑒𝑟𝑚𝑂𝑣𝑒𝑟𝑙𝑎𝑝 = 0, 0.33 , 𝑦𝑟𝑒𝑠𝑢𝑙𝑡𝑂𝑣𝑒𝑟𝑙𝑎𝑝 = 0, 0.06 }
Is 𝑟𝑒𝑠𝑢𝑙𝑡𝑂𝑣𝑒𝑟𝑙𝑎𝑝 ∈ (0.0, 0.06] too low?
• 0.06 seems very small
• However, given the size of the web, the probability of a
false match is (10−9 , 10−5 )
• 10−9 = probability of flipping a fair coin 29 times and
having it land heads
• 10−5 = “” 17 “”
Empirical Tests
• 22,000,000 queries
• 80% for training purposes
• 20% for testing purposes
• Sessionized testing set:
• 1, 10, 20, 30 minutes
• Only satisfied session with retype
• Number of queries ≥ 2
• Only the last query received a click
What are they testing for?
• Given a set of sessions 𝑆, a session 𝑠 = 𝑞1 , 𝑞2 , … , 𝑞𝑛 ∈
𝑆, and a set 𝑅𝑞 of recommended queries;
1, 𝑖𝑓 {𝑞𝑛 } ∩ 𝑅𝑞 ≠ ∅
• 𝑆𝑞 @𝑘 =
0, 𝑖𝑓 𝑞𝑛 ∩ 𝑅𝑞 = ∅
• With 𝑅𝑞 = 𝑘. Then,
• 𝑆@𝑘 =
𝑠∈𝑆 𝑆𝑞 @𝑘
|𝑆|
Real World Example for 𝑆@2
s
𝑹𝒒
𝒔𝒒 @𝟐
(a, b, c)
(a, b)
0
(d, e, f)
(a, f)
1
(a, c, f)
(c, f)
1
Final 𝑆𝑞 @2
2/3 = 0.666
Cache Policy
• OQR at a disadvantage because it cannot access queries
outside of the cache while other algorithms can
• Training data. Divide into:
• MCQ : most clicked query
• MFQ: most frequent query
• MFFQS: Most frequent final query in session
• MRQ: Most recent query
• Which performs best on the test data?
• MCQ
• Apply this policy to order queries returned by the method
shown in the previous slide
Results: MCQ is best
𝑆@10 as a Percentage
Is OQ Really Orthogonal?
User Evaluation
Conclusions
• OQ:
• Not universally useful
• When information need can only be satisfied by tweaks in user input
• Best for unseen queries
• Best for long-tail queries
• Can be used in conjunction with other query methods
(supplementary)