WikiQuery.org
-- An interactive collaboration interface for
creating, storing and sharing effective CNF queries
Le Zhao*, Xiaozhong Liu#, Jamie Callan*
*: Language Tech Institute, SCS, Carnegie Mellon University
#: School of Information, University of Indiana Bloomington
@OSIR 2012, Portland, OR
Status Quo
• Current open source search engines
– Good at: attracting software apps/service providers
• Lucene, Lemur, Terrier, …
–
Lacking: end users
• study users’ search behavior
• Two necessary conditions for success:
– Attract users
• with a unique feature/functionality
• (not offered by current Web search engines)
–
Retain users
• not easily copied by the current search engines
This Work
• Unique opportunity
– The term mismatch problem
– Significant problem w/ huge potential [Zhao10,12]
– Web search engines: automatic expansion
– Lots of room for manual expansion
• Solution
– Conjunctive normal form expansion
– Commonly & effectively used by expert searchers
[Lancaster68,Harter86,Hearst96,Baron07]
3
Term Mismatch Problem
• Average term mismatch rate: 30-40% [Zhao10]
Relevant docs not returned
Web, short queries,
stemmed, inlinks
included
• A common cause of search failure [Harman03, Zhao10]
• Frequent user frustration [Feild10]
• More than 50-300% gain in retrieval accuracy [Zhao12]
4
Term Mismatch & Boolean
Conjunctive Normal Form (CNF) Expansion
Keyword query:
approval of logos on television watched by children
Manual CNF (TREC Legal track 2006):
(approval OR guideline OR strategy)
AND (logos OR promotion OR signage OR brand OR mascot OR
marque OR mark)
AND (television OR TV OR cable OR network)
AND (watched OR view OR viewer)
AND (children OR child OR teen OR juvenile OR kid OR adolescent)
–
–
–
–
Expressive & compact (1 CNF == 100s alternatives)
Highly effective (50-300% over base keyword [Zhao12])
Widely used by experts (library, legal, medical …)
5
But, tedious to create
Goal of WikiQuery
To facilitate the
– creation (a proper tool is lacking for practitioners)
– storing (for refinding & sharing)
– and sharing (for collaboration on query parts)
of high quality CNF queries
6
One Wiki Page == One Search Need
Displaying CNF Query
Links to Search Result Pages
• Relying on existing search engines for results
Creating/Editing CNF Queries
Search Result Access When Editing
Interactions with search results are usually necessary to
ensure quality of the created query.
Looks Similar, but Different from
• Library advanced search: simple Boolean
– For example, Library of Congress Advance search:
–
Term1 AND/OR Term2 AND/OR Term3
• LexisNexis: free form Boolean
• ERIC: more flexible, joining with AND or OR
Evaluating the WikiQuery Interface
13
Experiment Setup
• Hypothesis
– Users (with limited knowledge about Boolean queries)
can create effective Boolean queries using WikiQuery
• Preliminary user studies
– Classroom users
• 6 students, limited prior knowledge of Boolean queries or IR
• a 10 minute session of editing Wiki pages
–
TREC topics => user Boolean CNF queries
• Total 12 topics, 12 final Boolean queries
• Interacted with Google and Yahoo (Spring, 2010)
(40 minutes per topic)
• Evaluating on the Web & TREC collections
14
Results
• :-)
–
on average 30-50% gain over keyword queries
• :-(
–
But, not consistent (Web eval > TREC eval)
– not statistically significant over strong kw baseline
• :-O
– 75% contain more restrictive formulations, unstable
– < 50% of queries are CNF expanded, bit more stable
• Need better tools to check quality of expansion terms
15
Conclusions
• Conjunctive Normal Form expansion has great potential
• Users need guidance and learning to create effective
CNF queries
– need to be warned against more restrictive queries
• Uses of WikiQuery:
– Expert searchers
– Classroom: becoming experts
– Whenever you face a hard query
16
Source Code
• Based on MediaWiki version 1.17
• Available at
https://github.com/lezhao/wikiquery
• Questions: now OR catch me at lunch!
17