Web Search Results Visualization:
Evaluation of Two Semantic Search
Engines
Kalliopi Kontiza, k.kontiza.12@ucl.ac.uk
Antonis Bikakis, a.bikakis@ucl.ac.uk
University College London, Department of Information Studies
Overview
● Semantic search engines improve the accuracy of search results:
- by understanding the meaning and context of terms as they appear in web
documents,
- by using semantics to represent and process the user’s queries and the
web data.
● Other parameters that define the quality of a search engine:
- its performance,
-its usability,
-the presentation of the search results
Overview
“Whether and how do semantic search engines improve
the visualization of search results, enhancing the search
experience? “
Structure of the Presentation
● Methodology
● Background information (InfoVis)
○ 1. Analytical Inspection
● Experiment
○ 2. The User Evaluation
● Results of the User Evaluation
● Discussion
Methodology
● An analytical Inspection
area of heuristic evaluation
‘the Visual Information-Seeking Mantra’ , Shneiderman (1996)
●
A user-oriented evaluation study
● Interactive Information Retrieval (IIR) systems:
semantic search engines Sig.ma and Kngine
1. Background Information
Information Visualization
- Works as umbrella for all kinds of visualizations
- Best applied for exploratory tasks
- Ultimate purpose : amplify cognition
- Requires well formed data
2. Analytical Inspection
1.
2.
3.
4.
5.
6.
Overview
Details on demand
Filter out/Highlight
Relate
History
Export
➔ Layout of the SUIs: Control (ie more),
Input (ie search box)
Personalised (ie move content)
Informational (ie result item)
Task-domain
information
actions supported
by an information
visualization system, that
users wish to perform
Features of SERP
interface
2. Analytical Inspection
(Questionnaire videos)
3. Design & Set up of the User Evaluation
A. Variables
a. Dependent
i. Task domain information actions
ii. User Satisfaction
b. Independent
Predefined queries: a) Web
b) Informational
1. Overview
2. Details on demand
3. Filtering out
4. Relate
5. History
6. Export
Transactional
Navigational
Factual
Source
3. Design & Set up of the User Evaluation
B. Questionnaire
- Online, closed-type questions, 5 point Likert,
- Pre assigned queries presented in playlist of videos
- Sections:
A. Introduction
B. Evaluation
C. About
C.
Sample
- 83 participants
4. Results of the User Evaluation
● 55% male, 45% female
● 34% 18-26 age group, 51% 27-33 age group, 11% 34-40 age
group
● 67% had used more than one search engines
● 86% rated their search skills with 4 and 5 on 5 point Likert scale
The comparative presentation of user ratings for the visualization of the task domain
information action criteria
4. Results of the User Evaluation
● Informational tasks received 71%,
61%
● Visualization was ranked 4th, 73%
graded it with 4 and 5 on 5 point
Likert scale
● User-satisfaction perceived
acceptance:
o good satisfaction for history
and export but
o more expectations from
overview and details on
demand
5. Discussion
-Different perspective to view data
Q1. The visualization of the search
results in semantic search engines
improves the understanding of data and
supports the user in assessing search
results.
-Non linear and dynamic visualization
favoured
5. Discussion
- Visualization was ranked important in
tasks as Overview, Details on demand,
Filter out, Relate
Q2. Semantic search engines make more
effective use of visualization in
displaying search results providing a
better user experience.
-Careful consideration regarding
additional visual representations
5. Discussion
-User can filter out due to the
semantically organised data in
properties and values
Q3. Semantics improve the visualization
of search results.
-The visualization of that task receives
high preference amongst users
5. Discussion
-Users satisfied in general with the
visualizations of the semantic search
engines
Q4. The visualization of search results in
semantic search engines provides a
better search and thus user’s
satisfaction.
-Visualization of search results plays a
significant role in shifting users
searching behaviour
Conclusions
● While visualization methods used by semantic search engines improves user
understanding of the results, the extent to which visualization methods are used
in such search engines can be improved even more.
● User experience rated positively but user satisfaction not accomplished in all
cases
Further questions to investigate
A more in-depth analysis needs to be performed on the collected data:
● Are there any differences in the results of the user evaluation for the different
types of queries, considering the type of data that is searched or the complexity
of the query?
● Is there any correlation, for example, between the user characteristics and the
obtained data?
● Could a standardized cognitive and ability test help us further investigate the
relationship between information visualization in semantic search engines and
knowledge visualization ?
Thank you for your attention