Relevance
in information science
© Tefko Saracevic
1
Preface
1975 & 1976

Relevance: A Review of the Literature
and a Framework for Thinking on the
Notion in Information Science
2006

© Tefko Saracevic
Relevance: A Review of the Literature
and a Framework for Thinking on the
Notion in Information Science. Part II
Relevance
2
Purpose
trace the evolution of thinking on
relevance in information science for
the past three decades
provide an updated framework within
which the still widely dissonant ideas
on relevance might be interpreted
and related to one another
© Tefko Saracevic
Relevance
3
Relevance & information
technology
Relevance around forever
Computers ~ 50 years

internet ~ 30; web ~ 15
IT changed information activities
Systems designed to provide
relevant information
Invisible hand of relevance always
present
© Tefko Saracevic
Relevance
4
… in information science
Information retrieval (IR) systems
offer their version of what may be
relevant
People go their way & asses relevance
The two worlds interact
Concern here:


© Tefko Saracevic
human world of relevance
how IR deals with relevance NOT covered
Relevance
5
Significance of relevance
research
Contribution of relevance
scholarship:

knowledge for knowledge sake
 better understanding of the notion of
relevance significant in itself

potential for making information systems
better – more human oriented
 technological advances often dependent on
better understanding of underlying notions,
phenomena
© Tefko Saracevic
Relevance
6
Historical note
Relevance came into IR unannounced
From very start after WWII IR was
about retrieval of relevant information
First concerns: about “false drops” –
non relevant retrievals
First direct recognition in 1955 with
proposal for precision & recall as
evaluation measures based on relevance
© Tefko Saracevic
Relevance
7
Organization of
presentation
1. What is the NATURE of relevance?
 1.1 Meaning?
 1.2 Theories?
 1.3 Models?
2. What are MANIFESTATIONS of
relevance?
3. What is the BEHAVIOR of
relevance?
4. What are the EFFECTS of
relevance?
© Tefko Saracevic
Relevance
8
1.1 Meaning of relevance
Intuitively well understood


same perception globally – “y’know”
a “to” and context always present
Relevance:


a relation between objects P & Q along
property R
may also include a measure S of the strength
of connection
© Tefko Saracevic
Relevance
9
… in information science
Relation between inf. or inf. objects
(Ps) & contexts (Qs) based on some
property R & measure of intensity S
But: relevance is not given – it is
established
Big questions & challenges


© Tefko Saracevic
How does relevance happen?
Who does it, under what circumstances,
and how?
Relevance
10
Inference:
created - derived
Systems & automatons create
relevance
Users derive relevance


but could be more complex & a continuum
derive could involve more than topicality
 greater expertise = greater derivative powers


create could involve “intelligence”
also involves
 intent
 selection & interaction processes
© Tefko Saracevic
Relevance
11
Summary:
Meaning of relevance
Relevance attributes involve:



relation
intention
context
 internal; external




© Tefko Saracevic
inference
selection
interaction
measurement
Relevance
12
1.2 Theories of relevance
Theories suggested in several fields


logic – in deduction of inferences, to reject
fallacies – relevance logics
philosophy – in phenomenology structure &
functioning of the “life-world” (Alfred Schutz)
 it is stratified & relevance is the principle for
stratification
 no single relevance but an interdependent
system of relevances (plural)
 thematic (topical), interpretational, &
motivational relevance
© Tefko Saracevic
Relevance
13
… theories in
communication
Sperber & Wilson: Relevance Theory




based on inferential model of communication
what must be relevant and why to an
individual with a single cognitive intention?
posited a cognitive & a communicative
principle of relevance
assessed in terms of cognitive effects &
processing effort
 individuals pick up the most relevant stimuli &
process them to maximize their relevance
© Tefko Saracevic
Relevance
14
Summary:
Theories of relevance
IS did not develop indigenous theory
but a few “theories-on-loan” attempts



Logic theories not applied, yet
Schutz’s life-world theory used to some
extend in specifying manifestations &
models
Sperber & Wilson’s theory used few
times as explanation & guide
 not tested
© Tefko Saracevic
Relevance
15
1.3 Models of relevance
Reviews – also produced models
Syracuse school of relevance

dynamic & situational model
& Nilan, 1990)
(Schamber, Eisenberg
 connection with human information behavior
“Whole history of relevance”

(Mizzaro, 1997)
duality in modeling & studying
 documents & queries, 1959-1976
 dynamics & multidimensionality, 1977-present
© Tefko Saracevic
Relevance
16
Split between system &
user models
Opposing views of IR: systems & users
 IR traditional model does not deal with users
Battle royal started by Dervin & Nilan
(1986)
 criticism of system viewpoint
 call for user orientation
Several user & interaction models
proposed – to reconcile, bridge

still relevance has two basic models &
cultures & they map like Australia
© Tefko Saracevic
Relevance
17
Stratified model
Situational
Affective
intent; motivation ...
Cognitive
knowledge; structure...
Query
USE
R
Ada
pta
t io n
characteristics …
hardware; connections...
Processing
software; algorithms …
Content
inf. objects; representations...
© Tefko Saracevic
n
o f in
form
atio
Engineering
MP
UTE
R
Ada
ptat
ion
INTERFACE
CO
INT
ER
AC
T
ION
&
Surface level
Use
RE
LE
VA
NC
E
ST
RA
TA
tasks; context...
Relevance
18
User vs system models
“Informing systems design”

became mantra of all relevance studies
“Tell us what to do and we will do it.”

response from systems side
But “telling” is not that simple
Issue is not a conflict but:

how can we make user & system side work
together for benefit of both?
© Tefko Saracevic
Relevance
19
Summary:
relevance models
All IR & inf. seeking models have
relevance at their base
Traditional IR model has most
simplified –“weak”- version of relevance

but with the weak model IR is successful
Variety of integrative models have
been proposed

more complex models = increased challenge
to incorporate in practice
© Tefko Saracevic
Relevance
20
© Tefko Saracevic
Relevance
21
2. Manifestations of
relevance
“How many relevances in IR?”
(Mizzaro, 1998)
 Several manifestations recognized since 1950s
 Issue: What given objects (Ps & Qs) are related
by what given property (Rs) as relation?
 [adjective] relevance or different name
Duality strikes again


subject (topic, system) relevance
vs user (psychological, cognitive) relevance
objective vs subjective relevance
© Tefko Saracevic
Relevance
22
Issue of primacy:
weak and strong relevance
Does topical relevance underlie all
others?
 predictably two answers: yes and no


in a strict correspondence between
query & answer topical is basic – weak
relevance
if derivation is involved topical may not
be basic – strong relevance
 weak relevance is more associated with
systems
 strong more with people
© Tefko Saracevic
Relevance
23
Beyond duality
Numerous other kinds of relevance
were identified

for user relevances:
 psychological, cognitive, affective, situational,
socio-cognitive, pertinence, utility …

for topical relevances:
 logical, systems, algorithmic, documentary,
bibliographic …

each indicates different relations
© Tefko Saracevic
Relevance
24
Summary:
Manifestations of relevance
There are a limited number of
manifestations – could be grouped:





System or algorithmic relevance
Topical or subject relevance
Cognitive relevance or pertinence
Situational relevance or utility
Affective relevance
These are interdependent – they
feed on each other interactively
© Tefko Saracevic
Relevance
25
3. Behavior of relevance
Relevance does not behave, people do


how humans determine relevance of
information or information objects?
reviewed only studies that have data
 some 30 studies – started in 1991
 related to information seeking & use studies &
implicit relevance studies – not reviewed
Pattern for review:
 [author] used [subjects] to do [tasks] in order to
© Tefko Saracevic
study [object of research]
Relevance
26
Relevance clues
What makes information or information
objects relevant? What do people look
for in order to infer relevance?
 two approaches: topic & clues analysis
Clues research:
 uncover & classify attributes or criteria that
users concentrate on while making relevance
inferences
 usually on documents but also other objects
© Tefko Saracevic
Relevance
27
Examples
Schamber (1991) interviewed 30 users of weather
information using different sources, from oral
reports to documents and maps in order to derive
and categorize their relevance criteria. She
identified 22 categories in 10 groups.
Barry (1994) interviewed 18 academic users (not
specified as being students or faculty) who had
requested an information search for documents
related to their work in order to categorize their
relevance criteria. She identified 23 categories in
7 groups
© Tefko Saracevic
Relevance
28
Relevance dynamics
Do relevance inferences and criteria
change over time for the same user
and task, and if so, how?
As a user progresses through various
stages of a task



© Tefko Saracevic
the user’s cognitive state changes
the task changes as well
thus, something about relevance also is
changing.
Relevance
29
Relevance feedback
What factors affect the process of
relevance feedback?

What types of feedback? How much is
feedback used?
Dealing here with manual not
automatic feedback

© Tefko Saracevic
behavior of people when involved in
manual feedback
Relevance
30
Summary on behavior
Caveats abound – nothing standardized

still refreshing to see data
Some generalizations on clues:

criteria finite in number, similar, but
different weights assigned
 Different users, tasks, progress in tasks, classes
of users = similar criteria = different weights
 Different ratings of relevance = similar criteria
= different weights.
© Tefko Saracevic
Relevance
31
…summary clues
Clues criteria:







© Tefko Saracevic
content
object
validity
use or situational match
cognitive match
affective match
belief match
Relevance
32
… summary clues
Criteria are not independent; people
apply multiple criteria; they interact


content (topic) criteria very important
but not sole – interact with others
for search outputs value of results as a
whole critical
Visual information = faster inference
than textual information
© Tefko Saracevic
Relevance
33
… summary dynamics
Inferences dependent on task stage




© Tefko Saracevic
criteria stable, selection changes
Different stages = differing selections
but different stages = similar criteria =
different weights
Increased focus = increased
discrimination = more stringent
relevance inferences
What is topical changes with progress in
time and task
Relevance
34
… summary feedback
Several kinds

search term, content, magnitude, tactics
Use of relevance feedback = increase
in performance

however, used rarely in practice
Searching behavior different when
using feedback
© Tefko Saracevic
Relevance
35
5. Effects of relevance
Works both ways: relevance affected by
and affects host of factors
Relevance judges


What factors inherent in relevance judges make
a difference in relevance inferences?
How large are and what affects individual
differences in relevance inferences?
 similar question asked for a number of information
activities – indexing, searching …
 most often studied: domain knowledge
© Tefko Saracevic
Relevance
36
Relevance judgments
What factors affect relevance
judgments?


short answer: a lot of them
approach: classify into tables e.g.
 Schamber (1994) 80 factors, 6 categories
 Harter (1996) 24 factors, 4 categories

different approach here:
 classify studies along basic assumptions in IR
evaluations
© Tefko Saracevic
Relevance
37
Central assumptions
Relevance is:






topical
binary
independent
stable
consistent
if pooling: complete
Not to prove or disprove these assumptions
but to organize studies along questions
© Tefko Saracevic
Relevance
38
Beyond topicality
Do people infer relevance based on
topicality only?
Other factors enter & interact
Only a few studies directly addressed
 Wang & Soergel (1998) 11 criteria for
selection, with topicality being top
 Xu & Chen (2006) in web searching: topicality &
novelty most significant, then reliability &
understandability
© Tefko Saracevic
Relevance
39
Beyond binary
Are relevance inferences binary i.e.
relevant – not relevant? If not, what
gradation do people use in inferences
about relevance of information or
information objects?
Number of studies addressed this


studied distributions of relevance
inferences
regions of relevance
© Tefko Saracevic
Relevance
40
Beyond independence
Are information objects assessed
independently of each other? Does the
order or size of the presentation affect
relevance judgments?


Only a few studies on the questions
Includes presentation of different
representations
© Tefko Saracevic
Relevance
41
Beyond stability
Are relevance judgments stable as
tasks and other aspects change? Do
relevance inferences and criteria
change over time for the same user and
task, and if so how?

mentioned already under dynamics
 judgments not completely stable, criteria are
 Plato: “Everything is flux.”
© Tefko Saracevic
Relevance
42
Beyond consistency
Are relevance judgments consistent
among judges or group of judges?
 human judgments about anything informational
are not consistent, relevance included
Gull (1956) opened the Pandora box
 classic example of law of unintended
consequences
Some 6 studies addressed consistency

© Tefko Saracevic
subjects: experts, students
Relevance
43
But does it matter?
How does inconsistency in human
relevance judgments affect results of
IR evaluation?

main contention by critics
Five studies 1968 – 2000 addressed
the question

© Tefko Saracevic
four also showed magnitude of agreement
Relevance
44
Summary: judges
Subject expertise accounts strongly
 higher expertise = higher agreement, less
differences
 lesser expertise = more leniency in judgment

large variability in relevance inferences
by individuals
 same range as in other cognitive processes
© Tefko Saracevic
Relevance
45
Summary: judgments
Relevance is measurable!
None of the 5 postulates hold

but by simplifying relevance for labs IR
made significant advances
Relevance judgments are not binary
but are bimodal

regions of low, middle and high relevance
 high peaks at both end
Order affect relevance judgment
© Tefko Saracevic
Relevance
46
… summary: judgments
Consistency:
 Higher expertise = higher consistency =
more stringent. Lower expertise = lower
consistency= more encompassing
 overlap using different populations hovers
around 30%
 higher expertise up to 80%
 when 3rd, 4th … judge added overlap falls
 Higher expertise =larger overlap. Lower
expertise =smaller overlap. More judges =
less overlap.
© Tefko Saracevic
Relevance
47
Summary: does it matter?
In lab conditions disagreement among
judges does not affect evaluation

rank order of different IR systems changes
minimally
 Different judges = same relative performance (on
the average)


swaps in ranking do occur = low probability
but performance for individual topics differs
significantly
 law of averages kicks in
© Tefko Saracevic
Relevance
48
Summary: measures
Users can use a variety of scales


© Tefko Saracevic
there is no “best” scale
magnitude scales very appropriate but
hard to explain & analyze
Relevance
49
Epilogue
Many things changed in IR &
information science but goals the same
As to nature of relevance:
 marked progress in understanding
 little in theory
 diversification in models in models
As to manifestations:
 consensus there are several kinds of
relevance, grouped in a half dozen or so well
distinguished classes - interdependent
© Tefko Saracevic
Relevance
50
… epilogue
As to behavior & effects:

seen a number of experimental &
observational studies
 lifted the discourse beyond debate,
anecdotes to data interpretation
 but generalizations difficult – findings
should be treated as hypotheses
© Tefko Saracevic
Relevance
51
… epilogue: reflections
Relevance is poor

no funding for relevance research
 of studies with data less than 17% mentioned outside
funding, half from outside the US
 scholarship progressed sporadically & all over the
place
Globalization of IR – globalization of
relevance

© Tefko Saracevic
as relevance went global & to the masses many
& different research questions emerged
Relevance
52
… epilogue: reflections…
Proprietary IR – proprietary relevance

major search engines proprietary =
relevance proprietary
 for innovation must study users & use, but
findings kept private
 relevance research into public & private branch
 paradox of the internet
© Tefko Saracevic
Relevance
53
… epilogue: research
agenda - beyonds
Beyond behaviorism & black box


© Tefko Saracevic
many studies stimulus-response & no
diagnostics
need to go use/adapt other approaches
Relevance
54
… epilogue: research
agenda - beyonds
Beyond mantra

“implication for system design”
 incorporating user concerns &
characteristics not that simple
 integration between user/cognitive &
systems approaches needed
 relevance research and IR research should
at least get engaged, if not married
 interactive research on the right track
© Tefko Saracevic
Relevance
55
… epilogue: research
agenda - beyonds
Beyond students

~70% of behavior & effect studies used
students as population
 not surprising – they are affordable
 we know a lot about student relevance
 does it generalize to other populations?
© Tefko Saracevic
Relevance
56
In conclusion
Information technology & systems
will change dramatically


even in the short run
and in unforeseeable directions
But relevance is here to stay!
© Tefko Saracevic
Relevance
57
© Tefko Saracevic
Relevance
58
© Tefko Saracevic
Relevance
59
Acknowledgments
Ying Zhang & Yuelin Li for searches &
organization
Sandra Lanman for editing
Students in Spring 2006 class Human
Information Behavior for comments

Jeanette de Richemond for corrections
SCILS for research & collegial
atmosphere. BEST COLLEAGUES!
Gus Friedrich for understanding & time
© Tefko Saracevic
Relevance
60