information
retrieval
perspectives: information retrieval
•
•
•
•
•
•
•
•
application(s) - surveillance
psychological - information overload
experimental - search engines
algorithmic - (media) similarity metrics
system - unified (media) repository
presentation - user profile / integration
search - precision & recall
commercial - applications & certification
media types
text, images, audio, video, structured data
retrieval
 image query -- all images with this person
 audio query -- identity of speaker
 text query -- all transactions with BANK Inc.
 video query -- all segments with victim
 complex queries -- convicted murderers with BANK transactions
 heterogeneous queries -- photograph + murderer + transaction
 complex heterogeneous queries -- in contact with + murderer + transaction
 precision -- how many answers are correct
 recall -- how many of the right documents are returned
precision = ( returned and relevant ) / returned
recall = ( returned and relevant ) / relevant
frequency table
term/document
d0
snacks 1
0
0
drinks 1
0
3
rock-roll
0
1
d1
d2
1
indexing
video content
video v, frame f
f has associated objects and activities
objects and activities have properties
object instance: (oid,os,ip)
 object-id -- oid
 object-schema -- os = (fd,fi)
 set of statements -- ip: name = v and name = v IN f
video
frame objects
frame-dependent properties
1
Jane
has(briefcase), at(path)
-
house door(closed)
-
briefcase
2
Jane
-
Dennis at(door)
-
house door(open)
-
briefcase
has(briefcase), at(door)
object frame-independent properties
value
Jane
35
age
height
170cm
house address
...
color
briefcase
size
brown
color
black
40 x 31
query language for video libraries
 segment retrievals -- exchange of briefcase
 object retrievals -- all people in v:[s,e]
 activity retrieval -- all activities in v:[s,e]
 property-based -- find all videos with object oid
SQL, XML, logic-based, ...
media
abstractions
media abstraction
 state -- smallest chunk of media data
 feature -- any object in a state
 attributes -- characteristics of objects
 feature extraction map -- to identify content
 relations -- to capture state-dependent information
 (inter)relations between 'states' or chunks
VO = { (O_i,Q_i,C_i) | 1 <= i <= k }
where
 C_1,...,C_k -- mutually exclusive conditions
 Q_1,...,Q_k -- queries
 O_1,...,O_k -- objects
virtual objects
user
interface
augmented virtuality
information retrieval: issues
• retrieval -- indexing & success metrics
• video -- (temporal) description logic
• applications -digital dossier
Create a VR that realizes a digital dossier for a work of a particular artist.
A digital dossier represents the information that is available for a particular
work of art, or a collection of works, of a particular artist.
http://www.cs.vu.nl/~eliens/casus/
perspectives: information retrieval
•
•
•
•
•
•
•
•
application(s) - digital dossier
psychological - focus
experimental - user interaction
algorithmic - (information) access
system - unified presentation space
presentation - embodied agents
search - semantic annotation
commercial - future systems