Adding semantics to multimedia European projects
Imagina 2002
Franco Mastroddi
European Commission
DG Information Society
1
‘Mega-trends (ISTAG expert Group)’
ICT today
“Ambient Intelligence” tomorrow
• Micro scale……………………
Nano-scale
•PC based ……………………
Information devices - many embedded..
• “Writing and reading”…….….
Use all senses, touch, vision, hearing...
• “Word” based metadata……
Content- and context-based systems
• Low bandwidth, separate networks.
Infinite bandwidth, cross-media ...
• Mobile telephony (voice)……..
Mobile/Wireless full multimedia
• eServices just emerging………
Wide adoption (eHealth, eLearning...)
•Only 5% of global population on-line..
>70% of world-wide population on line
2
Multimedia demographics have changed

In early 1990s the typical Web user was an American male student

Today
 many users are ‘normal’ consumers (Insight Express)
 female users overtaken males in US (Nielsen)
 worldwide, more than 65 million youths (5-17) have Internet
access at home or school (Datamonitor).
 Web is used increasingly for trade and professional services, ecommerce etc

vast new opportunities…
 broader market
 economies of scale
3
New forms of interactivity

from scribe to screen. Over 90 % of text / data is digital origin.
Soon also for audio-visual.
From

linear to non-linear content management
from mass broadcast to mass personalisation
from
single media to cross-media.
from
static to dynamic multimedia resources. Self-managing digital
objects, links, embedded semantic metadata / ontologies.
from
geographic branding to virtual community-building

from disintermediation to re-mediation through new content
middleware systems.
4
Is there too much content?

Example of consumer photographs

annual output: billions of photos / year.

Present storage method: albums & shoeboxes
“people want to do more than print a few
pictures and store the rest on their hard disk”


but the average consumer is not a good
indexer…

Professionals do not fare much better...
“1,5
hours to scan 42 photos...but 43 hours to
index them…” (Washington State 2000 Diglib
project)
5
Is there enough technology?

Increasing processing power


Data crunching < > content management
with new telecomms networks

mobile (UMTS still in the future - 4% global
penetration by 2004 - UMTS Forum)


Bluetooth, ambient systems
and new a/v devices

over 17 mio digi-cams shipped in US in 2001
(Hallmark Card Store)

can integrate into flexible content platforms

e.g. digital camera as front-end of an integrated
home image system via wireless/USB to PC or settop-box - thence to Internet.
6
Does the Web really help?

The Web paradigm does not necessarily help in
multimedia searching...

“Size of the database, frequency of update,
Which one is
the criminal?
search capability and design, and speed may lead to
amazingly different results” (Kansas City Public
Library)
“search engines are undergoing a profound
evolution ... such as... ways of automatically
classifying information” (B. Grossan, Lawrence
Berkely Library)

“a very small number of commercially available
products exist which perform content-based image
retrieval.” (Manchester Visualisation Centre, 2001)

Attrasoft
7
Towards a Semantic Middleware.
Human
Human
Machine-Machine
Multi-modal / visual
interfaces
Information Agents
Semantic middleware :
RDF, Ontologies, Semantic-based systems and services.
Documents
Databases
Email
Web
People
Processes
8
Content description is the key
Create
Host Connect Use
The linear content chain:
Source: Gemini Consulting
The non-linear
chain:
9
Projects: Automated multimedia content
analysis
Developing new search algorithms to work
on MPEG 4/7 metadata.
Other projects:
• personalised views of a
broadcast archive
(Primavera)
• advert analysis from TV
and Web (Aramis)
FAETHON- INTELLIGENT ACCESS TO
HETEROGENEOUS AUDIOVISUAL
CONTENT:
• extracting high level semantic out of
syntactic and low level semantic information
• filter responses according to user’s up-dated
profile
• automated music editing
& retrieval by content
(CUIDADO)
10
Projects: The visual interface
Visualising a semantic net in 3D. IICM.
Other projects:
•stereographic visualisation and
analysis of astronomy data
(Cosmo.Lab)
3D Search : next generation
knowledge retrieval & analysis
 selects objects from ontologies
 3D visualisation
 A.I. for updated filtering
adjustments
•Visual Data Navigators or
"Collaboratories" scaleale to the
Web (Smartdoc)
•MPEG 7 / 21 multimedia
retrieval across mobile (UMTS),
TV and Web (Perseo)
11
Projects: Still Image Retrieval
Other Projects:
• Automatic retrieval of
video clips (CIWOS)
• real-time matching of
video and database
images
(LIVE@Web.Com)
Cobweb: Retrieval of
images - based on their
content (colour, texture,
similarity etc) using
algorithms and semantic
analysis.
• retrieval of textile /
fashion images by
emotive values (FoundIt)
http://www.imageQuery.com
12
Projects: Adding semantics to 3D
objects
Describing photos with RDF
- W3C Lafon / Boss
Other Projects:
• add a semantic layer to
a dig-lib to support 3D
image processing
(Project 2)
Project 1: combine 3D graphics in
industrial product catalogues with
semantic tags:
• the 3D model can be directly
manipulated via the tags
• it can interact smartly with other
3D models
• enhancement of a video
sequence captured on a
real scene with real-time
rendered 3D objects, incl
indexing and retrieval
(ENREVI)
13
Projects - Semantic Web technologies
Projects:
• application service
provision of semantic
tools for “deep Web”
multimedia content
(Project E-S)
Project S-E: progressing the
Semantic Web, to “bring
structure to the meaningful
content of Web pages, creating an
environment where software
agents roaming from page to page
can readily carry out
sophisticated tasks for users”.
Tim Berners-Lee, Scientific American.
• a spatially-aware
search engine that finds
documents and datasets
relating exactly or
approximately to places
or regions referred to in
a query (Project SP)
14
Main ingredients of the Semantic Web
• Widespread mark-up of digital resources (using
RDF, XML) by their semantics e.g. “this picture is
me”, not their labels, e.g. “this picture is JPG 99325687”.
• Develop flexible, interoperable ontologies for
sharing metadata across different archives e.g. to
allow comparisons, context, and classification of
existing and new terms.
• agent-based techniques and cogitive systems which
can act on the improved metadata structure to help
with user profiling and collaborative filtering.
• SemWeb services e.g. in e-commerce, information
retrieval etc.
15
Sixth Framework Programme:
Proposed structure
Trust &
Security
IST for societal
challenges
IST for economic
challenges
Demanding
applications
Applied IST for major societal and economic challenges
Anywhere anytime natural and
enjoyable access to IST services for ALL
Pervasive, mobile,
wireless, trustful
infrastructures
Communication
& networking
Software
Miniaturised,
low cost low power
components & µsytems
µ, nano & opto
electronics
µ and nano
systems
Natural interactions
with ‘ knowledge ’
Knowledge
technologies
interfaces
16
Sixth Framework Programme:
“Knowledge Technologies”
• Keywords
•Technologies:
•Context - based
•Semantic-based
•Agent based
•Scaleable
• Web resource description
• Content-based multimedia indexing
• Knowledge Enginering / AI / agents
• Natural Language processing (text / speech)
• Interactivity and visualisation
• machine learning
•Research challenges:
• usability of knowledge systems
• content as knowledge
• standards and interoperability
• Knowledge communities & portals
Http://www.cordis.lu/ist/
17