PHAROS
1st year results
Francesco S. Nucci,
Vilamoura NAV Concertation meeting
16th April 2008
Partners and project key data
Academia:
Industry:
• Engineering, Spa
• Fast Search & Transfer ASA
• Sail Labs Technology AG
• Metaware, Spa
• Webmodels, Srl
• L3S Research Centre
• Fraunhofer IDMT
• EPF Lausanne
• Open University, KMI
• University Pompeu Fabra
• VTT Research Centre
Content and
Service Providers:
• France Telecom
• Circom Regional
13 partners from 9 countries!
Timescale and budget:
• Start January 2007
• Duration 36 months
• Total budget: €14.2m
• EC contribution: €8.5m
The search problem: state of the art

What is missing in state-of-the-art systems?

Flexible architecture across content and queries

Content analysis:



Flexible architecture which extends to support new
information stores, processing steps and content types
Scalable in traffic and volume of content as well as
diversity and freshness of content
Easy integration of latest content analysis technology
The search problem: example queries


“Find video shots where Sarkozy is speaking about…”

Combines shot detection and speech analysis

Uses structural information about shot boundaries and semantic
information about identified speakers
“Find videos with music similar to this one, with scenes of
sunsets”

Combines music analysis, shot detection and image classification

Uses content-based music features, structural information about
shot boundaries and information from image classification
First year results



Designed and developed SOA-compliant, open and
distributed technology platform for development of
arbitrary information access solutions for AV content.
Scalable, pluggable, and separable frameworks for

Content analysis

Search
Platform for innovation

Framework for integration of technologies

Quickly adapt to new user and application scenarios

Transform ideas into real solutions

Evaluate with real users - on the way
Why a Platform approach?

Need to have pluggable approach for core components

Content Analysis
 Search
 Query and Result Analysis
Mixed approach



Bottom up - 24 components representing the SOA step
forward for each specific technology
Top down – systematic approach for processes and data flow
control
“a Service Oriented Architecture have been designed to
tackle the needs from the different deployment
scenarios”
Annotation Library

Design and implement a multimedia annotation library
prototype compliant with a state-of-the-art rich multimedia
representation framework

We defined a Multimedia annotation library

We designed Specification of multimedia representation
Status of the annotators
Media Type
Annotator
Provider
Image / Video
Image Annotator
EPFL
Image / Video
Image Annotator
KMI
Image / Video
Object detector
EPFL
Video
Shot detector
KMI
Audio
Speech recognition
SAIL
Audio
Music description
UPF
Audio
Music segmentation
FHR
Audio
Audio fingerprinting
FHR
Audio
Audio Segmentator
VTT
Visual component

Annotate (EPFL)
sand, sea, blue sky, cloudy sky,
sunset sky, mountains,
ground, vegetation, grass,
architecture, flowers, and
snow.
PHAROS Publication Subscription Protocol



This protocol allows the distribution of content
descriptions among PHAROS systems and PHAROS
platform components.
This protocol also allows the development of new
content access services for monitoring the availability
of new content.
The AV-RSS XML schema is an extension of the
simple well-known and very used RSS schema (RSS
stands
for
Really
Simple
Syndication)
and
complementary to media RSS defined by Yahoo.
The main extensions are:

The definition of multiple identifiers on multiple registries,
including fingerprints

Language and scheme attributes for item, group and content

Links to external descriptions


Presentation permissions (including acquisition, licensing
and copyright information) which can e.g. be bound to user
groups
Content availability scheduling
Mediator Audiovisual
Search Engine
Distributed
PHAROS Systems
Broadcaster
Search Engine
AV RSS
Broadcaster
Broadcaster
B2B
B2B Exchange
Content sharing
The picture



The picture illustrates the case of two Broadcasters services for
professional content exchange or for regional general public
access, which are publishing descriptions of the content they
managed.
Those descriptions are received by a more general search
service targeting a larger public and also integrating other
sources of content accessible from other web sites for example.
This service allows users to discover contents managed by
several content or service providers by using one single service
(the one at the top of the figure). This simplification is currently
appreciated by all users of RSS feed readers syndicating
information from different sources.
Next steps:



First experiments and validation of this schema have been
done, however a more extensive validation still needs
to be done.
We also started the dissemination of this protocol and this
is planned to be continued, in order to receive the feedback
of the audio visual Community.
All this will be considered in order to produce an improved
version of the protocol by the end of the year 2008.
Action Plan: Road show




1st workshop: presentation of the PHAROS vision, objectives
and the first outcomes
(M24, end of 2008)
Final event: international event to present the PHAROS final
results
(M36, end of 2009)
Summer school (2009)
PHAROS focused workshop: local events offering training
sessions and targeted workshop to SMEs and young researchers
(3rd year)
Next years planning

Second year: Technology Transfer of the first concrete results
with the involvement of external federation members and
researchers



First PHAROS Search Engine Implementation Week
with final open workshop
Specific training session with federation involvement
Final year: consolidated results transfer in the search engine
communities, the PHAROS Summer school

Target industrial groups, venture capitalists, international
researchers (India and Japan), students, other projects,
ETPs (NESSI, NEM, eMobility)
Thank you ... and
see you in Faro, not the town ...
The project !!