E-Team 11: Multimodal Processing and
Multimedia Understanding
Leader: P. Maragos
Contents:



Video Analysis and Integration of Asynchronous Time-evolving Modalities
Multimodal Salient Event Detection
Integrated Multimedia Content Analysis
Participants:









P. Maragos, G. Evangelopoulos, K. Rapantzikos, S. Kollias (NTUA)
Patrick Gros, Manolis Delakis (INRIA-Texmex)
Costas Kotropoulos, N. Nikolaidis, I. Pitas (AUTH)
Andreas Rauber (TU Wien)
Alex Potamianos (TUC)
Sanni Siltanen (VTT)
Fred Stentiford, Wole Oyekoya (UCL)
Enis Cetin (Bilkent)
Cyril Horry (Univ. Surrey) ?
The List of all Active Participants will be updated and finalized by the next
MUSCLE plenary meeting (Dec. 6-8, 2006).
Related Links:


http://cvsp.cs.ntua.gr/projects/muscle
http://cvsp.cs.ntua.gr/projects/bin/view/MuscleWP6/ETeams
Research Agenda:
Research areas include:





Stochastic modeling with several data streams / several temporal rates /
weakly synchronized data
Audio-Visual Cooperative Feature Extraction and Salient Event Detection
Audio-visual Dialogue Understanding
Image + Text Integration
Audio + Text integration
Application areas include:

Understand (= structure) TV and other MM documents, and Prepare these
documents for applications (repurposing, archiving)




Event Detection and Segmentation in Sports videos.
Salient Event Detection and Dialogue Detection in Movies videos.
Speech Transcription and NLP.
Feature extraction both from Audio (music) as well as from Text (particulalry
song lyrics) to see how the two can be combined for genre analysis and music
retrieval.
Emerging Focus Themes:



Theme 1: Stochastic modeling with several data streams / several temporal
rates / weakly synchronized data and Multimodal Feature Fusion.
Theme 2: Salient Feature & Event Detection in Movies or Sports Videos.
Theme 3: Integrate/Associate Text (and possibly NLP) with other Modalities,
e.g. Speech, Music, Image.
Note: For each focus theme to continue existing and be meaningful for collaboration,
there must be at least two MUSCLE participants that are actively interested in and/or
have been doing research on it.
Goals:
The main goals of e-team10 are outlined below:




Goal 1: Contribute to the Update of the State-of-Art Surveys of the WP5
MUSCLE Book.
Goal 2: Co-Author New Research Chapters of the WP5 MUSCLE Book.
Goal 3: Co-author conference and journal Papers on some focus theme with
multiple MUSCLE partners (improve integration).
Goal 4: Collaboration on a common research agenda for multimodal feature
fusion, saliency detection and multimodal processing.
Recent research activity includes collaboration between TUC, NTUA and INRIATexmex on multimodal feature fusion. AUTH has been creating an annotated movie
database that will be distributed to the e-team partners to work on saliency detection
and event detection. The partners have also collaborated in creating the MUSCLE
book outline and have committed to updating the State-of-Art srveys and providing
new chapters for the MUSCLE book. The e-team holds conference calls, special
sessions at MUSCLE scientific workshops, and collaborates and presents work in
MUSCLE scientific meetings.
Ongoing and future e-team work includes: contributions of the partners on the
updating of the state-of-the-art and authoring of new chapters (contributions by
NTUA, TUC, AUTH, INRIA-TexMex, TUWien, Bilkent). Research collaboration for
the creation of a common task for saliency detection and event detection AUTH
movie database is currently underway. Two evaluation tasks will be set up and results
will be reported on the database.
Budget:
The total requested budget is 29,000 euros. See attached excel sheet for distribution to
partners.