Nuria Oliver, PhD
Telefonica Research
Multimedia and Data Mining & User Modeling
Scientific Director

Information Created, Captured and Replicated
6-Fold Growth in Four Years
2006
161 Exabytes
2010
988 Exabytes
Source: IDC, 2007

Photo
All words ever spoken by human beings (5 exa)
Print Collection
US Library of Congress
Book
Movie
All printed material
(multimedia)

2010 Data
988 Exabytes
563 Exabytes
User*
Generated
Content
692 Exabytes
Organizational**
Touch
Content
296 Exabytes
*Consumers and Workers creating, capturing or replicating personal informaton
**Transported, hosted managed or secured

Search &
Discovery
Media
Consume
Media
Produce
Media, tags, ratings, comments

A FEW RESEARCH
CHALLENGES AND
OPPORTUNITIES

Multi-modal approaches are needed to construct novel methodologies to fuse multimodal content and context information
Multi-modal Multimedia
Content Analysis:
Feature Extraction
Similarity Metrics
Ontology definition
Indexing schemes….
Multimedia Context:
Higher level knowledge generated by users
(tags, comments…)
User interaction data
Wisdom of the crowd
Fusion of content and context-based features
Creation of large collections of labeled training data
Noise filtering by aggregation of contextual information
Improved search results

Paris France
Vacation
Eiffel Tower
June 2009

• User generated content is rarely annotated  really difficult, if not impossible to later find it
• When annotated, it is typically done in batch, per session, not per item
• Tags significantly improve search results alone or combined with content-based techniques
• Need for novel interfaces to encourage users to annotate content
– Games with a Purpose
– Annotations at the time of capture
• More research on tag expansion and automatic tagging

• Retrieval accuracy is not sufficient due to vast amounts of available information  too many relevant results
• New orthogonal dimensions need to be used to extend the notion of relevance and improve retrieval performance  e.g., aesthetics
• User generated content is of varying quality
• Need for user centric approaches
• Multi-disciplinary approaches:
– Computer scientists, psychologists, human-computer interaction researchers
– Computer Vision, Pattern Recognition, Machine Learning,
Human Perception, Human Activity Recognition

Video and Audio Feature extraction
Signature Creation
Duplicate Detection
How do users perceive nearduplicate videos?
Do they care about them?
Which features are important when defining near-duplicates?
M. Cherubini, R. de Oliveira, and N. Oliver, “Understanding near-duplicate videos,” in
Proceedings of ACM MM’09, (Beijing, China), pp. 35–44, ACM Press, October 19-24 2009.

“The interest that a photograph, video or audio piece generates when perceived by human observers, and that incorporates both objective and subjective factors”

“Paris Louvre Night”

• User generated content has a wide range of quality and
aesthetic value for the same content
• Aesthetics influence our perception of content
• Highly disregarded in state-of-the-art multimedia retrieval systems
• Need for computational models of the aesthetic value of multimedia content
• Need for ground truth databases on image, audio and video aesthetics
• Need for deeper understanding of
– The role of aesthetics on user preferences and satisfaction
– Universal vs personal aesthetics
– Domain-dependent aesthetics
– Quality vs aesthetics

Personalization, Recommendation and
Exploratory Search

Personalization, Recommendation and
Exploratory Search
• Future multimedia search and retrieval systems will need to take into account the user’s preferences, interests and task at hand in order to return relevant content
• Huge amounts of multimedia data
– Need for recommendations rather than direct search
– Automatic discovery of relevant information to the users
• More research should be devoted to user modeling, personalization and recommendations of multimedia content
• Untapped research challenge: Role that the task at hand plays in determining the optimal multimedia content to retrieve for the user

“The conveying of events with words, images and sounds, often with embellishement. “
Stories or narratives have been shared in every culture and in every land as a means of entertainment , education, preservation of culture and in order to instill moral values.
Crucial elements of stories and storytelling include plot and characters , as well as the narrative point of view .

• Despite capturing large amounts of digital multimedia content, most users rarely access the content again
• Sharing the multimedia content is one of the main reasons why users capture it
• Lack of efficient and scalable tools for browsing, finding and selecting the desired content
• Multimedia Storytelling: User-friendly, semi-automatic and scalable (space and time) multimedia tools that enable users to
– Easily retrieve desired multimedia content
– Create and share the story they want to create from their content

Exemplary Workflow for MM Storytelling
Multimedia
Analysis Tools
Face Detection
Face Recognition
Smile Detection
Aesthetics Reranking
Clustering
Near-duplicate Detection
Tag expansion
Automatic Classification
Flickr e-mail communication
With story slideshow
Create Story Slideshow:
“Madrid Christmas 2009”
Storytelling
UI
Storytelling
Algorithms
Identify main Actors
Identify main Chapters
Complement content with external content
Select images based on
*Story Length
*Target Audience
*Target Device

• Users are increasingly seeking new ways to experience multimedia content
• Research opportunities combining:
– Music + Video: High-quality visual musical experiences
– Video + 3D reconstruction: 3D video
– Images + Context : Mobile Augmented Reality
• Research challenges in:
– Multimedia analysis: Machine learning, pattern recognition, computer vision
– User Modeling
– Human-computer interaction

http://research.tid.es/multimedia nuriao@tid.es