Adding semantics to multimedia European projects Imagina 2002 Franco Mastroddi European Commission DG Information Society 1 ‘Mega-trends (ISTAG expert Group)’ ICT today “Ambient Intelligence” tomorrow • Micro scale…………………… Nano-scale •PC based …………………… Information devices - many embedded.. • “Writing and reading”…….…. Use all senses, touch, vision, hearing... • “Word” based metadata…… Content- and context-based systems • Low bandwidth, separate networks. Infinite bandwidth, cross-media ... • Mobile telephony (voice)…….. Mobile/Wireless full multimedia • eServices just emerging……… Wide adoption (eHealth, eLearning...) •Only 5% of global population on-line.. >70% of world-wide population on line 2 Multimedia demographics have changed In early 1990s the typical Web user was an American male student Today many users are ‘normal’ consumers (Insight Express) female users overtaken males in US (Nielsen) worldwide, more than 65 million youths (5-17) have Internet access at home or school (Datamonitor). Web is used increasingly for trade and professional services, ecommerce etc vast new opportunities… broader market economies of scale 3 New forms of interactivity from scribe to screen. Over 90 % of text / data is digital origin. Soon also for audio-visual. From linear to non-linear content management from mass broadcast to mass personalisation from single media to cross-media. from static to dynamic multimedia resources. Self-managing digital objects, links, embedded semantic metadata / ontologies. from geographic branding to virtual community-building from disintermediation to re-mediation through new content middleware systems. 4 Is there too much content? Example of consumer photographs annual output: billions of photos / year. Present storage method: albums & shoeboxes “people want to do more than print a few pictures and store the rest on their hard disk” but the average consumer is not a good indexer… Professionals do not fare much better... “1,5 hours to scan 42 photos...but 43 hours to index them…” (Washington State 2000 Diglib project) 5 Is there enough technology? Increasing processing power Data crunching < > content management with new telecomms networks mobile (UMTS still in the future - 4% global penetration by 2004 - UMTS Forum) Bluetooth, ambient systems and new a/v devices over 17 mio digi-cams shipped in US in 2001 (Hallmark Card Store) can integrate into flexible content platforms e.g. digital camera as front-end of an integrated home image system via wireless/USB to PC or settop-box - thence to Internet. 6 Does the Web really help? The Web paradigm does not necessarily help in multimedia searching... “Size of the database, frequency of update, Which one is the criminal? search capability and design, and speed may lead to amazingly different results” (Kansas City Public Library) “search engines are undergoing a profound evolution ... such as... ways of automatically classifying information” (B. Grossan, Lawrence Berkely Library) “a very small number of commercially available products exist which perform content-based image retrieval.” (Manchester Visualisation Centre, 2001) Attrasoft 7 Towards a Semantic Middleware. Human Human Machine-Machine Multi-modal / visual interfaces Information Agents Semantic middleware : RDF, Ontologies, Semantic-based systems and services. Documents Databases Email Web People Processes 8 Content description is the key Create Host Connect Use The linear content chain: Source: Gemini Consulting The non-linear chain: 9 Projects: Automated multimedia content analysis Developing new search algorithms to work on MPEG 4/7 metadata. Other projects: • personalised views of a broadcast archive (Primavera) • advert analysis from TV and Web (Aramis) FAETHON- INTELLIGENT ACCESS TO HETEROGENEOUS AUDIOVISUAL CONTENT: • extracting high level semantic out of syntactic and low level semantic information • filter responses according to user’s up-dated profile • automated music editing & retrieval by content (CUIDADO) 10 Projects: The visual interface Visualising a semantic net in 3D. IICM. Other projects: •stereographic visualisation and analysis of astronomy data (Cosmo.Lab) 3D Search : next generation knowledge retrieval & analysis selects objects from ontologies 3D visualisation A.I. for updated filtering adjustments •Visual Data Navigators or "Collaboratories" scaleale to the Web (Smartdoc) •MPEG 7 / 21 multimedia retrieval across mobile (UMTS), TV and Web (Perseo) 11 Projects: Still Image Retrieval Other Projects: • Automatic retrieval of video clips (CIWOS) • real-time matching of video and database images (LIVE@Web.Com) Cobweb: Retrieval of images - based on their content (colour, texture, similarity etc) using algorithms and semantic analysis. • retrieval of textile / fashion images by emotive values (FoundIt) http://www.imageQuery.com 12 Projects: Adding semantics to 3D objects Describing photos with RDF - W3C Lafon / Boss Other Projects: • add a semantic layer to a dig-lib to support 3D image processing (Project 2) Project 1: combine 3D graphics in industrial product catalogues with semantic tags: • the 3D model can be directly manipulated via the tags • it can interact smartly with other 3D models • enhancement of a video sequence captured on a real scene with real-time rendered 3D objects, incl indexing and retrieval (ENREVI) 13 Projects - Semantic Web technologies Projects: • application service provision of semantic tools for “deep Web” multimedia content (Project E-S) Project S-E: progressing the Semantic Web, to “bring structure to the meaningful content of Web pages, creating an environment where software agents roaming from page to page can readily carry out sophisticated tasks for users”. Tim Berners-Lee, Scientific American. • a spatially-aware search engine that finds documents and datasets relating exactly or approximately to places or regions referred to in a query (Project SP) 14 Main ingredients of the Semantic Web • Widespread mark-up of digital resources (using RDF, XML) by their semantics e.g. “this picture is me”, not their labels, e.g. “this picture is JPG 99325687”. • Develop flexible, interoperable ontologies for sharing metadata across different archives e.g. to allow comparisons, context, and classification of existing and new terms. • agent-based techniques and cogitive systems which can act on the improved metadata structure to help with user profiling and collaborative filtering. • SemWeb services e.g. in e-commerce, information retrieval etc. 15 Sixth Framework Programme: Proposed structure Trust & Security IST for societal challenges IST for economic challenges Demanding applications Applied IST for major societal and economic challenges Anywhere anytime natural and enjoyable access to IST services for ALL Pervasive, mobile, wireless, trustful infrastructures Communication & networking Software Miniaturised, low cost low power components & µsytems µ, nano & opto electronics µ and nano systems Natural interactions with ‘ knowledge ’ Knowledge technologies interfaces 16 Sixth Framework Programme: “Knowledge Technologies” • Keywords •Technologies: •Context - based •Semantic-based •Agent based •Scaleable • Web resource description • Content-based multimedia indexing • Knowledge Enginering / AI / agents • Natural Language processing (text / speech) • Interactivity and visualisation • machine learning •Research challenges: • usability of knowledge systems • content as knowledge • standards and interoperability • Knowledge communities & portals Http://www.cordis.lu/ist/ 17