EU research initiatives in
multimodal interaction
Mats Ljungqvist
European Commission
Information Society DG,
Interfaces (Unit E1)
Joint AMI/Pascal/IM2/M4/ workshop, June 21-23 2004, Martigny
Outline of Presentation
•
•
•
Introduction
Who can participate?
“The story so far….”
• FP5 activities
• Overview FP6 call 1, 2 and 3
• Call 1: Multimodal Interfaces
• Call 2: Cognitive Systems
• Call 3: FET - Future and Emerging Technologies
•
The IST workprogramme for 2005-2006
•
• Calls 4 and 5
Preparation of the 7th EU Framework Programme
2007-2011
Joint AMI/Pascal/IM2/M4/ workshop, June 21-23 2004, Martigny
IST in FP6
(~4 Billion Euro,
2003-2006)
Knowledge
Communication Components
&
&
&
Interface
Micro-Systems
Computing
Technologies
Infrastructures
Applied IST Research addressing
Major Societal & Economic Challenges
Joint AMI/Pascal/IM2/M4/ workshop, June 21-23 2004, Martigny
Who can participate?
• All member states
• States associated to FP6: same conditions as
member states
– Bulgaria, Romania, Turkey, Iceland, Israel,
Liechtenstein, Norway, Switzerland
• "INCO countries“: specific earmarked funding budget
– Russia, New Independent States, Mediterranean
Countries, Western Balkans, Developing countries
• Countries with co-operation agreement participate
without EU funding (exceptionally with funding)
– Argentina, Australia, Brazil, Canada, China, Chile,
India, Japan, Kazakhstan, Russia South-Africa,
Ukraine, USA
• Other countries participate without funding
(exceptionally with funding).
Joint AMI/Pascal/IM2/M4/ workshop, June 21-23 2004, Martigny
Who can participate?
•
Another possibility
• Join a project as unfunded “external member”
sometimes possible according to modalities set up
by the consortium.
• Examples in some language resources projects
(China, Canada, USA, …).
•
Examples of funded and unfunded
participation (from 1st FP6 MMI call)
• Russia, USA, Senegal, Romania, Turkey, ….
Joint AMI/Pascal/IM2/M4/ workshop, June 21-23 2004, Martigny
Cognition and Interfaces
• Building on activities in earlier framework
programmes (…,ESPRIT, Telematics, FP5)
• Human Language Technologies
•
•
•
•
•
•
•
– speech, multilinguality, translation
Multimodal, multisensorial dialogue
Signal processing and coding
Machine vision
Affective computing
Robotics
Disappearing computer
Neuroinformatics, presence research
• Focus in FP6:
• Multimodal Interfaces (MMI)
• Cognitive Systems (CS)
• Beyond robotics, life-like perception systems, Bioinspired Intelligent Information Systems (in FET)
Interface related work also in FP6 application areas
Joint AMI/Pascal/IM2/M4/ workshop, June 21-23 2004, Martigny
FP5 projects
•
Projects in FP5 are laying the foundations for FP6 initiatives in the area:
•
•
•
KERMIT
Kernel Methods for Images and Text
M4
Nespole!
MultiModal Meeting Manager
•
FAME
Facilitating Agent for Multicultural Exchange
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
PF_STAR
Preparing Future Multisensorial Interaction Research
SIRIDUS
Specification, Interaction and Reconfiguration In Dialogue Understanding Systems
ALERT
Alert System for Selective Dissemination of Multimedia Information
CORETEX
Improving Core Speech Recognition Technology
VISATEC
Vision-based Integrated Systems Adaptive to Task and Environment with Cognitive abilities
ECVISION
European Research Network for Cognitive AI-enabled Computer Vision System
LAVA
Learning for Adaptable Visual Assistants
LC_STAR
Lexica and Corpora for Speech-to-Speech Translation Technologies
SPEECON
Speech Driven Interfaces for Consumer Applications
SPEECHDAT Speech Databases for Creation of Voice Driven Teleservices
ORIENTEL
Multilingual access to interactive communication services for the Mediterranean and the Middle East
NECA
A Net Environment for Embodied Emotional Conversational Agents
HUMODAN
An automatic human model animation environment for augmented reality interaction
SMARTSKETCHES A multimodal approach to improve usability in the early stages of product design
FET initiatives
…and more at…
www.cordis.lu/ist -->projectsJoint AMI/Pascal/IM2/M4/ workshop, June 21-23 2004, Martigny
Negotiating through SPOken Language in E-commerce
Call 1 overview
Multimodal Interfaces (MMI)
Develop natural and adaptive multimodal interfaces,
that respond intelligently to speech and language,
vision, gesture, haptics and other senses
• Interaction between and among human and the virtual
and physical environment
• Multilingual systems
•
Results
• 14 proposals selected for negotiations
• Most contracts signed December 2003
•
Funding
• 73 Meuro
Joint AMI/Pascal/IM2/M4/ workshop, June 21-23 2004, Martigny
MMI call results:
NoE (Networks of Excellence)
•
SIMILAR - The European taskforce creating human-machine
interfaces SIMILAR to human-human communication
• multimodal interface research, multiple sensory modalities, fusion of modalities:
visual interaction, interactive speech, interactive haptics, learning, and human
human-computer interaction (HCI)
•
HUMAINE - Human-Machine Interaction Network on Emotion
• study of systems that can register, model and/or influence human emotional
and emotion emotion-related states - central to future interfaces.
•
ENACTIVE - Enactive Interfaces
• interfaces and interaction paradigms based on Enactive Knowledge, i.e.,
knowledge stored in the form of motor responses and acquired by the act of
"doing” - a form of cognition inherently tied to actions
•
PASCAL - Pattern Analysis, Statistical Modelling and
Computational Learning
• pattern analysis, statistical modelling, and computational learning as core
enabling technology for discovering, structuring and presenting complex
information, or for processing of complex sensory data.
Academic research in key enabling technologies
Joint AMI/Pascal/IM2/M4/ workshop, June 21-23 2004, Martigny
MMI call results:
IP (Integrated Projects)
•
CHIL - Computers In the Human Interaction Loop
•
AMI - Augmented Multi-party Interaction
•
TC-STAR - Technology and Corpora for Speech to Speech
Translation
•
•
•
create environments in which computers serve humans who focus on interacting
with other humans
new multimodal technologies to support human interaction, in the context of
smart meeting rooms and remote meeting assistants.
advanced research in all core technologies for Speech-to-Speech-Translation:
speech recognition, translation, and synthesis.
- supporting human-human interaction
- perception in multiple modalities
Joint AMI/Pascal/IM2/M4/ workshop, June 21-23 2004, Martigny
STREP
•
MMI call results:
(Strategic Targeted Research Projects)
MATRIS -Markerless Real-time Tracking for Augmented Reality
Image Synthesis
• a system for markeless tracking, by analysis of the camera images and
exploitation of unobtrusive inertial motion sensors. This will enable the system
as a whole to determine its location and orientation mimicking the way a human
orients himself.
•
DIVINES - Diagnostic and Intrinsic Variabilities in Natural Speech
• improving speech recognition performance by research into feature extraction
and modelling techniques that would have better capacities, particularly in
handling speech intrinsic variabilities
•
TALK - Talk and Look, Tools for Ambient Linguistic Knowledge
• to advance our understanding of generic technologies that will extend the
Information State Update" (ISU) approach to adaptive multimodal and multi
multi-lingual interaction for developing adaptive multimodal dialogue systems.
•
T’nD - Touch and Design
• shape generation and modification based on novel haptic interaction and
intelligent shape manipulation operators in order to exploit existing manual skill
of designers
- small focussed efforts
- bootstrapping new research areas
Joint AMI/Pascal/IM2/M4/ workshop, June 21-23 2004, Martigny
MMI call results:
STREP, SSA (Specific Support Action)
•
TAI-CHI - Tangible Acoustic Interfaces for Computer-Human
Interaction
• development of tangible interfaces that employ physical objects and space as
media to bridge the gap between the virtual and physical worlds by exploring
how physical objects, augmented surfaces and space can be transformed into
tangible acoustic embodiments of natural seamless unrestricted interfaces.
•
HIWIRE - Human Input That Works In Real Environments
• embedded robust multimodal dialogue systems with flexible speech input in
mobile, open and noisy environments (e.g. aeroplanes, cars, street etc.).
•
MWEB - Multimodal Web Interaction
• support for the development and adoption of industry standards (W3C
Recommendations) enabling multimodal Web access using mobile devices.
Joint AMI/Pascal/IM2/M4/ workshop, June 21-23 2004, Martigny
Call 2 overview
Cognitive Systems
Construct physically instantiated or embodied systems that can
perceive (‘see’, ‘hear’, ’smell’), understand the semantics of
information conveyed through their perceptual input (‘recognise’,
‘categorise’) and interact with their environment (‘decide’,
‘act’), and evolve in order to achieve human-like performance in
activities requiring context specific knowledge.
• Artificial systems that combine perception, action
reasoning, learning and communication
• Provide enabling technologies for robotics, natural language
understanding, man-machine interaction, complex real
world systems
•
Results
• Negotiations with 8 projects concluded
• Most contracts to be signed in July-August 2004
•
Funding
• Around 25 Meuro
Joint AMI/Pascal/IM2/M4/ workshop, June 21-23 2004, Martigny
What is needed?
•
Cognitive Systems
• Extensive foundational work of exploratory nature
• “Cover ground” (as we don’t really know yet what is important)
–
The nature of cognition, knowledge, the role of language
–
Architectures
–
Perception, learning
–
Autonomy, notion of self, social interaction
• Interdisciplinary partnerships
•
Multimodal Interfaces
• Significant advances in key areas:
• “Emotional interfaces”
• Interfaces that are anticipatory, learning, evolving and
emerging: the “Do what I mean” paradigm
• Seamless control of IT enabled intelligent environment
• Fusion/synchronisation of multiple modalities
• Real-time, better than human interpretation of everyday
language
Joint AMI/Pascal/IM2/M4/ workshop, June 21-23 2004, Martigny
IST in FP6:
Work Programme 2005-2006
•
Evolution of running Strategic Objectives
• Calls in Dec 2004 or 2005
• Multimodal interfaces call in Dec 2004 or in 2005 ~50 m Euro?
• Cognitive systems ~50 m Euro?
• Applications and Services for the Mobile User & Worker
• Open Web Consultation and Consultation Meetings in April 2004
• Reports will be made available www.cordis.lu/ist
•
IST-FET: New Research Initiatives
• Bio-inspired Intelligent Information Systems
• ~20 m Euro?, Call closes on 22 Sept
• Presence and Interaction in Mixed Reality Environments
• ~20 m Euro?, Call in Dec 2004?
• Creating and Moulding Ambience
• Call in 2005?
•
www.cordis.lu/ist
New IST Work programme: autumn 2004
Joint AMI/Pascal/IM2/M4/ workshop, June 21-23 2004, Martigny
The Roadmap to
FP7
6th
FP
2003
2004
2005
2006
2007 – 2011 ...
7th
FP
Feb ’04
Financial Perspectives
(2007-2013)
Early ’05
Proposal
on FP
Adoption FP
12 May ’04
Preparing the future:
reinforcing European Research Policies
Joint AMI/Pascal/IM2/M4/ workshop, June 21-23 2004, Martigny
FP7: 6+2 axes
•
•
•
•
•
Collaboration
Collaborative Research
Private/public partnership
Technological platforms
Individual research teams
Basic Research Grants
Human resources
Mobility and Life Long Learning
Research Infrastructures
Access/Networking & support of new
Infrastructures
•
Co-ordination of national &
European Research Area
regional research programmes
and policies
+
•
•
Space (ESA)
Security
ICT research maps into all 6+2 axes
Joint AMI/Pascal/IM2/M4/ workshop, June 21-23 2004, Martigny
Contacts
• Next calls
• Call 3 now open (closes 22 September)
• Call 4 in late 2004, call 5 in 2005
• Based upon 2005-06 Work Programme (Oct/Nov
2004)
• Contact
• www.cordis.lu/ist/so/interfaces
• mats.ljungqvist@cec.eu.int
IST helpdesk
Fax : +32 2 296 83 88
e-mail : ist@cec.eu.int
Joint AMI/Pascal/IM2/M4/ workshop, June 21-23 2004, Martigny