The FAME Project
• Acronym: Facilitating Agent for Multicultural Exchange
• Partners:
Universität Karlsruhe,
UJF Grenoble,
UPC Bacelona,
ATLAS Barcelona
• Project volume:
• Duration:
• More info:
INPG Grenoble,
ITC-irst Trento,
SONY Europe, Stuttgart
5.5 M Euro
40 months, started October 2001
http://www.fame-project.org
Seminar „Multimodale Räume“
Uni Karlsruhe, 14.5.2003
1
The FAME Projekt
Facilitating Agent for Multicultural Exchange
• Volume:
• Duration: 40 months (since October 2001)
• Financial Volume:  5,5 Mio. €
• currently approx. 30 scientists
www.fame-project.org
• Partners:
Uni Karlsruhe
UPC Barcelona
, INPG Grenoble
, ITC-irst Trento
Stuttgart,
, UJF Grenoble,
,
Barcelona
Seminar „Multimodale Räume“
Uni Karlsruhe, 14.5.2003
2
Project Goals
• long term vision: facilitate communication between humans
• reduce the workload on the users of technical equipment
• observe humans and their activities in an intelligent room
and serve as a context-aware information butler
• FAME project goal:
provide and integrate core technologies (video and speech
perception, augmented reality, translation, information retrieval)
to show feasibility of the concept
• demonstrate system at fair
• scenario 1 (lecture scenario):
one person is giving a talk or lecture or presentation
• scenario 2 (meeting scenario):
several people are discussing / working on a common task
Seminar „Multimodale Räume“
Uni Karlsruhe, 14.5.2003
3
The FAME Showcases
scenario 1 (presentation)
• use A/V equipment
• intelligent cameraman
• presentation tracking
• summarisation + archiving
• translation, crosslingual IR
scenario 2 (meeting)
• augmented reality
• video-based activity tracking
• topic spotting
• information butler
• service: planning of fair visit
Seminar „Multimodale Räume“
Uni Karlsruhe, 14.5.2003
4
The FAME Demonstrator
(at Barcelona Fair „Forum of Cultures“ 2004)
FAME outside view
meeting inside
people mention topics
Seminar „Multimodale Räume“
Uni Karlsruhe, 14.5.2003
reception by FAME-guy
room gives information
about spotted topics
5
The FAME Demonstrator
(at Barcelona Fair „Forum of Cultures“ 2004)
at the phicon wall
gestures
multimodal input on table
interactive visit planning
output also on the wall
Seminar „Multimodale Räume“
Uni Karlsruhe, 14.5.2003
the projection table
borrow a camera for
photographs of the visit
6
The FAME Demonstrator
(at Barcelona Fair „Forum of Cultures“ 2004)
Back from the visit in the
FAME room: dowload ...
record testimony
... and look at photos
select, print, save photos
using phicon interaction
intelligent cameraman,
presentation tracker
take home photos and
information about FAME
Seminar „Multimodale Räume“
Uni Karlsruhe, 14.5.2003
7
Important Components
• multimodal environment
• context-aware intelligent camera-man
automatically track people and their activities
• augmented reality environment
move physical icons (phicons) on table/wall,
and interact with projection on table/wall
• spontaneous speech recognition (with distant microphones)
• translation and crosslingual information retrieval
in European-English, Catalan, and Spanish
• dialog and context model
Seminar „Multimodale Räume“
Uni Karlsruhe, 14.5.2003
8
Multimodal Environment at UKA
Smartboard as
Projection Wall
Microphon-Array
(Speaker Lokalization)
Livingroomg
Audio Signals
IR-Remote Control
X-10
Illumination
Loudspeakers
Microphones
Several Beamers
4 Cameras
TV/Video
Seminar „Multimodale Räume“
Uni Karlsruhe, 14.5.2003
9
Augmented Reality Table
• project virtual reality on real table
• move around physical icons (multiple users)
• interact with projection
• select, move, rotate, resize, delete, change color
• write on table, pass notes to others, point to items
Seminar „Multimodale Räume“
Uni Karlsruhe, 14.5.2003
10
Intelligent Camera Man
• follow speaker while
talking and moving around
• detect interaction from audience
• zoom on area of interest
e.g. when pointing somewhere or showing something
Seminar „Multimodale Räume“
Uni Karlsruhe, 14.5.2003
11
Lecture
Supporter
• track lecture or presentation
• operate FAME room equipment
by speech commands
• automatically switch slides during presentation
• automatically create transcript of lecture
• create summary, translate to other languages
• record and store all lectures in searchable database
• retrieve and browse through previously recorded lectures
Seminar „Multimodale Räume“
Uni Karlsruhe, 14.5.2003
12
estimate
72 classes
P(Cn|wn-1,wn-2)
· P(wn|Cn)
add to classes
Adaptation Overview
most frequent
40k words
least frequent
20k words
20% fewer
errors
60k vocabulary
HUB-4 corpus
trigrammmodel
P(wn|wn-1,wn-2)
tf-idf
wichtige
wichtige
wichtige
important
Wörter
Wörter
Wörter
words
presentationslides
±2 contexts
100
Links
100
Links
100
links
scores
top n
CLASS 32
CLASS 14
CLASS 57
CLASS 6
CLASS 70
perplexity
TO THE RECOGNITION OF THE
CONTINUOUS SPEECH RECOGNITION IN NOISY
AND PATTERN RECOGNITION NOT IN
SECURING DUE RECOGNITION AND RESPECT
Seminar „Multimodale Räume“
Uni Karlsruhe, 14.5.2003
13
Welcome in Barcelona
in Summer 2004
Seminar „Multimodale Räume“
Uni Karlsruhe, 14.5.2003
14