Collaborative Annotation of the
www.amiproject.org
AMI Meeting Corpus
Jean Carletta
University of Edinburgh
www.amiproject.org
AMI Partners
Carletta
20 June 2007
2
www.amiproject.org
NXT Major Development
Sites
Carletta
20 June 2007
3
www.amiproject.org
AMI's aim
• aim: to develop technologies for
browsing meetings and to assist people
during meetings
• interdisciplinary: signal processing,
language engineering, theoretical
linguistics, human-computer interfaces,
organizational psychology, ...
Carletta
20 June 2007
4
Why annotation?
www.amiproject.org
• For basic scientific understanding - e.g.,
• How do people choose a next speaker?
• What is the relationship between speech
and gesture during deixis?
• For machine learning
• Hand-code e.g. statement vs. question
• Identify features for each like word
sequences and prosody
• Use the data to fit a statistical classifier that
codes new data automatically
Carletta
20 June 2007
5
www.amiproject.org
Carletta
20 June 2007
6
www.amiproject.org
Carletta
20 June 2007
7
www.amiproject.org
AMI Meeting Rooms
4 close- and 2 wide-view cameras, 4 head-set
and 8 array microphones, presentation screen
capture, whiteboard capture, pen devices, plus
extra site-dependent devices
TNO
Edinburgh
Carletta
IDIAP
20 June 2007
8
www.amiproject.org
IS1004d, 3:07 - 4:11
Carletta
20 June 2007
9
www.amiproject.org
Corpus Overview
• 100 hrs of well-recorded meetings
• orthographically transcribed with
word timings by forced alignment
• ASR output
• heavily annotated by hand for
communicative behaviours
• Creative Commons Share-Alike
licensing, with demo DVD
Carletta
20 June 2007
10
www.amiproject.org
Hand Annotations
• transcription with word-level timings
from forced alignment (100%)
• timestamping against signal (10-30%)
• head gestures; hand gestures for
addressing and interactions with objects;
location in room; gaze; emotion?
• discourse structure (70%)
• dialogue acts (some w/ addressing), named
entities, topic segments, linked extractive
and abstractive summaries
Carletta
20 June 2007
11
www.amiproject.org
Costs in person-hrs/hr
transcription
topic segments + abstractive summaries
dialogue acts w/ some relations
addressing
extractive summaries linked to abstract
named entities
hand gestures (rough timings)
head gestures (rough timings)
head gestures (precision timings)
movement around room
Carletta
20 June 2007
30
6-10
20
12
1
2-5
6
6
20
4
12
www.amiproject.org
Core Problems
• How do we represent all of these
kinds of annotation on the same
base data, including both structural
relationships and timing?
• How do we allow for multiple
(human and machine) annotations
of the same property, so that we
can compare them?
Carletta
20 June 2007
13
www.amiproject.org
Carletta
20 June 2007
14
www.amiproject.org
Carletta
20 June 2007
15
NITE XML Toolkit
www.amiproject.org
•
•
Mature toolkit for handling annotations with
temporal ordering and full structural relations
Data storage format designed to support
distributed corpus development
• Libraries for data handling, query, and writing
graphical user interfaces
• End user annotation tools for common tasks
• Command line utilities for analysis, feature
extraction
•
Open source
Carletta
20 June 2007
16
www.amiproject.org
NXT corpus design
• data model is multi-rooted tree with arbitrary
graph structure over the top
• each node has one set of children, multiple parents
• annotations often naturally map to a tree
• corpus design to decide where trees intersect
• NXT can represent arbitrary graphs but the
more the data has this character, the less
useful the query language is
Carletta
20 June 2007
17
Stand-off XML
www.amiproject.org
extract from Bdb001.A.speech-quality.xml
<speechquality nite:id="Bdb001.emphasis.16" type="emphasis">
<nite:child href="Bdb001.A.words.xml#id(Bdb001.w.1,342)..id(Bdb001.w.1,344)" />
</speechquality>
extract from Bdb001.A.words.xml
<w nite:id="Bdb001.w.1,342" starttime="356.39" endtime="" c="W">time</w>
<w nite:id="Bdb001.w.1,343" starttime="" endtime="" c="HYPH">-</w>
<w nite:id="Bdb001.w.1,344" starttime="" endtime="356.59" c="W">line</w>
Carletta
20 June 2007
18
www.amiproject.org
Metadata file
Like set of DTDs for the XML files
plus:
• connections between the files
• list of "observations" (coded
dialogues/group discussions/texts)
• catalog for finding signals and data
on disk
Carletta
20 June 2007
19
www.amiproject.org
Simple example query
($w word)($r reference):
($w@POS = “NN”) && ($r ^ $w)
Return list of 2-tuples of words and
referring expressions where the word’s
part of speech is NN and the word is in
the referring expression.
Carletta
20 June 2007
20
General features of the
language
www.amiproject.org
• Match variable by no type, single type, or
disjunctive type
• Attribute and content tests for existence,
ordering, equality, match to regexp
• The usual boolean combinators
• Quantifiers forall and exists
• Filtering by passing results to another query
to create a result tree (not list)
Carletta
20 June 2007
21
www.amiproject.org
Uses for queries
•
•
•
•
•
Carletta
Exploring the data in a browser
Basic frequency counts
Verifying data quality
Indexing complexes for further use
Finding things for screen rendering
in GUI
20 June 2007
22
www.amiproject.org
Only configuration
needed to:
• search/index data in NXT format
• display data in a standardized
(ugly) way
• Set up annotation tools for some
common tasks
• dialogue act
• named entity
• time-stamped labelling
Carletta
20 June 2007
23
www.amiproject.org
• [named entity demo]
Carletta
20 June 2007
24
www.amiproject.org
Programming
tailored interfaces
• development time is 1.5 days - 2
weeks depending on
• how clear the spec is
• complexity of the interface and
whether our "transcription view"
middleware fits
• familiarity with Swing
Carletta
20 June 2007
25
www.amiproject.org
Named entity coder
Carletta
20 June 2007
26
www.amiproject.org
Carletta
20 June 2007
27
www.amiproject.org
Carletta
20 June 2007
28
www.amiproject.org
Carletta
20 June 2007
29
www.amiproject.org
Carletta
20 June 2007
30
www.amiproject.org
Carletta
20 June 2007
31
www.amiproject.org
Carletta
20 June 2007
32
www.amiproject.org
Carletta
20 June 2007
33
www.amiproject.org
Carletta
20 June 2007
34
www.amiproject.org
Summary
• NXT provides infrastructure for
collaborative annotation that
•
•
•
•
Is distributed
Provides structural relationships
Provides timing w.r.t signals
Works for large-scale projects
• NXT’s best current demonstration
is in the AMI Meeting Corpus
Carletta
20 June 2007
35