Interpreting the Temporal
Aspects of Language
Naushad UzZaman
Advisor: James Allen
Computer Science Department
University of Rochester, Rochester, NY
PhD Defense
20 July 2012
Motivation
Temporal Understanding
Temporal Evaluation
Summary
Research Context
PhD Defense
2
20 July 2012
Motivation
Temporal Understanding
Temporal Evaluation
Summary
Temporal Understanding
In 1492, Genoese explorer Christopher Columbus, under contract to the Spanish
crown, reached several Caribbean islands, making first contact with the indigenous
people. On April 2, 1513, Spanish conquistador Juan Ponce de León landed on what
he called "La Florida"—the first documented European arrival on what would become the
U.S. mainland. Spanish settlements in the region were followed by ones in the present-day
southwestern United States that drew thousands through Mexico. French fur traders
established outposts of New France around the Great Lakes; France eventually
claimed much of the North American interior, down to the Gulf of Mexico. The first
successful English settlements were the Virginia Colony in Jamestown in 1607 and the
Pilgrims' Plymouth Colony in 1620. The 1628 chartering of the Massachusetts Bay Colony
resulted in a wave of migration; by 1634, New England had been settled by some
10,000 Puritans. Between the late 1610s and the American Revolution, about 50,000
convicts were shipped to Britain's American colonies. Beginning in 1614, the Dutch settled
along the lower Hudson River, including New Amsterdam on Manhattan Island.
PhD Defense
3
20 July 2012
Motivation
Temporal Understanding
Temporal Evaluation
Summary
Temporal Structure
1492
April 2, 1513
Christopher Columbus
reached
Caribbean islands
Juan Ponce de León
landed
La Florida
documented
Spanish settlements
southwestern United States
thousands through Mexico
French fur traders
1607
are related with each other
first European arrival
landed
drew
France
• Understand how entities
established
claimed
outposts of New France
much of the North American interior
first successful English settlements
were
Virginia Colony in Jamestown
1610s
along the lower Hudson River,
1614
Dutch
settled
including New Amsterdam on Manhattan Island.
1620
first successful English settlements
1628
chartering of the Massachusetts Bay Colony
1634
10,000 Puritans
settled
were
Pilgrims' Plymouth Colony
resulted
migration
New England
American
Revolution
50,000 convicts
PhD Defense
shipped
Britain's American colonies
• We can:
‣ Summarize
‣ Answer questions
‣ Visualize
• Can benefit many domains
and applications
4
20 July 2012
Motivation
Temporal Understanding
Temporal Evaluation
Summary
Scenario - Medical Domain
• All clinical notes are becoming electronic
• Notes from different doctors visit
• Doctors can easily get answers like:
QA
‣ Did the patient get follow-up CT after PET/CT
‣ Did the patient had any abdominal pain earlier
PhD Defense
5
20 July 2012
Motivation
Temporal Understanding
Temporal Evaluation
Summary
Scenario - News domain
• Chronological summary of events in a news
‣ Helps busy people to skim through articles
‣ Helps people with problem in understanding
Summary
Iraq
war
Aug 2010 - Google launched voice
actions
2001 - September 11 incident
2003 - US invaded Iraq
Siri vs Oct 2011 - Apple released Siri with
Majel iPhone 4s
2006 - Saddam Hossain hanged
2008 - Obama became president
Dec 2011 - Google released Majel
to compete against Siri
2011 - Iraq War ended
PhD Defense
6
20 July 2012
Motivation
Temporal Understanding
Temporal Evaluation
Summary
Scenario - History domain
• Visualization to educate visitors in museum ‣ In front of Columbus section in museum
‣ Can check online wikipedia
‣ Automatically create the timeline http://upload.wikimedia.org/wikipedia/commons/1/1f/Antillas_%28orthographic_projection%29.svg
4/23/10 9:39 PM
reached
1492
Christopher Columbus
Caribbean islands
Page 1 of 1
landed
April 2, 1513
Juan Ponce de León
La Florida
first European arrival
documented
Visualization
Spanish settlements
drew
landed
southwestern United States
thousands through Mexico
http://upload.wikimedia.org/wikipedia/commons/8/8f/New_France_%28orthographic_projection%29.svg
4/23/10 10:18 PM
km
French fur traders
established
outposts of New France
France
much of the North American interior
Page 1 of 1
claimed
Llorens et al. 2011
1607
PhD Defense
7
first successful English settlements
were
Virginia Colony in Jamestown
20 July 2012
Motivation
Temporal Understanding
Temporal Evaluation
Summary
Temporal Information Understanding
• Applications: Summarization,Visualization,
Question Answering, and others
• Domains: news, medical, history, and others
PhD Defense
8
20 July 2012
Motivation
Temporal Understanding
Temporal Evaluation
Summary
Outline
• Temporal Information Understanding
• Evaluation of Temporal Information
• Summary and Future Work
PhD Defense
9
20 July 2012
Motivation
Temporal Understanding
Temporal Evaluation
Summary
Temporal Understanding in NL
The advisor came to the department at 11 am.
The student came before his advisor’s arrival.
The meeting began at 11:30am.
temporal
expression
event
relation
The student was productive in the meeting.
11:00am
camestudent
11:30am
cameadvisor
meeting1
arrival
meeting2
•TimeML annotation scheme
•TimeBank, AQUAINT corpus
•TempEval competition
productive
PhD Defense
10
20 July 2012
Motivation
Temporal Understanding
Temporal Evaluation
Summary
Our Approach
• Hybrid System
• Semantic information from semantic parser
• Machine learning classifier
PhD Defense
11
20 July 2012
Motivation
Temporal Understanding
Temporal Evaluation
Summary
Suggestions from Semantic Parser
Sentence
He fought in the war
(SPEECHACT V1 SA-TELL :CONTENT V2)
Semantic parser (F V2 (:* FIGHTING FIGHT) :AGENT V3 :MODS (V4) :TMA ((TENSE PAST)))
(PRO V3 (:* PERSON HE) :CONTEXT-REL HE)
output
(F V4 (:* SITUATED-IN IN) :OF V2 :VAL V5) (THE V5 (:* ACTION WAR))
ontology class
((THE ?x (? type SITUATION-ROOT))
100+
-extract-noms>
Extraction rules
(EVENT ?x (? type SITUATION-ROOT) :pos NOUN :class OCCURRENCE ))
Extracted with
extraction rules
<EVENT eid=V2 word=FIGHT pos=VERBAL ont-type=FIGHTING tense=PAST
class=OCCURRENCE voice=ACTIVE aspect=NONE polarity=POSITIVE>
<RLINK eventInstanceID=V2 ref-word=HE ref-ont-type=PERSON relType=AGENT>
<SLINK signal=IN eventInstanceID=V2 subordinatedEventInstance=V5
relType=SITUATED-IN>
<EVENT eid=V5 word=WAR pos=NOUN ont-type=ACTION class=OCCURRENCE
voice=ACTIVE polarity=POSITIVE aspect=NONE tense=NONE>
PhD Defense
12
20 July 2012
Motivation
Temporal Understanding
Temporal Evaluation
Summary
Temporal Tasks
event-timex
The advisor came to department at 11 am.
main-sub
The student came before his advisor’s arrival.
They had a meeting from 11:30 am.
main-main
The student was productive in the meeting.
11:00am
cameadvisor
meeting1
cameadvisor
meeting1
arrival
PhD Defense
meeting2
11:30am
productive
camestudent
meeting2
13
arrival
20 July 2012
Motivation
Temporal Understanding
Temporal Evaluation
Summary
Events and its Attributes
Event: something that happens or a dynamic property which
holds the truth. Attributes: class, tense, aspect and pos.
Class attribute:
1.
2.
3.
4.
5.
6.
7.
PhD Defense
Occurrence: die, crash, build
State: on board, alive
Reporting: say, report
I-Action: attempt, try, promise
I-State: believe, intend, want
Aspectual: begin, stop, continue
Perception: see, hear, watch, feel
14
20 July 2012
Motivation
Temporal Understanding
Temporal Evaluation
Summary
Event Extraction
• Semantic parser suggestions and features
• Train events and attributes on TimeML data
• Markov Logic Network (MLN) classifier
‣ first order logic + markov network
PhD Defense
15
20 July 2012
Motivation
Temporal Understanding
Temporal Evaluation
Summary
Event Extraction Features
Word-level Semantic Features
ontology type - WordNet class
pos
class
tense
voice
polarity
aspect
modality
penn tag
verb pos sequence
prev pos
next pos
pos x prev pos
Lexical Features
word
base word
constituent
verb word sequence
previous word of verb sequence
prev word
next word
word x prev word
index range
Sentence-level Semantic Features
Rlink relation type
Rlink reference ont-type
Slink core relation
Slink reference relation
Tlink reference
Tlink signal
PhD Defense
16
20 July 2012
Motivation
Temporal Understanding
Temporal Evaluation
Summary
Temporal Expression Extraction
• Recognition with Conditional Random Field
• Normalization with rule based system
Temporal expression
DCT (given):
March 1, 1998; 14:11 hours
Sunday
last week
mid afternoon
nearly two years
each month
Type
Value
TIME
DATE
DATE
TIME
DURATION
SET
1998-03-01T14:11:00
1998-03-01
1998-W08
1998-03-01TAF
P2Y
P1M
cw -2
cw - 1
current word
last
next
five
ten
months
years
init-list
is number
is time string
Table 1: Examples of normalized values and types for temporal expressions
according to TimeML
PhD Defense
17
20 July 2012
Motivation
Temporal Understanding
Temporal Evaluation
Summary
Overview
Text
TRIPS
Semantic
parser
Parser
MLN event
feature extractor
CRF timex
extractor
MLN event
event
filtering
extractor
events
timex
normalizer
event
features
timex
timex
features
Event and Temporal Expression extraction from raw text: first step towards a temporally aware system.
Naushad UzZaman and James F. Allen. International Journal of Semantic Computing, 2011.
PhD Defense
18
20 July 2012
Motivation
Temporal Understanding
Temporal Evaluation
Summary
Identifying Temporal Relations
event-timex
The advisor came to department at 11 am.
main-sub
The student came before his advisor’s arrival.
They had a meeting from 11:30 am.
main-main
The student was productive in the meeting.
11:00am
cameadvisor
meeting1
cameadvisor
meeting1
arrival
PhD Defense
meeting2
11:30am
productive
camestudent
meeting2
19
arrival
20 July 2012
Motivation
Temporal Understanding
Temporal Evaluation
Summary
Temporal Relations
simultaneous
identity
during
begins/
begun by
Y
before /
after
X
iBefore /
iAfter
X
overlaps/
overlapped by
PhD Defense
X
ends/
ended by
Y
Y
includes/
included by
X
X
Y
X
Y
X
Y
Y
20
20 July 2012
Motivation
Feature category
lexical
features
word-level
semantics
features
sentence-level
semantic
features
Source of Feature
Semantic parser
MLN classifier
Rule based
PhD Defense
Temporal Understanding
Features
Event Class
Event Tense
Event Aspect
Event Polarity
Event Stem
Event Word
Event Constituent
Event Ont-type
Event LexAspect x Tense
Event Pos
Timex Word
Timex Type
Timex Value
Timex DCT relation
Event’s semantic role
Event’s argument’s ont-type
TLINK event-time signal
SLINK event-event relation type
Temporal Evaluation
Task C event-dct
Task D
event-timex
YES
YES
YES
YES
YES
YES
YES
YES
YES
YES
YES
YES
YES
YES
YES
YES
YES
YES
YES
YES
YES
YES
YES
YES
YES
YES
YES
YES
YES
YES
YES
YES
Summary
Task E
main-main
e1 x e2
e1 x e2
e1 x e2
e1 x e2
e1 x e2
YES
e1 x e2
e1 x e2
e1 x e2
e1 x e2
Task
F
main-sub
e1 x e2
e1 x e2
e1 x e2
e1 x e2
e1 x e2
YES
e1 x e2
e1 x e2
e1 x e2
e1 x e2
YES
Table 1: Features used for Tasks C, D, E and F. 20 July 2012
21
Motivation
Temporal Understanding
Temporal Evaluation
Summary
Performance in TempEval-2
Task
Temporal Expression
Event
Event-Temporal Expression
Event-DCT
Consecutive Main Events
Main-Subordinate Events
Performance
0.85 (Fscore)
0.77 (Fscore)
0.65 (Precision)
0.79 (Precision)
0.56 (Precision)
0.6 (Precision)
Comparison
Second best
Third best
Best
Second Best
Best
Second Best
Table 1: Performance of TRIOS in TempEval-2
TRIPS and TRIOS System for TempEval-2: Extracting Temporal Information from Text.
Naushad UzZaman and James F. Allen. International Workshop on Semantic Evaluations
(SemEval-2010), Association for Computational Linguistics (ACL), Sweden, July 2010.
PhD Defense
22
20 July 2012
Motivation
Temporal Understanding
Temporal Evaluation
Summary
Creating Temporal Structure
11:00am
meeting2
11:30am
cameadvisor
meeting1
cameadvisor
meeting1
productive
meeting2
arrival
11:00am
camestudent
arrival
camestudent
11:30am
cameadvisor
meeting1
arrival
meeting2
productive
PhD Defense
23
20 July 2012
Motivation
Temporal Understanding
Temporal Evaluation
Summary
Creating Temporal Structure
• Temporal closure algorithm • Adopted from Timegraph ‣
‣
‣
‣
Miller and Schubert (1990) Scalable and fast Convert time intervals to time points Ignore inconsistent relations PhD Defense
24
20 July 2012
Motivation
Temporal Understanding
Temporal Evaluation
Summary
Timegraph Example
• A < B, B < C, D < B
• a <a <b <b
• b <b <c <c
• d <d <b <b
s
s
s
e
e
e
s
s
s
e
e
e
as
ae
bs
be
10
20
30
40
as
ae
bs
be
cs
ce
10
20
30
40
50
60
as
ae
bs
be
cs
ce
10
ds
20
de
30
40
50
60
10
20
chain1
chain1
chain1
chain2
PhD Defense
25
20 July 2012
Motivation
Temporal Understanding
Temporal Evaluation
Summary
Temporal Structure
11:00am
camestudent
11:30am
cameadvisor
meeting1
arrival
meeting2
productive
• Can infer how two entities are related
• Can answer temporal questions from a
document
PhD Defense
26
20 July 2012
Motivation
Temporal Understanding
Temporal Evaluation
Summary
Summary of Temporal Understanding
• Hybrid system
• Automatically extract events, temporal
expressions, identify temporal relations
• Competitive performance in TempEval-2
• Temporal Structure based on Timegraph for
temporal reasoning
PhD Defense
27
20 July 2012
Motivation
Temporal Understanding
Temporal Evaluation
Summary
Outline
• Temporal Information Understanding
• Evaluation of Temporal Information
• Summary and Future Work
PhD Defense
28
20 July 2012
Motivation
Temporal Understanding
Temporal Evaluation
Summary
Evaluation of Temporal Information
• Evaluation of Temporal Annotation
• Evaluation of Temporal Understanding
PhD Defense
29
20 July 2012
Motivation
Temporal Understanding
Temporal Evaluation
Summary
Problem with Current Metric
• Precision/recall from IR community
• Relation in different but equivalent ways
system
reference
e1
e2
e1
PhD Defense
e1
e3
e3
e3
30
20 July 2012
Motivation
Temporal Understanding
Temporal Evaluation
Summary
Related Work
• Semantic matching of temporal relations
‣ for judging inter-annotator agreement • Setzer et al. (2003)
‣ use temporal closure; inaccurate • Tannier and Muller (2008) ‣ use only core relations; approximation
PhD Defense
31
20 July 2012
Motivation
Temporal Understanding
Temporal Evaluation
Summary
Our Approach
• Closure to verify equivalent relations • Single score to measure the temporal
awareness of systems • Use temporal structure
Temporal Evaluation.
Naushad UzZaman and James F. Allen. Proc. of The 49th Annual Meeting of the Association for
Computational Linguistics: Human Language Technologies (Short Paper), USA, June 2011.
PhD Defense
32
20 July 2012
Motivation
Temporal Understanding
Temporal Evaluation
Summary
Precision, Recall, Fscore
Precision = (# of system temporal relations that can be verified from reference
annotation temporal closure graph / # of temporal relations in system output)
Recall = (# of reference temporal relations that can be verified from system
temporal closure graph / # of temporal relations in reference annotation)
a
Sys+ = System Closure Graph
Ref + = Reference Closure Graph
a
F score =
PhD Defense
a
P recision =
Recall =
|Sys\Ref + |
|Sys|
|Ref \Sys+ |
|Ref |
2⇤P recision⇤Recall
P recision+Recall
33
20 July 2012
Motivation
Temporal Understanding
Temporal Evaluation
Summary
Example
• All systems get 100% precision,
including S2 with B<D edge
• Similar to MUC-6 scoring for
coreference resolution
• They estimate the minimal number
of missing links to complete chain
to match human annotation
• In S1, S3, we are missing 1 edge,
hence 2/3 for recall is appropriate
PhD Defense
34
20 July 2012
Motivation
Temporal Understanding
Temporal Evaluation
Summary
Example
System Precision
PhD Defense
35
Recall
Fscore
S1
2/2=1
2/3=0.66
0.8
S2
2/2=1
1/3=0.33
0.5
S3
2/2=1
2/3=0.66
0.8
20 July 2012
Motivation
Temporal Understanding
Temporal Evaluation
Summary
Properties of our Metric
• Performance decreases with the decrease of
information
• If system fails to extract important entities the
performance will decrease more
• Implementation is scalable
PhD Defense
36
20 July 2012
Motivation
Temporal Understanding
Temporal Evaluation
Summary
Summary
• F score -- temporal awareness score
• F score captures entity & relation performance
• F score can help to find overall best system
• Updated solution to reward implicit relations
appropriately
PhD Defense
37
20 July 2012
Motivation
Temporal Understanding
Temporal Evaluation
Summary
Evaluate Temporal Understanding
• How annotation started?
• TimeML for Temporal QA
• Ideal case - evaluate Temporal QA?
PhD Defense
38
20 July 2012
Motivation
Temporal Understanding
Temporal Evaluation
Summary
Why Evaluate with Temporal QA
• Answering questions to judge understanding
• Creating questions easier than annotating
• Temporal IE performance might not reflect real
task performance
• Completeness of human annotations
• Comparing human and system annotations
PhD Defense
39
20 July 2012
Motivation
Temporal Understanding
Temporal Evaluation
Summary
Temporal Question Taxonomy
• yes/no: Was Fein called after the killing?
killing
captured
called
• list: What happened after the crash?
• list between: What happened between the
crash and today?
• factoid: When DT Inc. holders adopted a
shareholder-rights plan?
PhD Defense
40
during the crisis
20 July 2012
Motivation
Temporal Understanding
Temporal Evaluation
Summary
Temporal QA System
• Take TimeML annotation as input
• Temporal reasoning with Timegraph (Miller and
Schubert, 1990) - efficient
• Maintains necessary relations in graph to infer
relations between all entities
• Can answer yes/no, list, factoid
Evaluating Temporal Information Understanding with Temporal Question Answering.
Naushad UzZaman, Hector Llorens and James F. Allen. Proceedings of IEEE
International Conference on Semantic Computing, Italy, September 2012.
PhD Defense
41
20 July 2012
Motivation
Temporal Understanding
Temporal Evaluation
Summary
Temporal QA as Evaluation
• 189 questions (79 yes/no, 63 list, 47 factoid) from
25 TimeML documents (1442 tlinks)
• Evaluate how good are the system annotations
• Get answer from gold, then compare with system
• Compare against UzZaman and Allen (2011) metric
Systems
TIPSem
TRIOS
Yes/No
0.48
0.34
List
0.43
0.37
Factoid
0.53
0.22
IE-score
0.31
0.27
Table 1: Performance in Temporal QA and corpus based IE against annotation
PhD Defense
42
20 July 2012
Motivation
Temporal Understanding
Temporal Evaluation
Summary
Observations
• For yes/no: only 16.6% explicit, rest implicit
• Manually answered yes/no questions
Yes/No against human annotations
Yes/No against human answers
Gold
1.00
0.48
TIPSem
0.48
0.38
TRIOS
0.34
0.23
Table 1: Comparison of gold and system annotations against human answers
• Human annotation not 100% (48%) - annotation
coverage not complete
• Difference between human and system lower
PhD Defense
43
20 July 2012
Motivation
Temporal Understanding
Temporal Evaluation
Summary
Summary of Temporal QA
QA can better evaluate temporal
• Temporal
information understanding ‣ Natural task; easy to create questions ‣ Can evaluate human temporal annotations • Developed a system for temporal QA
‣ Can reason with any TimeML annotations ‣ Released the toolkit PhD Defense
44
20 July 2012
Motivation
Temporal Understanding
Temporal Evaluation
Summary
Outline
• Temporal Information Understanding
• Evaluation of Temporal Information
• Summary and Future Work
PhD Defense
45
20 July 2012
Motivation
Temporal Understanding
Temporal Evaluation
Summary
Summary
• System to extract temporal information
‣ Competitive performance at TempEval-2
• Metric to evaluate temporal annotation
‣ Adopted to evaluate participants at TempEval-3
• Temporal QA system with reasoning
‣ Can evaluate temporal understanding
• TRIOS-TimeBank, Merging, TempEval-3
PhD Defense
46
20 July 2012
Motivation
Temporal Understanding
Temporal Evaluation
Summary
Future Work
• Temporal Visualization or Timeline
• Temporal QA+summarization+visualization
• Future TempEval
PhD Defense
47
20 July 2012
Motivation
Temporal Understanding
Temporal Evaluation
Summary
Temporal Visualization or Timeline
http://upload.wikimedia.org/wikipedia/commons/1/1f/Antillas_%28orthographic_projection%29.svg
1492
April 2, 1513
Christopher Columbus
reached
Caribbean islands
Juan Ponce de León
landed
La Florida
documented
4/23/10 9:39 PM
reached
1492
Christopher Columbus
Caribbean islands
first European arrival
Page 1 of 1
landed
April 2, 1513
Spanish settlements
southwestern United States
landed
Juan Ponce de León
La Florida
first European arrival
documented
thousands through Mexico
drew
French fur traders
established
outposts of New France
Spanish settlements
France
claimed
landed
southwestern United States
much of the North American interior
drew
thousands through Mexico
http://upload.wikimedia.org/wikipedia/commons/8/8f/New_France_%28orthographic_projection%29.svg
1607
first successful English settlements
were
4/23/10 10:18 PM
Virginia Colony in Jamestown
1610s
along the lower Hudson River,
1614
Dutch
settled
km
French fur traders
including New Amsterdam on Manhattan Island.
established
outposts of New France
Page 1 of 1
1620
first successful English settlements
1628
chartering of the Massachusetts Bay Colony
were
Pilgrims' Plymouth Colony
resulted
France
10,000 Puritans
settled
much of the North American interior
migration
1607
1634
claimed
first successful English settlements
were
Virginia Colony in Jamestown
New England
American
Revolution
PhD Defense
50,000 convicts
shipped
Britain's American colonies
48
20 July 2012
Motivation
Temporal Understanding
Temporal Evaluation
Summary
Temporal QA+summarization+viz
• Temporal Summary and Visualization with QA
capability
• Can benefit: doctors, students, people with
reading disabilities
PhD Defense
49
20 July 2012
Motivation
Temporal Understanding
Temporal Evaluation
Summary
Future TempEval
• 1: given entities and attributes, classify relations
• 2: different subtasks to extract entities and
classify relations
• 3: given raw text, annotate all information
• Future: evaluate with temporal QA
PhD Defense
50
20 July 2012
Motivation
Temporal Understanding
Temporal Evaluation
Summary
Publications at URCS
Naushad UzZaman, Hector Llorens and James Allen. Evaluating Temporal Information Understanding with
Temporal Question Answering. Proceedings of IEEE International Conference on Semantic Computing, Italy,
September 2012.
1.Naushad UzZaman, Hector Llorens and James F. Allen. Merging Temporal Annotations. Proceedings of 19th International
Symposium on Temporal Representation and Reasoning, Leicester, UK, September 2012.
2.Naushad UzZaman and James F. Allen. Temporal Evaluation. Proc. of The 49th Annual Meeting of the
Association for Computational Linguistics: Human Language Technologies (Short Paper), Portland, Oregon,
USA, June 2011.
3.Naushad UzZaman and James F. Allen. Event and Temporal Expression extraction from raw text: first step
towards a temporally aware system. International Journal of Semantic Computing, 2011.
4.Naushad UzZaman, Jeffrey P. Bigham and James F. Allen. Multimodal Summarization of Complex Sentence. Proc. of
International Conference on Intelligent User Interfaces (IUI), Palo Alto, California, 2011.
5.Naushad UzZaman, Jeffrey P. Bigham and James F. Allen. Multimodal Summarization for people with cognitive disabilities in
reading, linguistic and verbal comprehension. 2010 Coleman Institute Conference, Denver, CO, 2010.
6.Naushad UzZaman and James F. Allen. Extracting Events and Temporal Expressions from Text. Fourth IEEE
International Conference on Semantic Computing (IEEE ICSC2010), Pittsburgh, USA, September 2010.
7.Naushad UzZaman and James F. Allen. TRIPS and TRIOS System for TempEval-2: Extracting Temporal
Information from Text. International Workshop on Semantic Evaluations (SemEval-2010), Association for
Computational Linguistics (ACL), Sweden, July 2010.
8.Naushad UzZaman and James F. Allen. TRIOS-TimeBank Corpus: Extended TimeBank corpus with help of Deep
Understanding of Text. Proceedings of The seventh international conference on Language Resources and Evaluation
(LREC), Malta, May 2010.
9.Naushad UzZaman, Jeffrey P. Bigham and James F. Allen. Pictorial Temporal Structure of Documents to Help People who
have Trouble Reading or Understanding. International Workshop on Design to Read, ACM Conference on Human Factors in
Computing Systems (CHI), Atlanta, GA, April 2010.
PhD Defense
51
20 July 2012
Acknowledgement
Collaborators
Advisor
James Allen
James Allen
Jeffrey Bigham
Committee
Hector Llorens
Jeffrey Bigham
James Pustejovsky
Greg Carlson
Marc Verhagen
Leon Derczynski
Daniel Gildea
Bosch RTC
Lenhart Schubert
Junling Hu
Chair
Microsoft Medical Lab
Jeffrey Runner
Michael Gillam
Hank Rappaport & PD Singh
And...
Yahoo! Research
Department staff
Ben, George, Mary, Will
Fall 2007 class
Bangladeshi students at UR
PhD Defense
Peter Mika
& My Family
52
Michael Matthews & Roi Blanco
20 July 2012
www.cs.rochester.edu/~naushad
naushad@cs.rochester.edu
PhD Defense
53
20 July 2012
Motivation
Temporal Understanding
Bigger Picture
Temporal Evaluation
Summary
Natural Language Processing
Natural Language Understanding
Applications
temporal information
understanding,
social
media information extraction,
question answering,
multimodal
summarization, game
car
prediction, dialog
system
Domains
news
medical
PhD Defense
54
20 July 2012
Motivation
Temporal Understanding
Temporal Evaluation
Summary
Next ...
• Mobility group of Nuance in Burlington, MA
• Siri-like applications for
‣ smart phones
‣ smart cars
‣ smart tv and others
55
Motivation
Temporal Understanding
Temporal Evaluation
Summary
Multimodal Summarization
In 1492, Genoese explorer Christopher Columbus, under contract to the Spanish crown,
reached several Caribbean islands, making first contact with the indigenous people.
Multimodal Summarization of Complex Sentence.
Naushad UzZaman, Jeffrey P. Bigham and James F. Allen. Proc. of International Conference on Intelligent User
Interfaces (IUI), Palo Alto, California, 2011.
Multimodal Summarization for people with Cognitive Disabilities in reading, linguistic and verbal comprehension.
Naushad UzZaman, Jeffrey P. Bigham and James F. Allen. 2010 Coleman Institute Conference, Denver, CO, 2010.
PhD Defense
56
20 July 2012
Motivation
Temporal Understanding
Temporal Evaluation
Summary
TwitterPaul: Retrieving, Ranking and
Aggregating Twitter Predictions
http://www.prefixmag.com/
rediff.com
57
Motivation
Temporal Understanding
Temporal Evaluation
Summary
Medical IE on Demand
RECORD #106886
Discharge Date: 3/17/2006
ATTENDING: HEISSER III , MONROE
JOSEPH MD
SERVICE: Cardiac Surgery Service.
PRINCIPAL DIAGNOSIS: Coronary
artery disease.
HISTORY OF PRESENT ILLNESS: Mr.
Midget is a 56-year-old male ..... ..... .....
He had parkinson's disease
since ..... ..... ..... ..... ..... ..... ..... ..... ..... ..... ..... ..
... ..... ..... ..... ..... ..... ..... ..... ..... ..... ..... ..... ..... ...
RECORD #165885
Discharge Date: 10/20/1994
Attending: DENNIS O. STROHSCHEIN ,
M.D. CU6
SERVICE: Cardiac Surgery Service.
PRINCIPAL DIAGNOSIS: Colon cancer
status post sigmoid colectomy with a
primary anastomosis
HISTORY OF PRESENT ILLNESS: Patient
is a 73 year old male who immigrated
from Co Garlpaaran approximately
fifteen years ago. He has had a several
year history of tuberculosis ..... ..... ..... .....
RECORD #193097
Discharge Date: 9/4/1998
Attending: FRITZ E. BROMBERG , M.D.
SF41 ED166/6793
PRINCIPAL DIAGNOSIS: 1. SEVERE
PANCREATIC EXOCRINE AND
ENDOCRINE INSUFFICIENCY.
HISTORY OF PRESENT ILLNESS:
Problem list:
....... ....... ....... ....... ....... ....... .......
....... ....... ....... ....... ....... ....... .......
8. CORONARY ARTERY DISEASE ,
STATUS POST SILENT IMI IN 1988.
....... ....... ....... ....... ....... ....... .......
1. In the HISTORY OF PRESENT ILLNESS section
2. UMLS semantic category Disease or Syndrome
58
Medical Media Lab
Motivation
Temporal Understanding
Temporal Evaluation
Summary
In-Car Dialog System
• Dialog assistant like iPhone’s Siri
• Navigation and MP3 domain
‣
‣
‣
‣
i want to go to dave's house
i would like a chinese restaurant
i want to go to gym on my way home
play music by scorpion, raise volume
• Developed syntactic and semantic grammar
59
Motivation
Temporal Understanding
Temporal Evaluation
Summary
Timegraph
•
Timegraph construction
‣
diamond 1
chain and pseudo time calculated when point
first entered into the Timegraph
‣
relation between two points in the same chain
is constant, by comparing pseudo-time
‣
relation between two points in different chain
is found by searching in cross-chain links
‣
search in cross-chain links depends on # of
chain + # of cross-chain links
‣
goal in Timegraph construction: minimize # of
chain and # of cross-chains
PhD Defense
60
(a)
diamond 2
circle1
square1
circle 2
square 2
circle 3
circle 4
square 3
Chain 1
(circles)
(b)
Chain 2
(squares)
Chain 3
(diamonds)
Examples: (a) timegraph (each node represents a time
point); (b) metagraph (each node represents a chain). (
)
in-chain links; (--->) cross-chain links.
20 July 2012
Motivation
Temporal Understanding
Temporal Evaluation
Summary
Problem with ACL 2010
<
K
A
<
B
<
C
<
D
<
<
S1
A
<
B
C
D
<
S2
A
<
B
C
A
<
B
S4
A
<
B
C
<
<
C
D
D
<
<
S5
A
<
B
<
C
D
<
<
<
S6
A
<
B
C
<
D
<
<
<
S7
A
<
B
<
C
Both S1 and S2 gets 100%
precision, 1/3 recall
•
•
•
•
S2 extracts extra implicit
D
<
S3
•
<
No extra reward for implicit
But S4 > S2 > S1
S3 = S4, but S2 > S1
D
<
<
PhD Defense
61
20 July 2012
Motivation
Temporal Understanding
Temporal Evaluation
Summary
Metrics for Temporal Evaluation
Evaluation Metric
TempEval-2
Setzer et al.
Tannier and Muller
Our ACL’11 metric
Our Updated metric
1
where, w =
PhD Defense
Recall
Precision
Ref \Sys
Ref
+
Ref \Sys+
Ref +
Ref \Sys
Ref
|Ref \Sys+ |
|Ref |
+
|Ref \Sys |+w⇤|(Sys
Ref )\Ref + | 1
|Ref |
Sys\Ref
Sys
+
Sys \Ref +
Sys+
Sys \Ref
Sys
|Sys \Ref + |
|Sys |
|Sys \Ref + |
|Sys |
0.99
(1+|#Ref + | |Ref \Sys+ |)
62
20 July 2012
Motivation
Temporal Understanding
Temporal Evaluation
Summary
Improvement
|Sys \ Ref + |
P recision =
|Sys |
Recall =
|Ref
* G = Reduced Graph of G,
which includes only the core relations
\ Sys+ | + w ⇤ |(Sys
Ref ) \ (Ref +
|Ref |
Ref )|
a
* (Ref \ Sys+ ) captures system explicit relations that are found in gold explicit relations
* (Sys
Ref ) captures implicit system relations in terms of Ref
* (Ref + Ref ) captures implicit gold relations
* (Sys
Ref )\(Ref + Ref ) captures implicit system relations that exists
in gold implicit relations
PhD Defense
63
20 July 2012
Motivation
Temporal Understanding
Temporal Evaluation
Summary
Improvement
|Sys \ Ref + |
P recision =
|Sys |
Recall =
|Ref
Recall =
|Ref
* G = Reduced Graph of G,
which includes only the core relations
\ Sys+ | + w ⇤ |(Sys
Ref ) \ (Ref +
|Ref |
\ Sys+ | + w ⇤ |(Sys
|Ref |
Ref )|
Ref ) \ Ref + |
a
* (Sys
Ref ) \ Ref + can be easily computed efficiently by checking, Sys
relations that can not be found in Ref but can be verified in Timegraph of
Ref
a
\ ) \ (Ref + \ Ref
\) =
(Sys
Ref ) \ (Ref + Ref ) = (Sys \ Ref
\ \ Ref + ) \ (Sys \ Ref
\ \ Ref
\ ) = (Sys \ Ref
\ \ Ref + ) =
(Sys \ Ref
(Sys
Ref ) \ Ref + = (Ref + Ref ) \ Sys
PhD Defense
64
20 July 2012
Motivation
Temporal Understanding
Temporal Evaluation
Summary
How it solves the problem
<
K
A
<
B
<
C
<
D
<
<
S1
A
<
B
C
D
<
S2
A
<
B
C
D
<
S3
A
<
B
S4
A
<
B
C
<
<
C
D
D
<
<
S5
A
<
B
<
C
D
<
<
system
S1
S2
S3
S4
S5
S6
S7
precision
1/1=1
2/2=1
2/2=1
2/2=1
3/3=1
4/4=1
3/3=1
recall
(1+0.3*0)/3=0.333
(1+0.3*1)/3=0.433
(2+0.3*0)/3=0.666
(2+0.3*0)/3=0.666
(2+0.3*1)/3=0.766
(2+0.3*2)/3=0.866
(3+0.3*0)/3=1.000
fscore
0.5000
0.6043
0.8000
0.8000
0.8674
0.9281
1.0000
<
S6
A
<
B
C
<
D
<
D
<
Table 1: Updated precision, recall and fscore for systems
<
<
S7
A
<
B
<
C
<
<
PhD Defense
65
20 July 2012