CS 4705
1


Data as input (database, software trace, expert system), text summary as output

Text as input (one or more articles), paragraph summary as output

Multimedia in input or output

Summaries must convey maximal information in minimal space
2


Karl Malone scored 39 points Friday night as the Utah Jazz defeated the
Boston Celtics 118-94.

Karl Malone tied a season high with 39 points Friday night….
 … the Utah Jazz handed the Boston
Celtics their sixth straight home defeat
118-94.
Streak, Jacques Robin, 1993
3


Linguistic summarization : How to pack in as much information as possible in as short an amount of space as possible?

Streak: Jacques Robin

MAGIC: James Shaw

PLanDoc: Karen Kukich, James Shaw, Rebecca
Passonneau, Hongyan Jing, Vasilis Hatzivassiloglou

Conceptual summarization : What information should be included in the summary?
4

score (Jazz, 118) score (Celtics, 94)
The Utah Jazz beat the
Celtics 118 - 94.
points (Malone, 39) Karl Malone scored 39 points location(game,
Boston)
#homedefeats(Celtics, 6)
It was a home game for the Celtics
It was the 6th straight home defeat
5

beat hand
Jazz Celtics
Jazz defeat Celtics
6

S U M M O N S Q U E R Y O U T P U T
:
S u m m a r y
W e d n e s d a y , A p r i l 1 9 , 1 9 9 5 , C N N r e p o r t e d t h a t a n e x p l o s i o n s h o o k a g o v e r n m e n t b u i l d i n g i n
O k l a h o m a C i t y . R e u t e r s a n n o u n c e d t h a t a t l e a s t 1 8 p e o p l e w e r e k i l l e d . A t 1 P M , R e u t e r s a n n o u n c e d t h a t t h r e e m a l e s o f M i d d l e E a s t e r n o r i g i n w e r e p o s s i b l y r e s p o n s i b l e f o r t h e b l a s t . T w o d a y s l a t e r ,
T i m o t h y M c V e i g h , 2 7 , w a s a r r e s t e d a s a s u s p e c t ,
U . S . a t t o r n e y g e n e r a l J a n e t R e n o s a i d . A s o f M a y
2 9 , 1 9 9 5 , t h e n u m b e r o f v i c t i m s w a s 1 6 6 .
I m a g e ( s ) :
1 ( o k f e d 1 . g i f ) ( W e b S e e k )
A r t i c l e ( s ) :
( 1 ) B l a s t h i t s O k l a h o m a C i t y b u i l d i n g
( 3 ) A t l e a s t 1 8 k i l l e d i n b o m b b l a s t - C N N
( 5 ) W A S H I N G T O N ( R e u t e r ) - A s u s p e c t i n t h e O k l a h o m a C i t y b o m b i n g
( 2 ) S u s p e c t s ' t r u c k s a i d r e n t e d f r o m D a l l a s
( 4 ) D E T R O I T ( R e u t e r ) - A f e d e r a l j u d g e
M o n d a y o r d e r e d J a m e s
Summons, Dragomir Radev, 1995
7

Transitional

Automatically summarize series of articles

Input = templates from information extraction

Merge information of interest to the user from multiple sources

Show how perception changes over time

Highlight agreement and contradictions

Conceptual summarization : planning operators

Refinement (number of victims)

Addition (Later template contains perpetrator)
8


4 input articles parsed by information extraction system

4 sets of templates produced as output

Content planner uses planning operators to identify similarities and trends

Refinement (Later template reports new # victims)

New template constructed and passed to sentence generator
9

Message ID
TST-COL-0001
10


Input: one or more text documents

Output: paragraph length summary

Sentence extraction is the standard method

Using features such as key words, sentence position in document, cue phrases

Identify sentences within documents that are salient

Extract and string sentences together

Luhn – 1950s

Hovy and Lin 1990s

Schiffman 2000

Machine learning for extraction

Corpus of document/summary pairs

Learn the features that best determine important sentences

Kupiec 1995: Summarization of scientific articles
11


Shallow analysis instead of information extraction

Extraction of phrases rather than sentences

Generation from surface representations in place of semantics
12


Extraneous phrases
 “The five were apprehended along Interstate 95, heading south in vehicles containing an array of gear including …
...
authorities said.”

Dangling noun phrases and pronouns
 “The five”

Misleading

Why would the media use this specific word
(fundamentalists), so often with relation to Muslims?
*Most of them are radical Baptists, Lutheran and
Presbyterian groups .
13


Humans also reuse the input text to produce summaries

But they “cut and paste” the input rather than simply extract
 our automatic corpus analysis

300 summaries, 1,642 sentences

81% sentences were constructed by cutting and pasting
 linguistic studies
14


(1) Sentence reduction
~~~~~~~~~~~~
15


(1) Sentence reduction
~~~~~~~~~~~~
16


(1) Sentence reduction
~~~~~~~~~~~~

(2) Sentence Combination
~ ~~~ ~~~
~~ ~~ ~ ~ ~
~~ ~~~ ~
17


(3) Syntactic Transformation
~~~ ~~ ~~ ~~~

(4) Lexical paraphrasing
~~~~~~~~~~~
~~~
18


News: Newsblaster, GALE

Email

Meetings

Journal articles

Open-ended question-answering

What is a Loya Jurga?

Who is Mohammed Naeem Noor Khan?

What do people think of welfare reform?
19


News

Single Document

Multi-document

Email

Meetings

Journal articles

Open-ended question-answering

What is a Loya Jurga?

Who is Al Sadr?

What do people think of welfare reform?
20

Cut and Paste Based Single Document
Summarization -- System Architecture
Input: single document
Parser
Co-reference
Extraction
Extracted sentences
Generation
Sentence reduction
Sentence combination
Corpus
Decomposition
Lexicon
Output: summary
21


Input:
 a human-written summary sentence
 the original document

Decomposition analyzes how the summary sentence was constructed

The need for decomposition
 provide training and testing data for studying cut and paste operations
22

Summary sentence:
Arthur B. Sackler, vice president for law and public policy of Time
Warner Cable Inc. and a member of the direct marketing association told the Communications
Subcommittee of the
Senate Commerce
Committee that legislation to protect children’s privacy on-line could destroy the spondtaneous nature that makes the
Document sentences:
S1: A proposed new law that would require web publishers to obtain parental consent before collecting personal information from children could destroy the spontaneous nature that makes the internet unique, a member of the
Direct Marketing Association told a Senate panel Thursday.
S2: Arthur B. Sackler, vice president for law and public policy of Time Warner Cable Inc., said the association supported efforts to protect children on-line, but he…
S3: “For example, a child’s e-mail address is necessary …,” Sackler said in testimony to the
Communications subcommittee of the Senate
Commerce Committee.
S5: The subcommittee is considering the
Children’s Online Privacy Act, which was drafted…
23
Internet unique.

Document sentences:
Summary sentence:
S1: A proposed new law that would require
president for law and
before collecting personal information from children could destroy the spontaneous nature that makes the internet unique, a member of the
Warner Cable Inc.
and a
Direct Marketing Association told a Senate member of the direct panel Thursday.
marketing association told S2: Arthur B. Sackler, vice president for law the Communications and public policy of Time Warner Cable Inc., said the association supported efforts to protect
Subcommittee of the
Senate Commerce
Committee that legislation to protect children’s children on-line, but he…
S3: “For example, a child’s e-mail address is necessary …,” Sackler said in testimony to the
Communications subcommittee of the Senate privacy on-line could Commerce Committee.
destroy the spondtaneous nature that makes the
S5: The subcommittee is considering the
Children’s Online Privacy Act, which was drafted…
24
Internet unique.

Arthur B. Sackler, vice
Document sentences:
web publishers to obtain parental consent before collecting personal information from children could destroy the spontaneous nature public policy of Time that makes the internet unique, a member of the
Warner Cable Inc.
and a
Direct Marketing Association told a Senate member of the direct panel Thursday.
marketing association told S2: Arthur B. Sackler, vice president for law the Communications and public policy of Time Warner Cable Inc., said the association supported efforts to protect
Subcommittee of the
Senate Commerce
Committee that legislation to protect children’s children on-line, but he…
S3: “For example, a child’s e-mail address is necessary …,” Sackler said in testimony to the
Communications subcommittee of the Senate privacy on-line could Commerce Committee.
destroy the spondtaneous nature that makes the
S5: The subcommittee is considering the
Children’s Online Privacy Act, which was drafted…
25
Internet unique.

Summary sentence:
public policy of Time
Warner Cable Inc.
and a member of the direct marketing association told the Communications
Subcommittee of the
Senate Commerce
Committee that legislation to protect children’s privacy on-line could destroy the spondtaneous nature that makes the
Internet unique.
Document sentences:
S1: A proposed new law that would require web publishers to obtain parental consent before collecting personal information from children could destroy the spontaneous nature that makes the internet unique , a member of the
Direct Marketing Association told a Senate panel Thursday.
S2: Arthur B. Sackler, vice president for law and public policy of Time Warner Cable Inc., said the association supported efforts to protect children on-line, but he…
S3: “For example, a child’s e-mail address is necessary …,” Sackler said in testimony to the
Communications subcommittee of the Senate
Commerce Committee .
S5: The subcommittee is considering the
Children’s Online Privacy Act, which was drafted…
26


A Hidden Markov Model based solution

Evaluations:

Human judgements

50 summaries, 305 sentences

93.8% of the sentences were decomposed correctly

Summary sentence alignment

Tested in a legal domain

Details in (Jing&McKeown-SIGIR99)
27


An example:
Original Sentence: When it arrives sometime next year in new TV sets, the V-chip will give parents a new and potentially revolutionary device to block out programs they don’t want their children to see.
Reduction Program: The V-chip will give parents a new and potentially revolutionary device to block out programs they don’t want their children to see.
Professional: The V-chip will give parents a device to block out programs they don’t want their children to see.
28


Preprocess: syntactic parsing

Step 1: Use linguistic knowledge to decide what phrases MUST NOT be removed

Step 2: Determine what phrases are most important in the local context

Step 3: Compute the probabilities of humans removing a certain type of phrase

Step 4: Make the final decision
29


Syntactic knowledge from a large-scale, reusable lexicon convince: meaning 1: NPPP :PVAL (“of”)
(E.g., “He convinced me of his innocence”)
NP-TO-INF-OC
(E.g., “He convinced me to go to the party”) meaning 2: ...

Required syntactic arguments are not removed
30

 Saudi Arabia on Tuesday decided to sign…

The official Saudi Press Agency reported that King Fahd made the decision during a cabinet meeting in Riyadh, the Saudi capital.
 The meeting was called in response to … the
Saudi foreign minister, that the Kingdom…

An account of the Cabinet discussions and decisions at the meeting…

The agency...
31

 Saudi Arabia on Tuesday decided to sign…

The official Saudi Press Agency reported that King Fahd made the decision during a cabinet meeting in Riyadh, the Saudi capital.
 The meeting was called in response to … the
Saudi foreign minister, that the Kingdom…

An account of the Cabinet discussions and decisions at the meeting…

The agency...
32

 Saudi Arabia on Tuesday decided to sign…

The official Saudi Press Agency reported that King Fahd made the decision during a cabinet meeting in Riyadh, the Saudi capital.
 The meeting was called in response to … the
Saudi foreign minister, that the Kingdom…

An account of the Cabinet discussions and decisions at the meeting…

The agency...
33

verb
(will give) vsubc
(when) subj
(V-chip) iobj
(parents) obj
(device) ndet
(a) adjp
(and)
Prob(“when_clause is removed”| “v=give”) lconj
(new)
Prob (“to_infinitive modifier is removed” | “n=device”) rconj
(revolutionary)
34

verb L Cn Pr
(will give) vsubc
(when)
L Cn Pr subj L Cn Pr
(V-chip) iobj
L Cn Pr
(parents)
L Cn Pr
(device)
L -- linguistic
Cn -- context
Pr -- probabilities
L Cn Pr ndet
(a) adjp
(and)
L Cn Pr lconj
(new)
L Cn Pr rconj
(revolutionary)
L Cn Pr
35


Success rate: 81.3%

500 sentences reduced by humans

Baseline: 43.2% (remove all the clauses, prepositional phrases, toinfinitives,…)

Reduction rate: 32.7%

Professionals: 41.8%

Details in (Jing-ANLP00)
36


Monitor variety of online information sources

News, multilingual

Email

Gather information on events across source and time

Same day, multiple sources

Across time

Summarize

Highlighting similarities, new information, different perspectives, user specified interests in real-time
37


Use a hybrid of statistical and linguistic knowledge

Statistical analysis of multiple documents

Identify important new, contradictory information

Information fusion and rule-driven content selection

Generation of summary sentences

By re-using phrases

Automatic editing/rewriting summary
38

http://newsblaster.cs.columbia.edu/

Clustering articles into events

Categorization by broad topic

Multi-document summarization

Generation of summary sentences

Fusion

Editing of references
39

Crawl News Sites
Form Clusters
Categorize
Title
Clusters
Summary
Router
Event
Summary
Biography
Summary
Multi-
Event
Convert Output to HTML
Select
Images
40

41

42


Common information identification

Alignment of constituents in parsed theme sentences: only some subtrees match

Bottom-up local multi-sequence alignment

Similarity depends on
 Word/paraphrase similarity
 Tree structure similarity

Fusion lattice computation

Choose a basis sentence

Add subtrees from fusion not present in basis

Add alternative verbalizations

Remove subtrees from basis not present in fusion

Lattice linearization

Generate all possible sentences from the fusion lattice

Score sentences using statistical language model
43

44

45


Users want to follow a story across time and watch it unfold

Network model for connecting clusters across days
 Separately cluster events from today’s news
 Connect new clusters with yesterday’s news

Allows for forking and merging of stories

Interface for viewing connections
 Summaries that update a user on what’s new

Statistical metrics to identify differences between article pairs

Uses learned model of features

Identifies differences at clause and paragraph levels
46

47

48

49

50


Hierarchical clustering

Each event cluster is divided into clusters by country

Different perspectives can be viewed side by side

Experimenting with update summarizer to identify key differences between sets of stories
51

52

53

54


Given a set of documents on the same event

Some documents are in English

Some documents are translated from other languages
55


Problem: Translated text is errorful

Exploit information available during summarization

Similar documents in cluster

Replace translated sentences with similar
English

Edit translated text

Replace named entities with extractions from similar English
56

BAGDAD. - A total of 21 prisoners has been died and a hundred more hurt by firings from mortar in the jail of Abu Gharib (to 20 kilometers to the west of Bagdad), according to has informed general into the U.S.A. Marco Kimmitt.
Bagdad in the Iraqi capital Aufstaendi attacked Bagdad on Tuesday a prison with mortars and killed after USA gifts 22 prisoners . Further 92 passengers of the Abu Ghraib prison were hurt, communicated a spokeswoman of the
American armed forces.
The Iraqi being stationed US military shot on the 20th, the same day to the allied forces detention facility which is in アブグレイブ of the Baghdad west approximately 20 kilometers, mortar 12 shot and you were packed, 22 Iraqi human prisoners died , it announced that nearly 100 people were injured.
BAGHDAD, Iraq – Insurgents fired 12 mortars into Baghdad's Abu Ghraib prison
Tuesday, killing 22 detainees and injuring 92, U.S. military officials said.
57

BAGDAD. - A total of 21 prisoners has been died and a hundred more hurt by firings from mortar in the jail of Abu Gharib (to 20 kilometers to the west of Bagdad), according to has informed general into the U.S.A. Marco Kimmitt.
Bagdad in the Iraqi capital Aufstaendi attacked Bagdad on Tuesday a prison with mortars and killed after USA gifts 22 prisoners . Further 92 passengers of the Abu Ghraib prison were hurt, communicated a spokeswoman of the
American armed forces.
The Iraqi being stationed US military shot on the 20th, the same day to the allied forces detention facility which is in アブグレイブ of the Baghdad west approximately 20 kilometers, mortar 12 shot and you were packed, 22 Iraqi human prisoners died , it announced that nearly 100 people were injured.
BAGHDAD, Iraq – Insurgents fired 12 mortars into Baghdad's Abu Ghraib prison
Tuesday, killing 22 detainees and injuring 92, U.S. military officials said.
58

Multilingual Similarity-based
Summarization
باوجتس
فلاقلا
日本語
テキ
スト
Machine
Translation
Arabic
Text
Germa n
Japane se
Text
Text
Deutsche r
Text
Select summary sentences
Documents on the same
Event
English
Text
Simplify
English
Sentences
Detect
Similar
Sentences
Replace / rewrite MT sentences
Related English Documents
English
Summar y
59


Simfinder computes similarity between sentences based on multiple features:

Proper Nouns

Verb, noun, adjective

WordNet (synonyms)
 word stem overlap

New:

Noun phrase and noun phrase variant feature
(FASTR)
60

Iraqi President Saddam Hussein that the government of Iraq over 24 years in a "black" near the port of the northern Iraq after nearly eight months of pursuit was considered the largest in history .
Similarity 0.27: Ousted Iraqi President Saddam Hussein is in custody following his dramatic capture by US forces in
Iraq.
Similarity 0.07: Saddam Hussein, the former president of
Iraq, has been captured and is being held by US forces in the country .
Similarity 0.04: Coalition authorities have said that the former Iraqi president could be tried at a war crimes tribunal, with Iraqi judges presiding and international legal experts acting as advisers .
61


Machine translated sentences long and ungrammatical

Use sentence simplification on English sentences to reduce to approximately
“one fact” per sentence

Use Arabic sentences to find most similar simple sentences

Present multiple high similarity sentences
62


'Operation Red Dawn', which led to the capture of
Saddam Hussein, followed crucial information from a member of a family close to the former Iraqi leader .

' Operation Red Dawn' followed crucial information from a member of a family close to the former Iraqi leader.

Operation Red Dawn led to the capture of Saddam Hussein .

Saddam Hussein had been the object of intensive searches by US-led forces in Iraq but previous attempts to locate him had proved unsuccessful .

Saddam Hussein had been the object of intensive searches by US-led forces in Iraq.

But previous attempts to locate him had proved unsuccessful .
63

F-measure vs. Threshold
0.75
0.7
0.65
0.6
0.55
0.5
0.45
0.4
0.35
0.3
0.25
0.2
0.15
0.1
0.05
0
0 0.1
0.2
0.3
0.4
0.5
Threshold
0.6
0.7
0.8
0.9
Full Threshold .10
Full Threshold .65
Simplified Threshold .
10
Simplified Threshold .
65
FASTR Simplified .10
FASTR Simplified .65
64


DUC (Document Understanding Conference): run by NIST

Held annually

Manual creation of topics (sets of documents)

2-7 human written summaries per topic

How well does a system generated summary cover the information in a human summary?
68


Does multi-document summarization help?

Do summaries help the user find information needed to perform a report writing task?

Do users use information from summaries in gathering their facts?

Do summaries increase user satisfaction with the online news system?

Do users create better quality reports with summaries?

How do full multi-document summaries compare with minimal 1-sentence summaries such as Google News?
69


Four parallel news systems

Source documents only; no summaries

Minimal single sentence summaries (Google
News)

Newsblaster summaries

Human summaries

All groups write reports given four scenarios

A task similar to analysts

Can only use Newsblaster for research

Time-restricted
70


4 scenarios

4 event clusters each

2 directly relevant, 2 peripherally relevant

Average 10 documents/cluster

45 participants

Balance between liberal arts, engineering

138 reports

Exit survey

Multiple-choice and open-ended questions

Usage tracking

Each click logged, on or off-site
71


The conflict between Israel and the
Palestinians has been difficult for government negotiators to settle. Most recently, implementation of the “road map for peace”, a diplomatic effort sponsored by ……

Who participated in the negotiations that produced the
Geneva Accord?

Apart from direct participants, who supported the Geneva
Accord preparations and how?

What has the response been to the Geneva Accord by the Palestinians?
72


Score report content and compare across summary conditions

Compare user satisfaction per summary condition

Comparing where subjects took report content from
73

74


More effective than a web search with
Newsblaster

Not true with documents only or single-sentence summaries

Easier to complete the task with summaries than with documents only

Enough time with summaries than documents only

Summaries helped most

5% single sentence summaries

24% Newsblaster summaries

43% human summaries
75


Summaries measurably improve a news browswer’s effectiveness for research

Users are more satisfied with Newsblaster summaries are better than single-sentence summaries like those of Google News

Users want search

Not included in evaluation
76


Cross between speech and text
 Elements of dialog
 Informal language
 More context explicitly repeated than speech

Wide variety of types of email
 Conversation to decision-making

Different reasons for summarization


Browsing large quantities of email – a mailbox
Catch-up: join a discussion late and participate – a thread
77


Collected and annotated multiple corpora of email

Hand-written summary, categorization threads&messages

Identified 3 categories of email to address:

Event planning, Scheduling, Information gathering

Developed tools:

Automatic categorization of email

Preliminary summarizers

Statistical extraction using email specific features

Components of category specific summarization
78


Use features to identify key sentences

Non-email specific: e.g., similarity to centroid

Email specific: e.g., following quoted material

Rule-based supervised machine learning

Training on human-generated summaries
 Add “wrappers” around sentences to show who said what
79


Columbia ACM chapter executive board mailing list

Approximately 10 regular participants

~300 Threads, ~1000 Messages

Threads include: scheduling and planning of meetings and events, question and answer, general discussion and chat.

Annotated by human annotators:

Hand-written summary

Categorization of threads and messages

Highlighting important information (such as questionanswer pairs)
80


Creation of Training Data

Start with human-generated summaries

Use SimFinder (a trained sentence similarity measure –
Hatzivassiloglou et al 2001) to label sentences in threads as important

Learning of Sentence Extraction Rules

Use Ripper (a rule learning algorithm – Cohen 1996) to learn rules for sentence classification

Use basic and email-specific features in machine learning

Creating summaries

Run learned rules on unseen data

Add “wrappers” around sentences to show who said what

Results

Basic: .55 precision; .40 F-measure

Email-specific: .61 precision; .50 F-measure
81

Regarding "meeting tonight...", on Oct 30, 2000,
David Michael Kalin wrote: Can I reschedule my C session for Wednesday night, 11/8, at 8:00?
Responding to this on Oct 30, 2000, James J Peach wrote: Are you sure you want to do it then?
Responding to this on Oct 30, 2000, Christy
Lauridsen wrote: David , a reminder that your scheduled to do an MSOffice session on Nov. 14, at
7pm in 252Mudd.
82

Summary from our rule-based sentence extractor:
Regarding "acm home/bjarney", on Apr 9, 2001, Mabel Dannon wrote:
Two things: Can someone be responsible for the press releases for Stroustrup?
Responding to this on Apr 10, 2001, Tina Ferrari wrote:
I think Peter, who is probably a better writer than most of us, is writing up something for dang and Dave to send out to various
ACM chapters.
Peter, we can just use that as our "press release", right?
In another subthread, on Apr 12, 2001, Keith Durban wrote:
Are you sending out upcoming events for this week?
83

Questions in interrogative form: inverted subject-verb order
Supervised rule induction approach, training Switchboard, test ACM corpus

Results:
Recall 0.56
Precision 0.96
F-measure 0.70

Recall low because:
Questions in ACM corpus start with a declarative clause

So, if you're available, do you want to come?
 if you don't mind, could you post this to the class bboard?

Results without declarative-initial questions:
Recall
Precision
0.72
0.96
F-measure 0.82
84


Use human annotated data to generate gold standard training data

Annotators were asked to highlight and associate question-answer pairs in the ACM corpus.

Learn a classifier that predicts if a subsequent segment to a question segment answers it

Represent each question and candidate answer segment by a feature vector
Labeller Labeller Union
1 2
Precision 0.690
0.680
0.728
Recall 0.652
F1-Score 0.671
0.612
0.644
0.732
0.730
85


Use QA labels as features in sentence extraction (F=.545)

Add automatically detected answers to questions in extractive summaries
(F=.566)

Start with QA pair sentences and augmented with extracted sentences
(F=.573)
86

87

(joint with Berkeley, SRI, Washington)

Goal: automatic summarization of meetings by generating “minutes” highlighting the debate that affected each decision.

Work to date: Identification of agreement/disagreement

Machine learning approach: lexical, structure, acoustic features

Use of context: who agreed with who so far?
 Adressee identification
 Bayesian modeling of context
88


Non-extractive summarization is practical today

User studies show summarization improves access to needed information

Advances and ongoing research in tracking events, multilingual summarization, perspective identification

Moves to new media (email, meetings) raise new challenges with dialog, informal language
89