Learning Semantic Subgraphs for Document
Summarization
Jure Leskovec, Marko Grobelnik
Jozef Stefan Institute, Slovenia
Natasa Milic-Frayling
Microsoft Research, Cambridge, UK
What is summarization?
The task is to produce shorter summary
version of an original document
 Two main approaches to the problem:

based – summary is a selection of
sentences from an original document
 Knowledge rich – performing semantic
We do this
analysis, representing the meaning and
generating the text satisfying length restriction
 Selection
Selection based summarization

Three main phases:




Ranking is based on





Analyzing the source text
Ranking units of the text by importance for the summary
Synthesizing an appropriate output
a location of the unit in the text
presence of cue phrases (example: It is important that…)
additional statistical attributes
…lots of heuristics and tuning of parameters (also with ML)
…output consists from most highly ranked text units
(keywords, sentences, paragraphs)
Microsoft Word’s
sentence ranking
summarizer
Knowledge rich summarization


To generate ‘true’ summary of a document we
need to (at least partially) ‘understand’ the
document text
Why is knowledge rich summarization difficult?
 One
document gives us a little of useful information
 Can not count of statistical properties of the text (lack
of data)
 Must rely on syntactical and logical structure of the
document (sentences)
Our Approach


The task is to produce shorter version of an
original document by selecting sentences from
the text
Approach:
 Learn
a machine learning model for selecting
sentences
 Use information about semantic structure of the
document (concepts and relations among concepts)
Hypothesis:
Extracted summaries should capture prominent concepts and
relations in the text.
Structure of the semantic graph of a document could help
identify the key concepts and relations for summarization.
Original
document
Cracks Appear in U.N. Trade Embargo Against Iraq.
Cracks appeared Tuesday in the U.N. trade embargo against Iraq as Saddam Hussein sought to circumvent the economic noose around his country. Japan, meanwhile,
announced it would increase its aid to countries hardest hit by enforcing the sanctions. Hoping to defuse criticism that it is not doing its share to oppose Baghdad, Japan
said up to $2 billion in aid may be sent to nations most affected by the U.N. embargo on Iraq. President Bush on Tuesday night promised a joint session of Congress and a
nationwide radio and television audience that ``Saddam Hussein will fail'' to make his conquest of Kuwait permanent. ``America must stand up to aggression, and we will,''
said Bush, who added that the U.S. military may remain in the Saudi Arabian desert indefinitely. ``I cannot predict just how long it will take to convince Iraq to withdraw
from Kuwait,'' Bush said. More than 150,000 U.S. troops have been sent to the Persian Gulf region to deter a possible Iraqi invasion of Saudi Arabia. Bush's aides said the
president would follow his address to Congress with a televised message for the Iraqi people, declaring the world is united against their government's invasion of Kuwait.
Saddam had offered Bush time on Iraqi TV. The Philippines and Namibia, the first of the developing nations to respond to an offer Monday by Saddam of free oil _ in
exchange for sending their own tankers to get it _ said no to the Iraqi leader. Saddam's offer was seen as a none-too-subtle attempt to bypass the U.N. embargo, in effect
since four days after Iraq's Aug. 2 invasion of Kuwait, by getting poor countries to dock their tankers in Iraq. But according to a State Department survey, Cuba and
Romania have struck oil deals with Iraq and companies elsewhere are trying to continue trade with Baghdad, all in defiance of U.N. sanctions. Romania denies the
allegation. The report, made available to The Associated Press, said some Eastern European countries also are trying to maintain their military sales to Iraq. A wellinformed source in Tehran told The Associated Press that Iran has agreed to an Iraqi request to exchange food and medicine for up to 200,000 barrels of refined oil a day
and cash payments. There was no official comment from Tehran or Baghdad on the reported food-for-oil deal. But the source, who requested anonymity, said the deal was
struck during Iraqi Foreign Minister Tariq Aziz's visit Sunday to Tehran, the first by a senior Iraqi official since the 1980-88 gulf war. After the visit, the two countries
announced they would resume diplomatic relations. Well-informed oil industry sources in the region, contacted by The AP, said that although Iran is a major oil exporter
itself, it currently has to import about 150,000 barrels of refined oil a day for domestic use because of damages to refineries in the gulf war. Along similar lines, ABC News
reported that following Aziz's visit, Iraq is apparently prepared to give Iran all the oil it wants to make up for the damage Iraq inflicted on Iran during their conflict.
Secretary of State James A. Baker III, meanwhile, met in Moscow with Soviet Foreign Minister Eduard Shevardnadze, two days after the U.S.-Soviet summit that produced
a joint demand that Iraq withdraw from Kuwait. During the summit, Bush encouraged Mikhail Gorbachev to withdraw 190 Soviet military specialists from Iraq, where they
remain to fulfill contracts. Shevardnadze told the Soviet parliament Tuesday the specialists had not reneged on those contracts for fear it would jeopardize the 5,800 Soviet
citizens in Iraq. In his speech, Bush said his heart went out to the families of the hundreds of Americans held hostage by Iraq, but he declared, ``Our policy cannot
change, and it will not change. America and the world will not be blackmailed.'' The president added: ``Vital issues of principle are at stake. Saddam Hussein is literally
trying to wipe a country off the face of the Earth.'' In other developments: _A U.S. diplomat in Baghdad said Tuesday up to 800 Americans and Britons will fly out of Iraqioccupied Kuwait this week, most of them women and children leaving their husbands behind. Saddam has said he is keeping foreign men as human shields against attack.
On Monday, a planeload of 164 Westerners arrived in Baltimore from Iraq. Evacuees spoke of food shortages in Kuwait, nighttime gunfire and Iraqi roundups of young
people suspected of involvement in the resistance. ``There is no law and order,'' said Thuraya, 19, who would not give her last name. ``A soldier can rape a father's
daughter in front of him and he can't do anything about it.'' _The State Department said Iraq had told U.S. officials that American males residing in Iraq and Kuwait who
were born in Arab countries will be allowed to leave. Iraq generally has not let American males leave. It was not known how many men the Iraqi move could affect. _A
Pentagon spokesman said ``some increase in military activity'' had been detected inside Iraq near its borders with Turkey and Syria. He said there was little indication
hostilities are imminent. Defense Secretary Dick Cheney said the cost of the U.S. military buildup in the Middle East was rising above the $1 billion-a-month estimate
generally used by government officials. He said the total cost _ if no shooting war breaks out _ could total $15 billion in the next fiscal year beginning Oct. 1. Cheney
promised disgruntled lawmakers ``a significant increase'' in help from Arab nations and other U.S. allies for Operation Desert Shield. Japan, which has been accused of
responding too slowly to the crisis in the gulf, said Tuesday it may give $2 billion to Egypt, Jordan and Turkey, hit hardest by the U.N. prohibition on trade with Iraq. ``The
pressure from abroad is getting so strong,'' said Hiroyasu Horio, an official with the Ministry of International Trade and Industry. Local news reports said the aid would be
extended through the World Bank and International Monetary Fund, and $600 million would be sent as early as mid-September. On Friday, Treasury Secretary Nicholas
Brady visited Tokyo on a world tour seeking $10.5 billion to help Egypt, Jordan and Turkey. Japan has already promised a $1 billion aid package for multinational
peacekeeping forces in Saudi Arabia, including food, water, vehicles and prefabricated housing for non-military uses. But critics in the United States have said Japan should
do more because its economy depends heavily on oil from the Middle East. Japan imports 99 percent of its oil. Japan's constitution bans the use of force in settling
international disputes and Japanese law restricts the military to Japanese territory, except for ceremonial occasions. On Monday, Saddam offered developing nations free
oil if they would send their tankers to pick it up. The first two countries to respond Tuesday _ the Philippines and Namibia _ said no. Manila said it had already fulfilled its oil
requirements, and Namibia said it would not ``sell its sovereignty'' for Iraqi oil. Venezuelan President Carlos Andres Perez dismissed Saddam's offer of free oil as a
``propaganda ploy.'' Venezuela, an OPEC member, has led a drive among oil-producing nations to boost production to make up for the shortfall caused by the loss of Iraqi
and Kuwaiti oil from the world market. Their oil makes up 20 percent of the world's oil reserves. Only Saudi Arabia has higher reserves. But according to the State
Department, Cuba, which faces an oil deficit because of reduced Soviet deliveries, has received a shipment of Iraqi petroleum since U.N. sanctions were imposed five
weeks ago. And Romania, it said, expects to receive oil indirectly from Iraq. Romania's ambassador to the United States, Virgil Constantinescu, denied that claim Tuesday,
calling it ``absolutely false and without foundation.''.
Our approach to
summarization
Automatically generated
document summary
Linguistic processing
and Creation of
semantic graph
Automatic summarization
by selecting relevant
parts of the graph
We use machine learning
to learn selection model
Cracks appeared in the U.N. trade embargo
against Iraq. The State Department reports
that Cuba and Romania have struck oil deals
with Iraq as others attempt to trade with
Baghdad in defiance of the sanctions. Iran has
agreed to exchange food and medicine for Iraqi
oil. Saddam has offered developing nations free
oil if they send their tankers to pick it up. Thus
far, none has accepted. Japan, accused of
responding too slowly to the Gulf crisis, has
promised $2 billion in aid to countries hit hardest
by the Iraqi trade embargo. President Bush has
promised that Saddam's aggression will not
succeed.
Natural
language
generation
Detailed Summarization Procedure
Linguistic analysis of the text
- Deep parsing of sentences
Refinement of the text parse
- Named-entity consolidation
Determine that ’George Bush’ = ‘Bush’
= ‘U.S. president’
- Anaphora resolution
Link pronouns with name-entities
Extract Subject–Predicate–Object
triples
Compose a graph from triples
Describe each triple with a set of
features for learning
Learn a model to classify triples into
the summary
Generate a summary graph
Use summary graph to generate
textual document summary
Tom Sawyer went to
town. He met a friend.
Tom was happy. …
Tom Sawyer went to town.
He [Tom Sawyer] met a
friend. Tom [Tom Sawyer]
was happy. …
Tom  go  town
Tom  meet  friend
Tom  is  happy
Detailed Summarization Procedure
Linguistic analysis of the text
- Deep parsing of sentences
Refinement of the text parse
- Named-entity consolidation
Determine that ’George Bush’ = ‘Bush’
= ‘U.S. president’
- Anaphora resolution
Link pronouns with name-entities
Extract Subject–Predicate–Object
triples
Compose a graph from triples
Describe each triple with a set of
features for learning
Learn a model to classify triples into
the summary
Generate a summary graph
Use summary graph to generate
textual document summary
Tom Sawyer went to
town. He met a friend.
Tom was happy. …
Tom Sawyer went to town.
He [Tom Sawyer] met a
friend. Tom [Tom Sawyer]
was happy. …
Tom  go  town
Tom  meet  friend
Tom  is  happy
Linguistic analysis


We use Microsoft’s NLPWin linguistic processing
tool (deep parser)
NLPWin uses various external resources:
 knowledge



base MindNet, dictionary, thesaurus
Input to NLPWin: a sentence
Output from NLPWin: a parse tree of
syntactical and logical form of the sentence
NLPWin parse tree is the input to procedures for
anaphora resolution, name-entity consolidation
and extraction of triples
Linguistic processing tool NLPWin
Syntactic tree
for the sentence
“Jure sent
Marko a letter”
Logical form the sentence
“Jure sent Marko a letter”
Past – Past Tense for ‘send’
Sing – Singular
PrprN – Proper Name
Pers3 – Third person singular
Detailed Summarization Procedure
Linguistic analysis of the text
- Deep parsing of sentences
Refinement of the text parse
- Named-entity consolidation
Determine that ’George Bush’ = ‘Bush’
= ‘U.S. president’
- Anaphora resolution
Link pronouns with name-entities
Extract Subject–Predicate–Object
triples
Compose a graph from triples
Describe each triple with a set of
features for learning
Learn a model to classify triples into
the summary
Generate a summary graph
Use summary graph to generate
textual document summary
Tom Sawyer went to
town. He met a friend.
Tom was happy. …
Tom Sawyer went to town.
He [Tom Sawyer] met a
friend. Tom [Tom Sawyer]
was happy. …
Tom  go  town
Tom  meet  friend
Tom  is  happy
Named entities consolidation (1)



Named entities are names of people, places,
companies, …
Pronouns refer (link) to named entities
For each named entity determine the type of pronoun
that can refer to it
Named entity types: He, She, He-or-She, They, Name
… types are ordered into the hierarchy:
He is more specific than He-or-She
… for example: a name entity of type He has to be masculine,
singular and human


Consolidation procedure:




Determine name entity type
Split each entity into words
Remove all stop words and common words
Try to match with previously seen entities
Common words: mr,
ms, co, inc, federal,
international, school,
national, group, …
Named entities consolidation (2)

Examples of names which are found to refer to the
same named entity:
 Hillary
Rodham Clinton, Hillary Clinton, Hillary Rodham,
Mrs. Clinton


Accuracy around 90%
Typical errors:
 President Kennedy ≠ Kennedy School of Government
 After common words removal, only word Kennedy is left
of companies Reuter(s), Levi(‘s) and
Lloyd(‘s) are mistaken for a person
 We do not distinguish between first and last name
 Names


… in some cases brothers can become one named entity
Michael Milken = Milken = Robert Milken
Pronomial anaphora resolution

We link pronouns with their references
Mary likes Paul. She went to buy him a present.
 Mary likes Paul. She [Mary] went to buy him [Paul] a present.

Method:





We restrict anaphora resolution to 5 pronouns: she, he, who, I,
they.
From the pronoun, traverse the text searching for candidate
references and assign a score
The score is based on the distance from the pronoun and
semantic information
Note we assume that pronouns refer only to named entities
found in the document
Problem:

One passenger in King's car said they had been drinking liquor.
Anaphora resolution (2)

What we don’t find:







Error: he == John
Quoted speech: John said: “He is sick.”
he == I
But: “I hope so," he replies after a pause.
Error: him == Tom
Tom wrote a letter to Bill. He told him …
The relationship between active volcanoes and the communities that
surround them is not always confrontational.
We don’t link them
Jordan's King Hussein and Yasser Arafat's open sympathy for Iraq
has strained their relations with the U.S.
Can’t link they
Mistakes are not so common
The most fatal case is when we wrongly resolve first
occurrence of a pronoun and then follow many sentences
using only the pronoun to refer to a person.
Anaphora resolution evaluation

We manually labeled 91 articles
 Containing
1506 pronouns
 1024 (68%) pronouns are he, she, I, they, who

We try to link all of them
 Other
482 (32%) pronouns are: it, you, we,
what, …
Anaphora resolution evaluation
Pronoun
Frequency
Frequency [%]
Accuracy [%]
He
681
45.22
86.9
They
244
16.20
67.2
It
204
13.55
I
64
4.25
You
50
3.32
We
44
2.92
That
44
2.92
What
27
1.79
She
24
1.59
This
22
1.46
Who
11
0.73
63.6
1506
100
81.2
82.8
62.5
…
Total
Accuracy on 5 selected 81.2% (55.2% if counting all pronouns)
Detailed Summarization Procedure
Linguistic analysis of the text
- Deep parsing of sentences
Refinement of the text parse
- Named-entity consolidation
Determine that ’George Bush’ = ‘Bush’
= ‘U.S. president’
- Anaphora resolution
Link pronouns with name-entities
Extract Subject–Predicate–Object
triples
Compose a graph from triples
Describe each triple with a set of
features for learning
Learn a model to classify triples into
the summary
Generate a summary graph
Use summary graph to generate
textual document summary
Tom Sawyer went to
town. He met a friend.
Tom was happy. …
Tom Sawyer went to town.
He [Tom Sawyer] met a
friend. Tom [Tom Sawyer]
was happy. …
Tom  go  town
Tom  meet  friend
Tom  is  happy
Extracting triples

Enhanced parse tree is traversed to identify
Subject–Predicate–Object triples

Example:
“Conservatives embraced the nomination
while liberals were cautious or hostile”
Resulting triples:
conservative  embrace  nomination
liberal  is  cautious
liberal  is  hostile
Detailed Summarization Procedure
Linguistic analysis of the text
- Deep parsing of sentences
Refinement of the text parse
- Named-entity consolidation
Determine that ’George Bush’ = ‘Bush’
= ‘U.S. president’
- Anaphora resolution
Link pronouns with name-entities
Extract Subject–Predicate–Object
triples
Compose a graph from triples
Describe each triple with a set of
features for learning
Learn a model to classify triples into
the summary
Generate a summary graph
Use summary graph to generate
textual document summary
Tom Sawyer went to
town. He met a friend.
Tom was happy. …
Tom Sawyer went to town.
He [Tom Sawyer] met a
friend. Tom [Tom Sawyer]
was happy. …
Tom  go  town
Tom  meet  friend
Tom  is  happy
Learning: Feature construction

Graph consists of nodes, referred as
concepts, which can be subjects or
objects and edges which are
predicates and capture relations
among concepts.

We use Word net to identify and
compact synonym nodes – as they
correspond to the same concepts.
To create semantic graph
find Subjects
supply (Objects)
Typ_obj with
Purpose
the same meaning
chicken
WordNet
Is_a
clean
Is_a
smooth
Is_a
poultry
Quesp
hen
Typ_obj
Typ_subj
preen
Means
chatter
Is_a
Typ_obj
gaggle
Classifier
Is_a
peck
number
Is_a Means
Is_a
bird
sound
Concept
Is_a
goose
beak
Part
Typ_subj
face
26 types of relations
115,000 concepts
Location
Is_a
feather
wing
Is_a
hawk
Is_a
claw
Part
turtle
mouth
creature
Part
Concept Typ_subj
fly
plant
Is_a
Relation
Typ_subj
strike
egg
Not_is_a
Is_a
Is_a
Is_a
make
meat
Purpose
animal
Typ_subj
keep
Typ_obj
duck
Caused_by
quack
Is_a
bill
Is_a
Is_a
Is_a
Is_a
Typ_obj
leg
catch
Is_a
limb
opening
arm
Constructing semantic graph (2)
Two words (Subjects, Objects) are
synonyms if they are synonyms in any of
their meanings in WordNet
 Examples:
Using WordNet we find

 watcher 
that the following triples
follow  moon
have the same meaning
spectator  watch  moon
 governor Patten  change  position
governor Patten  modify  attitude
 earthquake  hit  Northern Iran
quake  strike  Northern Iran
Experiments
We would like learn how to select relevant
parts of document semantic graph into the
summary
 For learning we will use summaries
created by humans
 Each Subject-Predicate-Object triple will
be a learning example
 Each triple will be labeled whether it is
interesting for the summary or not

Example of automatic summary
Cracks Appear in U.N. Trade Embargo Against Iraq.
Human extracted
summary
Cracks appeared Tuesday in the U.N. trade embargo against Iraq as Saddam Hussein sought to circumvent the economic noose around his country. Japan,
meanwhile, announced it would increase its aid to countries hardest hit by enforcing the sanctions. Hoping to defuse criticism that it is not doing its share to oppose
Baghdad, Japan said up to $2 billion in aid may be sent to nations most affected by the U.N. embargo on Iraq. President Bush on Tuesday night promised a joint
session of Congress and a nationwide radio and television audience that ``Saddam Hussein will fail'' to make his conquest of Kuwait permanent. ``America must
stand up to aggression, and we will,'' said Bush, who added that the U.S. military may remain in the Saudi Arabian desert indefinitely. ``I cannot predict just how
long it will take to convince Iraq to withdraw from Kuwait,'' Bush said. More than 150,000 U.S. troops have been sent to the Persian Gulf region to deter a possible
Iraqi invasion of Saudi Arabia. Bush's aides said the president would follow his address to Congress with a televised message for the Iraqi people, declaring the world
is united against their government's invasion of Kuwait. Saddam had offered Bush time on Iraqi TV. The Philippines and Namibia, the first of the developing nations
to respond to an offer Monday by Saddam of free oil _ in exchange for sending their own tankers to get it _ said no to the Iraqi leader. Saddam's offer was seen as
a none-too-subtle attempt to bypass the U.N. embargo, in effect since four days after Iraq's Aug. 2 invasion of Kuwait, by getting poor countries to dock their
tankers in Iraq. But according to a State Department survey, Cuba and Romania have struck oil deals with Iraq and companies elsewhere are trying to continue
trade with Baghdad, all in defiance of U.N. sanctions. Romania denies the allegation. The report, made available to The Associated Press, said some Eastern
European countries also are trying to maintain their military sales to Iraq. A well-informed source in Tehran told The Associated Press that Iran has agreed to an
Iraqi requestto exchange
andafter
medicine
for up to 200,000
refinedanger
oil a day
andeconomic
cash payments.
Therein
was
no official comment from Tehran or
Eight food
years
a volcano
scare barrels
incitedof fear,
and
gloom
Sierra
Baghdad on the reported food-for-oil deal. But the source, who requested anonymity, said the deal was struck during Iraqi Foreign Minister Tariq Aziz's visit Sunday
about
renewed
underground
lava
movement
thatresume
is
to Tehran, the first resorts,
by a seniorresidents
Iraqi officialare
sincenonchalant
the 1980-88 gulf
war. After
the visit,
the two countries
announced
they would
diplomatic relations. Wellinformed oil industry
sources in the
region, contacted
The AP, said that although Iran is a major oil exporter itself, it currently has to import about 150,000 barrels
triggering
thousands
of tinybyearthquakes.
of refined oil a day for domestic use because of damages to refineries in the gulf war. Along similar lines, ABC News reported that following Aziz's visit, Iraq is

The
resort
town's
in Long
a 19-mile-long,
9-mileapparently prepared
to give
Iran all
the oil 4,700
it wants permanent
to make up forresidents
the damage live
Iraq inflicted
onValley,
Iran during
their conflict. Secretary
of State James A. Baker III,
meanwhile, met in Moscow
with Sovietcrater
Foreignknown
Minister as
Eduard
Shevardnadze, two days after the U.S.-Soviet summit that produced a joint demand that Iraq
wide volcanic
a caldera.
withdraw from Kuwait. During the summit, Bush encouraged Mikhail Gorbachev to withdraw 190 Soviet military specialists from Iraq, where they remain to fulfill

The told
Earth's
crustparliament
is beingTuesday
stretched
apart in
allowing
molten
fill halfcontracts. Shevardnadze
the Soviet
the specialists
hadthe
notregion,
reneged on
those contracts
forrock
fear itto
would
jeopardize the 5,800 Soviet citizens
in Iraq. In his speech,
Bush said chambers
his heart wentunder
out to the
of the hundreds of Americans held hostage by Iraq, but he declared, ``Our policy cannot change,
mile-wide
thefamilies
caldera.
and it will not change. America and the world will not be blackmailed.'' The president added: ``Vital issues of principle are at stake. Saddam Hussein is literally trying

The
was
created
730,000
years_Aago
one in
ofBaghdad
Earth'ssaid
most
powerful
eruptions,
a Britons will fly out of Iraqito wipe a country
off
the valley
face of the
Earth.''
In other
developments:
U.S. by
diplomat
Tuesday
up to 800
Americans and
occupied Kuwait this
week,that
mostspewed
of them women
and children
leaving
their husbands
behind.
has said heof
is keeping
blast
600 times
more
material
than the
May Saddam
1980 eruption
Mountforeign
St. men as human shields against
attack. On Monday,Helens
a planeload
164 Westerners
arrived in Baltimore from Iraq. Evacuees spoke of food shortages in Kuwait, nighttime gunfire and Iraqi roundups
in of
Washington
state.
of young people suspected of involvement in the resistance. ``There is no law and order,'' said Thuraya, 19, who would not give her last name. ``A soldier can rape
a father's daughter
in front of the
him and
he can't
do anything
it.'' _The of
State
Department
said Iraq had
U.S. officials
that American males residing in Iraq

Despite
current
activity,
theabout
probability
a major
earthquake
ortold
a volcanic
eruption
and Kuwait who were
born
in Arabiscountries
will be1allowed
to leave.
Iraq
generally
has
not let Hill,
American
males
leave. It was not known how many men the Iraqi
in
the
area
“less
than
percent
each
year,”
said
David
the
U.S.
move could affect. _A Pentagon spokesman said ``some increase in military activity'' had been detected inside Iraq near its borders with Turkey and Syria. He said
there was little
hostilitiesSurvey
are imminent.
Defense Secretary
Dick Cheney
said the cost
of the Valley.
U.S. military
buildup
in the Middle East was rising above the $1
 indication
Geological
geophysicist
in charge
of research
at Long
Mono
County
billion-a-month estimate
generally
used
by
government
officials.
He
said
the
total
cost
_
if
no
shooting
war
breaks
out
_
could
total $15 billion in the next fiscal year
Sheriff Martin Strelneck called such estimates “a scientific guessing game,” and said
beginning Oct. 1. Cheney promised disgruntled lawmakers ``a significant increase'' in help from Arab nations and other U.S. allies for Operation Desert Shield.
area
residents
rarely
discuss
the
latest
swarm
of
earthquakes,
which
started
in
May
Japan, which has been accused of responding too slowly to the crisis in the gulf, said Tuesday it may give $2 billion to Egypt, Jordan and Turkey, hit hardest by the
U.N. prohibition on 1989.
trade with Iraq. ``The pressure from abroad is getting so strong,'' said Hiroyasu Horio, an official with the Ministry of International Trade and
Industry. Local news reports said the aid would be extended through the World Bank and International Monetary Fund, and $600 million would be sent as early as
 OnAs
a result,
the
Geological
issued
“notice
potential
volcanic
hazard”
forJordan and Turkey. Japan has
mid-September.
Friday,
Treasury
Secretary
NicholasSurvey
Brady visited
Tokyoaon
a world of
tour
seeking $10.5
billion to
help Egypt,
already promised a Long
$1 billion
aid package
for 1982.
multinational peacekeeping forces in Saudi Arabia, including food, water, vehicles and prefabricated housing for nonValley
in May
military uses. But critics in the United States have said Japan should do more because its economy depends heavily on oil from the Middle East. Japan imports 99
That constitution
warning, bans
coupled
jarring
earthquakes,
and
aggravated
a to Japanese territory, except for
percent of itsoil. Japan's
the usewith
of force
in settling
international damaged
disputes and tourism
Japanese law
restricts
the military
ceremonial occasions.
On Monday,
offered developing
nations
free market.
oil if they would send their tankers to pick it up. The first two countries to respond
recession
inSaddam
the once-booming
real
estate
Tuesday _ the Philippines and Namibia _ said no. Manila said it had already fulfilled its oil requirements, and Namibia said it would not ``sell its sovereignty'' for Iraqi
oil. Venezuelan President Carlos Andres Perez dismissed Saddam's offer of free oil as a ``propaganda ploy.'' Venezuela, an OPEC member, has led a drive among oilproducing nations to boost production to make up for the shortfall caused by the loss of Iraqi and Kuwaiti oil from the world market. Their oil makes up 20 percent
of the world's oil reserves. Only Saudi Arabia has higher reserves. But according to the State Department, Cuba, which faces an oil deficit because of reduced Soviet
deliveries, has received a shipment of Iraqi petroleum since U.N. sanctions were imposed five weeks ago. And Romania, it said, expects to receive oil indirectly from
Iraq. Romania's ambassador to the United States, Virgil Constantinescu, denied that claim Tuesday, calling it ``absolutely false and without foundation.''.
7800 characters, 1300 words
Human written summary

California's Long Valley is a 19-mile caldera created 730,000 years ago by
an eruption 600 times larger than Mount St. Helens. In May 1989, new
underground lava movement began triggering thousands of tiny
earthquakes and raising the valley floor. Residents refuse to heed warnings,
remembering a 1982 false alarm that damaged tourism and aggravated a
recession. Afterward, journalists were accused of sensationalism and
scientists of scaring people to get more funding. Currently, 5-10 small
quakes happen daily as the Earth's crust is stretched apart and magma fills
half-mile-wide chambers 4 miles under the caldera, but the probability of
eruption is less than one percent.
Human sentence selection summary








Eight years after a volcano scare incited fear, anger and economic gloom in Sierra resorts, residents are
nonchalant about renewed underground lava movement that is triggering thousands of tiny earthquakes.
The resort town's 4,700 permanent residents live in Long Valley, a 19-mile-long, 9-mile-wide volcanic
crater known as a caldera.
The Earth's crust is being stretched apart in the region, allowing molten rock to fill half-mile-wide
chambers under the caldera.
The valley was created 730,000 years ago by one of Earth's most powerful eruptions, a blast that
spewed 600 times more material than the May 1980 eruption of Mount St. Helens in Washington state.
Despite the current activity, the probability of a major earthquake or a volcanic eruption in the area is
“less than 1 percent each year,” said David Hill, the U.S.
Geological Survey geophysicist in charge of research at Long Valley. Mono County Sheriff Martin
Strelneck called such estimates “a scientific guessing game,” and said area residents rarely discuss the
latest swarm of earthquakes, which started in May 1989.
As a result, the Geological Survey issued a “notice of potential volcanic hazard” for Long Valley in May
1982.
That warning, coupled with jarring earthquakes, damaged tourism and aggravated a recession in the
once-booming real estate market.
Full document
semantic graph
Full document
semantic graph
Automatically generated summary
Experimental settings


We use Support Vector Machines machine
learning algorithm with linear kernel
Binary classification problem:
 Positive
examples are triples from the sentences
which humans selected into the summary
 Negative examples are all other triples

Treat each document as a unit:
 Take
all triples from the document for either testing or
learning

Report microaveraged values of precision, recall
and F1
DUC 2001 dataset
Document understanding conference
 300 newspaper articles on 30 different
topics: people, natural disasters, events, …
 Each topic has about 10 articles
Learn if a triple comes
 For each article we have…

… hand made summary
… sentence selection summary

from the sentence which
was selected into the
summary
Articles are long (1000 words,
50 sentences, each
Only about 30% of the triples
has 22 words)
from the human written abstract
can be found in the document
Triple attributes

Features used in the learning process include triples described by
the following attributes:
 Positional information
 Of the sentence from which the triple was derived relative to
the document text
 Of the triple relative to the beginning of the sentence
 NLPWin linguistic attributes of the nodes in the triple:
 18 syntactic attributes
 100 semantic attributes
 14 graph attributes: PageRank, In/Out Degree, reachable
neighbours, etc.
On our dataset this yield:
TOTAL of 466 attributes
On average 72 non-zero attributes per triple.
Performance for various attribute sets
Attribute set
Training Set
Test Set
Precision
Recall
F1
Precision
Recall
F1
Sentence Position +
Terms
65.87
92.48
76.94
28.87
37.08
32.46
only Position
(triple + sentence)
31.21
52.49
39.15
31.05
52.58
39.05
only Graph
27.78
57.46
37.46
27.25
56.90
36.85
only Linguistic
29.77
61.79
40.18
22.29
47.52
30.29
Position + Linguistic
31.16
67.00
42.54
28.67
62.57
39.33
Position + Graph
33.51
63.85
43.95
42.71
63.02 43.07
Position + Graph
+ Linguistic
35.82
72.69
47.99
31.41
64.88 42.33
Baseline performance (sentence position + selected terms from the
Performance
for various attribute sets
sentence) F1=32.46 is lower than in any of the other runs, except
for ‘only linguistic’ attributes (F1=30.29).
Training Set
Test Set
‘only set
linguistic’ run includes only generic syntactic and semantic
Attribute
Precision
Recall
F1
Precision
Recall
labels - not expected
to be good
discriminators
on their own.
F1
Sentence Position +
Terms
65.87
92.48
76.94
28.87
37.08
32.46
only Position
(triple + sentence)
31.21
52.49
39.15
31.05
52.58
39.05
only Graph
27.78
57.46
37.46
27.25
56.90
36.85
only Linguistic
29.77
61.79
40.18
22.29
47.52
30.29
Position + Linguistic
31.16
67.00
42.54
28.67
62.57
39.33
Position + Graph
33.51
63.85
43.95
42.71
63.02 43.07
Position + Graph
+ Linguistic
35.82
72.69
47.99
31.41
64.88 42.33
Adding generic linguistic attributes reduces precision
Performance
various
attribute sets
Position of triples andfor
sentences
 P=31.05
Adding linguistic attributes
 P=28.67
Training Set
but
consistently
Attribute
set increases recall
Test Set
Precision
Recall
F1
Precision
Recall
F1
Sentence Position +
Terms
65.87
92.48
76.94
28.87
37.08
32.46
only Position
(triple + sentence)
31.21
52.49
39.15
31.05
52.58
39.05
only Graph
27.78
57.46
37.46
27.25
56.90
36.85
only Linguistic
29.77
61.79
40.18
22.29
47.52
30.29
Position + Linguistic
31.16
67.00
42.54
28.67
62.57
39.33
Position + Graph
33.51
63.85
43.95
32.71
63.02 43.07
Position + Graph
+ Linguistic
35.82
72.69
47.99
31.41
64.88 42.33
Performance
various
attribute
sets
Information for
about
the graph
structure helps
Position of triples and sentences  F1=39.05
Training Set
Test Set
Attribute
set
Adding structure information
 F1=43.07
Precision Recall
F1
Precision Recall
F1
Sentence Position +
Terms
65.87
92.48
76.94
28.87
37.08
32.46
only Position
(triple + sentence)
31.21
52.49
39.15
31.05
52.58
39.05
only Graph
27.78
57.46
37.46
27.25
56.90
36.85
only Linguistic
29.77
61.79
40.18
22.29
47.52
30.29
Position + Linguistic
31.16
67.00
42.54
28.67
62.57
39.33
Position + Graph
33.51
63.85
43.95
42.71
63.02 43.07
Position + Graph
+ Linguistic
35.82
72.69
47.99
31.41
64.88 42.33
Insights

We determine the median and
quartiles of the ranks across 10 runs.
Most highly ranked features in SVM normal:
Attribute
1st quartile
Median 3rd quartile
Object – Authority weight
1
1
2
Object – size of weakly connected
component
2
2.5
3
Object – degree of a node
2
3
3
Object – is name of a country
4
5
5
Subject – size of weakly connected
component
6
7
9
Subject – degree of a node
6
10.5
12
Object – PageRank weight
6
11
12
Object – is name of a geographical
location
8
13
16
Subject – Authority weight
13
18.5
23
Insights

We determine the median and
quartiles of the ranks across 10 runs.
Most highly ranked features in SVM normal for nonsummary triples:
Attribute
1st quartile Median
3rd quartile
Subject – 1st person
1
1
1
Predicate – quoted speech
2
2
2
Predicate – position of a word in the
sentence
3
3
3
Object – quoted speech
4
4.5
6
Object – feminine gender
5
5
7
Object – position of a word in the
sentence
8
10
14
Subject – position of a sentence in the
document
14
16
19
Learning cross-topic summarization

10 fold cross validation for different sample size of
training data:
Training Set
Test Set
Learning
documents
Precision
Recall
F1
Precision
Recall
F1
10
33.48
88.44
48.58
23.05
64.67
33.99
20
31.26
86.45
45.92
24.49
68.69
36.11
50
29.42
82.53
43.37
25.75
72.81
38.04
100
28.64
79.91
42.16
26.25
73.22
38.64
Learn if a triple comes from the sentence
which was selected into the summary
Precision = TP / (TP + FP)
Recall = TP / (TP + FN)
F1 = 2 Prec Rec/(Prec+Rec)
Within-topic summarization

Leave one out cross validation averaged over 30
different topics:
Learning using
samples of 5
Training Set
Test Set
Precision
Recall
F1
Precision
Recall
F1
Within-topic
36.49
90.63
52.03
23.60
60.05
33.89
Cross-topic
36.59
92.23
52.40
20.73
60.28
30.85
documents
Further work

Additional experiments:





no-structure vs. structure (graph vs. no-graph)
different levels of linguistic processing
Use WordNet less naively for graph
construction, abstraction
New sets of attributes: WordNet
Study the differences between the triples from
human written abstracts and sentence selection
abstracs
Conclusion

Experiments on the dataset used show:


Attributes that characterize the document semantic graph
improve selection of triples for summarization.
 This results need to be verified on additional data sets
 Need to perform comparison with additional summarization
methods
 Explore various strategies for extracting and generating
summaries based on extracted triples.
We observe:

No combination of features that we examined lead to good
separation of positive and negative triples in the feature space
 Opportunity for further investigations and improvements.
Example of automatic summary
Cracks Appear in U.N. Trade Embargo Against Iraq.
Human written
summary
Cracks appeared Tuesday in the U.N. trade embargo against Iraq as Saddam Hussein sought to circumvent the economic noose around his country. Japan,
meanwhile, announced it would increase its aid to countries hardest hit by enforcing the sanctions. Hoping to defuse criticism that it is not doing its share to oppose
Baghdad, Japan said up to $2 billion in aid may be sent to nations most affected by the U.N. embargo on Iraq. President Bush on Tuesday night promised a joint
session of Congress and a nationwide radio and television audience that ``Saddam Hussein will fail'' to make his conquest of Kuwait permanent. ``America must
stand up to aggression, and we will,'' said Bush, who added that the U.S. military may remain in the Saudi Arabian desert indefinitely. ``I cannot predict just how
long it will take to convince Iraq to withdraw from Kuwait,'' Bush said. More than 150,000 U.S. troops have been sent to the Persian Gulf region to deter a possible
Iraqi invasion of Saudi Arabia. Bush's aides said the president would follow his address to Congress with a televised message for the Iraqi people, declaring the world
is united against their government's invasion of Kuwait. Saddam had offered Bush time on Iraqi TV. The Philippines and Namibia, the first of the developing nations
to respond to an offer Monday by Saddam of free oil _ in exchange for sending their own tankers to get it _ said no to the Iraqi leader. Saddam's offer was seen as
a none-too-subtle attempt to bypass the U.N. embargo, in effect since four days after Iraq's Aug. 2 invasion of Kuwait, by getting poor countries to dock their
tankers in Iraq. But according to a State Department survey, Cuba and Romania have struck oil deals with Iraq and companies elsewhere are trying to continue
trade with Baghdad, all in defiance of U.N. sanctions. Romania denies the allegation. The report, made available to The Associated Press, said some Eastern
European countries also are trying to maintain their military sales to Iraq. A well-informed source in Tehran told The Associated Press that Iran has agreed to an
Iraqi request to exchange food and medicine for up to 200,000 barrels of refined oil a day and cash payments. There was no official comment from Tehran or
Baghdad on the reported food-for-oil deal. But the source, who requested anonymity, said the deal was struck during Iraqi Foreign Minister Tariq Aziz's visit Sunday
to Tehran, the first by a senior Iraqi official since the 1980-88 gulf war. After the visit, the two countries announced they would resume diplomatic relations. Wellinformed oil industry sources in the region, contacted by The AP, said that although Iran is a major oil exporter itself, it currently has to import about 150,000 barrels
of refined oil a day for domestic use because of damages to refineries in the gulf war. Along similar lines, ABC News reported that following Aziz's visit, Iraq is
apparently prepared to give Iran all the oil it wants to make up for the damage Iraq inflicted on Iran during their conflict. Secretary of State James A. Baker III,
meanwhile, met in Moscow with Soviet Foreign Minister Eduard Shevardnadze, two days after the U.S.-Soviet summit that produced a joint demand that Iraq
withdraw from Kuwait. During the summit, Bush encouraged Mikhail Gorbachev to withdraw 190 Soviet military specialists from Iraq, where they remain to fulfill
contracts. Shevardnadze told the Soviet parliament Tuesday the specialists had not reneged on those contracts for fear it would jeopardize the 5,800 Soviet citizens
in Iraq. In his speech, Bush said his heart went out to the families of the hundreds of Americans held hostage by Iraq, but he declared, ``Our policy cannot change,
and it will not change. America and the world will not be blackmailed.'' The president added: ``Vital issues of principle are at stake. Saddam Hussein is literally trying
to wipe a country off the face of the Earth.'' In other developments: _A U.S. diplomat in Baghdad said Tuesday up to 800 Americans and Britons will fly out of Iraqioccupied Kuwait this week, most of them women and children leaving their husbands behind. Saddam has said he is keeping foreign men as human shields against
attack. On Monday, a planeload of 164 Westerners arrived in Baltimore from Iraq. Evacuees spoke of food shortages in Kuwait, nighttime gunfire and Iraqi roundups
of young people suspected of involvement in the resistance. ``There is no law and order,'' said Thuraya, 19, who would not give her last name. ``A soldier can rape
a father's daughter in front of him and he can't do anything about it.'' _The State Department said Iraq had told U.S. officials that American males residing in Iraq
and Kuwait who were born in Arab countries will be allowed to leave. Iraq generally has not let American males leave. It was not known how many men the Iraqi
move could affect. _A Pentagon spokesman said ``some increase in military activity'' had been detected inside Iraq near its borders with Turkey and Syria. He said
there was little indication hostilities are imminent. Defense Secretary Dick Cheney said the cost of the U.S. military buildup in the Middle East was rising above the $1
billion-a-month estimate generally used by government officials. He said the total cost _ if no shooting war breaks out _ could total $15 billion in the next fiscal year
beginning Oct. 1. Cheney promised disgruntled lawmakers ``a significant increase'' in help from Arab nations and other U.S. allies for Operation Desert Shield.
Japan, which has been accused of responding too slowly to the crisis in the gulf, said Tuesday it may give $2 billion to Egypt, Jordan and Turkey, hit hardest by the
U.N. prohibition on trade with Iraq. ``The pressure from abroad is getting so strong,'' said Hiroyasu Horio, an official with the Ministry of International Trade and
Industry. Local news reports said the aid would be extended through the World Bank and International Monetary Fund, and $600 million would be sent as early as
mid-September. On Friday, Treasury Secretary Nicholas Brady visited Tokyo on a world tour seeking $10.5 billion to help Egypt, Jordan and Turkey. Japan has
already promised a $1 billion aid package for multinational peacekeeping forces in Saudi Arabia, including food, water, vehicles and prefabricated housing for nonmilitary uses. But critics in the United States have said Japan should do more because its economy depends heavily on oil from the Middle East. Japan imports 99
percent of its oil. Japan's constitution bans the use of force in settling international disputes and Japanese law restricts the military to Japanese territory, except for
ceremonial occasions. On Monday, Saddam offered developing nations free oil if they would send their tankers to pick it up. The first two countries to respond
Tuesday _ the Philippines and Namibia _ said no. Manila said it had already fulfilled its oil requirements, and Namibia said it would not ``sell its sovereignty'' for Iraqi
oil. Venezuelan President Carlos Andres Perez dismissed Saddam's offer of free oil as a ``propaganda ploy.'' Venezuela, an OPEC member, has led a drive among oilproducing nations to boost production to make up for the shortfall caused by the loss of Iraqi and Kuwaiti oil from the world market. Their oil makes up 20 percent
of the world's oil reserves. Only Saudi Arabia has higher reserves. But according to the State Department, Cuba, which faces an oil deficit because of reduced Soviet
deliveries, has received a shipment of Iraqi petroleum since U.N. sanctions were imposed five weeks ago. And Romania, it said, expects to receive oil indirectly from
Iraq. Romania's ambassador to the United States, Virgil Constantinescu, denied that claim Tuesday, calling it ``absolutely false and without foundation.''.
Cracks appeared in the U.N. trade embargo against Iraq.
The State Department reports that Cuba and Romania have
struck oil deals with Iraq as others attempt to trade with
Baghdad in defiance of the sanctions. Iran has agreed to
exchange food and medicine for Iraqi oil. Saddam has offered
developing nations free oil if they send their tankers to
pick it up. Thus far, none has accepted.
Japan, accused of responding too slowly to the Gulf crisis, has
promised $2 billion in aid to countries hit hardest by the Iraqi
trade embargo. President Bush has promised that
Saddam's aggression will not succeed.
7800 characters, 1300 words
Iraq
More examples