Using Machine Learning to Discover
and Understand Structured Data
William W. Cohen
Machine Learning Dept. and Language Technologies Inst.
School of Computer Science
Carnegie Mellon University
Outline
• Information integration:
– Some history
– The problem, the economics, and the
economic problem
• “Soft” information integration
• Concrete uses of “soft” integration
– Classification
– Collaborative filtering
– Set expansion
When are two entities the same?
•
•
•
•
•
•
Bell Labs [1925]
Bell Telephone Labs
AT&T Bell Labs
A&T Labs
AT&T Labs—Research
AT&T Labs Research,
Shannon Laboratory
• Shannon Labs
• Bell Labs Innovations
• Lucent Technologies/Bell
Labs Innovations
History of Innovation: From 1925
to today, AT&T has attracted some
of the world's greatest scientists,
engineers and developers….
[www.research.att.com]
Bell Labs Facts: Bell Laboratories,
the research and development arm
of Lucent Technologies, has been
operating continuously since 1925…
[bell-labs.com]
In the once upon a time days of the First Age of Magic, the prudent
sorcerer regarded his own true name as his most valued possession but
also the greatest threat to his continued good health, for--the stories go-once an enemy, even a weak unskilled enemy, learned the sorcerer's
true name, then routine and widely known spells could destroy or
enslave even the most powerful. As times passed, and we graduated to
the Age of Reason and thence to the first and second industrial
revolutions, such notions were discredited. Now it seems that the Wheel
has turned full circle (even if there never really was a First Age) and we
are back to worrying about true names again:
The first hint Mr. Slippery had that his own True Name might be known-and, for that matter, known to the Great Enemy--came with the
appearance of two black Lincolns humming up the long dirt driveway ...
Roger Pollack was in his garden weeding, had been there nearly the
whole morning.... Four heavy-set men and a hard-looking female piled
out, started purposefully across his well-tended cabbage patch.…
This had been, of course, Roger Pollack's great fear. They had
discovered Mr. Slippery's True Name and it was Roger Andrew Pollack
TIN/SSAN 0959-34-2861.
Deduction via co-operation
User
Economic issues:
Integrated KB
Site1
Site3
Site2
KB1
KB3
KB2
Standard Terminology
• Who pays for integration?
Who tracks errors &
inconsistencies? Who fixes
bugs? Who pushes for clarity
in underlying concepts and
object identifiers?
• Standards approach 
publishers are responsible 
publishers pay
• Mediator approach: 3rd party
does the work, agnostic as to
cost
Traditional approach:
Linkage
Queries
Uncertainty about what to link
must be decided by the integration
system, not the end user
WHIRL approach:
SELECT R.a,S.a,S.b,T.b FROM R,S,T
WHERE R.a=S.a and S.b=T.b
Link items as
needed by Q
Query Q
R.a
S.a
S.b
T.b
Anhai
Anhai
Doan
Doan
Dan
Dan
Weld
Weld
Weaker links: those
agreeable to some users
William
Will
Cohen
Cohn
Steve
Steven
Minton
Mitton
even weaker links…
William
David
Cohen
Cohn
Strongest links: those
agreeable to most users
WHIRL approach:
SELECT R.a,S.a,S.b,T.b FROM R,S,T
WHERE R.a~S.a and S.b~T.b
Link items as
needed by Q
Incrementally produce a
ranked list of possible links,
with “best matches” first. User
(or downstream process)
decides how much of the list to
generate and examine.
(~ TFIDF-similar)
Query Q
R.a
S.a
S.b
T.b
Anhai
Anhai
Doan
Doan
Dan
Dan
Weld
Weld
William
Will
Cohen
Cohn
Steve
Steven
Minton
Mitton
William
David
Cohen
Cohn
WHIRL queries
• Assume two relations:
review(movieTitle,reviewText): archive of reviews
listing(theatre, movieTitle, showTimes, …): now showing
The
Hitchhiker’s
Guide to the
Galaxy, 2005
This is a faithful re-creation of
the original radio series – not
surprisingly, as Adams wrote
the screenplay ….
Men in
Black, 1997
Will Smith does an excellent
job in this …
Space Balls,
1987
Only a die-hard Mel Brooks
fan could claim to enjoy …
…
…
Star Wars
Episode
III
The
Senator
Theater
1:00,
4:15, &
7:30pm.
Cinderella
Man
The
Rotunda
Cinema
1:00,
4:30, &
7:30pm.
…
…
…
WHIRL queries
• “Find reviews of sci-fi comedies [movie domain]
FROM review SELECT * WHERE r.text~’sci fi comedy’
(like standard ranked retrieval of “sci-fi comedy”)
• “ “Where is [that sci-fi comedy] playing?”
FROM review as r, LISTING as s, SELECT *
WHERE r.title~s.title and r.text~’sci fi comedy’
(best answers: titles are similar to each other – e.g., “Hitchhiker’s
Guide to the Galaxy” and “The Hitchhiker’s Guide to the Galaxy,
2005” and the review text is similar to “sci-fi comedy”)
WHIRL queries
•
•
Similarity is based on TFIDF rare words are most important.
Search for high-ranking answers uses inverted indices….
- It is easy to find the (few) items that match on “important” terms
- Search for strong matches can prune “unimportant terms”
The
Star Wars Episode III
Hitchhiker’s Guide to the Galaxy,
2005
Hitchhiker’s Guide to the Galaxy
Men in Black, 1997
Cinderella Man
…
Space Balls, 1987
…
Years are common in the
review archive, so have
low weight
hitchhiker
movie00137
the
movie001,movie003,movie007,movie008,
movie013,movie018,movie023,movie0031,
…..
Outline
• Information integration:
– Some history
– The problem, the economics, and the
economic problem
• “Soft” information integration
• Concrete uses of “soft” integration
– Classification
– Collaborative filtering
– Set expansion
Outline
• Information integration:
– Some history
– The problem, the economics, and the
economic problem
• “Soft” information integration
• Concrete uses of “soft” integration
– Classification
– Collaborative filtering
– Set expansion
Outline
• Information integration:
– Some history
– The problem, the economics, and the economic
problem
• “Soft” information integration
• Concrete uses of “soft” integration
– Classification
– Collaborative filtering
– Set expansion: using generalized notion of
similarity
Recent work: non-textual similarity
“William W. Cohen, CMU”
cohen
dr
william
w
“Dr. W. W. Cohen”
“Christos
Faloutsos,
CMU”
cmu
“George H.
W. Bush”
“George W. Bush”
Recent work
• Personalized PageRank aka Random Walk with
Restart:
– Similarity measure for nodes in a graph, analogous
to TFIDF for text in a WHIRL database
– natural extension to PageRank
– amenable to learning parameters of the walk
(gradient search, w/ various optimization
metrics):
• Toutanova, Manning & NG, ICML2004; Nie
et al, WWW2005; Xi et al, SIGIR 2005
– various speedup techniques exist
– queries:
Given type t* and node x, find y:T(y)=t* and y~x
Learning to Search Email
Einat Minkov, CMU; Andrew Ng, Stanford
[SIGIR 2006, CEAS 2006,
WebKDD/SNA 2007]
CALO
Sent
To
Term In
Subject
William
graph
proposal
CMU
6/17/07
6/18/07
einat@cs.cmu.edu
Tasks that are like similarity queries
Person name
disambiguation
[ term “andy”
file
msgId ]
“person”
Threading
 What are the adjacent messages in
this thread?
 A proxy for finding “more
messages like this one”
Alias finding
What are the email-addresses of
Jason ?...
[ file msgId ]
“file”
[ term Jason ]
“email-address”
Meeting
attendees finder
Which email-addresses (persons)
should I notify about this meeting?
[ meeting mtgId ]
“email-address”
Learning to search better
Task T
(query class)
Query a
+ Rel. answers a
Query b
+ Rel. answers b
…
Query q
+ Rel. answers q
GRAPH WALK
 node rank 1
 node rank 1
 node rank 1
 node rank 2
 node rank 2
 node rank 2
 node rank 3
 node rank 3
 node rank 3
 node rank 4
 node rank 4
 node rank 4
…
…
…
 node rank 10
 node rank 10
 node rank 10
 node rank 11
 node rank 11
 node rank 11
 node rank 12
 node rank 12
 node rank 12
…
…
…
 node rank 50
 node rank 50
 node rank 50
Learning
Node re-ordering:
train task
Graph
walk
Feature
generation
Learn
re-ranker
Re-ranking
function
Learning Approach
Node re-ordering:
train task
Graph
walk
Feature
generation
Learn
re-ranker
test task
Graph
walk
Feature
generation
Score by
re-ranking
function
[Collins & Koo, CL 2005; Collins, ACL 2002]
Re-ranking
function
Boosting
Voted Perceptron;
RankSVM;
PerceptronCommittees;
…
[Joacchim KDD 2002,
Elsas et al WSDM 2008]
Learning approaches
Edge weight tuning:
Graph
walk
Weight
update
Theta*
Learning approaches
Edge weight tuning:
Graph
walk
task
Weight
update
Theta*
[Diligenti et al, IJCAI
2005; Toutanova &
Ng, ICML 2005; … ]
Graph
walk
Question: which is better?
Node re-ordering:
Graph
walk
Feature
generation
Learn
re-ranker
Graph
walk
Feature
generation
Score by
re-ranking
function
Re-ranking
function
Boosting;
Voted Perceptron
Results on one task
100%
80%
Recall
PERSON NAME DISAMBIGUATION
Mgmt. game
60%
40%
20%
0%
1
2
3
4
5
6
Rank
7
8
9
10
Results on several tasks (MAP)
0.85
0.8
0.75
Name
disambiguation
0.7
0.65
*
*
*
0.6
0.55
0.5
0.45
++
0.4
M.game
Threading
sager
* *
0.85
0.8
*
0.75
0.7
0.65
0.6
Shapiro
*
*
**
*
*
0.55
0.5
0.45
0.4
+ +
M.game
Alias finding
0.85
0.8
0.75
0.7
0.65
0.6
0.55
0.5
0.45
0.4
Meetings
+
Farmer
Germany
1.
2.
3.
•
•
•
Canon
Nikon
Olympus
Set Expansion using the Web
Richard Wang, CMU
Fetcher: download web pages from the Web
Extractor: learn wrappers from web pages
Ranker: rank entities extracted by wrappers
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
Pentax
Sony
Kodak
Minolta
Panasonic
Casio
Leica
Fuji
Samsung
…
The Extractor
• Learn wrappers from web documents
and seeds on the fly
– Utilize semi-structured documents
– Wrappers defined at character level
• No tokenization required; thus language
independent
• However, very specific; thus page-dependent
– Wrappers derived from document d is applied to d only
<li class="ford"><a href="http://www.curryauto.com/">
<img src="/common/logos/ford/logo-horiz-rgb-lg-dkbg.gif" alt="3"></a>
<ul><li class="last"><a href="http://www.curryauto.com/">
<span class="dName">Curry Ford</span>...</li></ul>
</li>
<li class="honda"><a href="http://www.curryauto.com/">
<img src="/common/logos/honda/logo-horiz-rgb-lg-dkbg.gif" alt="4"></a>
<ul><li><a href="http://www.curryhonda-ga.com/">
<span class="dName">Curry Honda Atlanta</span>...</li>
<li><a href="http://www.curryhondamass.com/">
<span class="dName">Curry Honda</span>...</li>
<li class="last"><a href="http://www.curryhondany.com/">
<span class="dName">Curry Honda Yorktown</span>...</li></ul>
</li>
<li class="acura"><a href="http://www.curryauto.com/">
<img src="/curryautogroup/images/logo-horiz-rgb-lg-dkbg.gif" alt="5"></a>
<ul><li class="last"><a href="http://www.curryacura.com/">
<span class="dName">Curry Acura</span>...</li></ul>
</li>
<li class="nissan"><a href="http://www.curryauto.com/">
<img src="/common/logos/nissan/logo-horiz-rgb-lg-dkbg.gif" alt="6"></a>
<ul><li class="last"><a href="http://www.geisauto.com/">
<span class="dName">Curry Nissan</span>...</li></ul>
</li>
<li class="toyota"><a href="http://www.curryauto.com/">
<img src="/common/logos/toyota/logo-horiz-rgb-lg-dkbg.gif" alt="7"></a>
<ul><li class="last"><a href="http://www.geisauto.com/toyota/">
<span class="dName">Curry Toyota</span>...</li></ul>
</li>
Building a Graph
“ford”, “nissan”, “toyota”
Wrapper #2
find
northpointcars.com
extract
curryauto.com
“chevrolet”
22.5%
Wrapper #3
“honda”
26.1%
“acura”
34.6%
derive
Wrapper #1
“volvo chicago”
8.4%
Wrapper #4
“bmw pittsburgh”
8.4%
• A graph consists of a fixed set of…
– Node Types: {seeds, document, wrapper, mention}
– Labeled Directed Edges: {find, derive, extract}
• Each edge asserts that a binary relation r holds
• Each edge has an inverse relation r-1 (graph is cyclic)
Minkov et al. Contextual Search and Name Disambiguation in Email using Graphs. SIGIR 2006
Top three mentions are the seeds
Try it out at http://rcwang.com/seal
Relational Set Expansion
Seeds
Additional relevant research
• Alon Halevey and friends:
– “Pay as you go”, “on the fly”, data integration (e.g., SIGMOD 98):
integrate partially, then allow user to perform search to make up for
inaccuracy of result
• Anhai Doan and friends:
– “Best effort” information extraction (SIGMOD 98): write an approximate
program for extraction from web pages, then allow user to perform search
to make up for inaccuracy of result
• Semi-structured extensions:
– Kushmeric’s ELIXIR (SIGIR 2001); Bernstein’s iSPARQL (eg ESWC
2008)
• Soft joins:
– Gravano et al WWW2003: Text Joints in an RDMS for Web Data
Integration
– Bayardo et al, WWW2007: Scaling up all-pairs similarity search.
– Koudas et al, SIGMOD 2006: Record linkage: similarity measures and
algorithms (survey)
Outline
• Information integration:
– Some history
– The problem, the economics, and the
economic problem
• “Soft” information integration
• Concrete uses of “soft” integration
– Classification
– Collaborative filtering
– Set expansion
– Questions?