Document Similarities
Anand Bahety
Cody Dunne
Project Idea
• Find similar segments of documents
Project Idea (cont…)
• Inexact matching
– Local alignment (Smith-Waterman, BLAST)
– Based on character
• Meaningless to score character differences
– Based on word
• Need a good scoring function
Project Idea (cont…)
• Scoring function based on word relationships
– Part of speech
• Noun -> pronoun (ok)
• Noun ->verb (worse)
– Synonyms – positive score
– Antonyms – negative score
– Network of word relationships
• WordNet – publicly available lexical English database
– Gaps
• Different numbers of adjectives/adverbs
• Prepositions, pronouns
Related Work
• Document versioning (Versioning Machine, etc…)
• Detecting plagiarism (Bagdis, etc…)
Potential Pitfalls
• False positives
• The Great Wall of China is very famous.
• The Fantastic Wall by XYZ is very famous.
– Pick correct word meanings
• False negatives
– Database isn’t perfect/complete
• Incomplete scoring function
– Only examines particular types of words
– Depends on order
• Limited to English
– EuroWordNet