Rotten or Fresh?
NLP applied to Sentiment Analysis
Two NLP tasks
• Part-of-speech tagging: marking up a word in
a text (corpus) as corresponding to a
particular part of speech, based on both its
definition, as well as its context
• Negation scope analysis: determining which
word a “negation word” (such as “no” or
“not”) binds to
Sentiment Analysis
• Aims to determine the attitude of a speaker or
a writer with respect to some topic
• Well studied topic in both ML and NLP
• Important not only from research perspective,
but also for many companies.
Question
• Can part-of-speech tagging and negationscope analysis be used to improve sentiment
analysis tools?
Rotten Tomatoes
• Aggregate movie and TV database
• Includes many reviews for each movie,
including a “fresh” or “rotten” indicator for
each review
• Ideal for training sentiment analysis classifier,
since “ground truth” is given.
• Offers API which gives “quote” of movie
review and “rotten/fresh”
Sentiment Analysis
• Simple open source PHP tool (phpInsight)
• Uses positive/neutral/negative word
dictionary
• Assigns positive/neutral/negative score to
entire quote, proportional to number of
positive/neutral/negative words contained
POS tagger
• Open source PHP interface to Stanford tagger,
which we discussed in class
• Generally accepted as one of the most widely
used POS tagger
Negation scope analysis
• Start with dictionary of negation words: “'not',
'isn\'t', 'wasn\'t', 'aren\'t', 'neither', 'nor‘”
• Use heuristics from relevant paper [1] to
decide which word to bind to
• Negation word is concatenated to bound word
and added to opposite sentiment dictionary
• For example, “notbad” would be added to
positive dictionary
Negation scope analysis
• Negations tend to bind to adjectives/verbs
• Also tend to bind to words proceeding
negation word
• Search window of five words after negation
word, looking for first adjective/verb
• Simple, but effective heuristic
Naïve Bayes Classifier
• Used Naïve Bayes classifier from Python’s
NLTK toolkit to predict “rotten/fresh”-ness
from sentiment scores
• 500 pos/neg/neutral scores along with their
rotten/freshness
• Split into 400 training and 100 test examples
• Data reshuffled 200 times and average
precision was taken
Methods
• Method 1: Quote taken directly from
RottenTomatoes
• Method 2: Quote preprocessed using POS
tagger, only adjectives passed to Sentiment
Analysis
• Method 3: Quote preprocessed using POS,
adjectives/verbs found and used to assist in
negation scope analysis
Comparison of precision by method
• 64.2%
• 65.2%
• 66.6%
Rotten
Tomatoes
POS
tagger
Negation
scope
analysis
Sentiment
Analysis
Freshness
Classifier
Conclusions
• Adjectives-only gave us best results. We
hypothesize this is due to higher information
content of adjectives, resulting in less noise in
data
• Negation scope analysis increase not as much
as adjective. Hypothesize this is due to
relatively rare appearance of negations within
sample quote.
Conclusions
• Although increases not very large, precision
was aggregated over 200 runs, so confidence
is still large
• We conclude that POS tagging and negation
scope analysis can improve sentiment analysis
using simple “bag-of-words” model
Fun Examples/Demo!
References
• 1.http://www.researchgate.net/publication/4
8340764_Scope_of_negation_detection_in_se
ntiment_analysis