When Specialists and Generalists Work
Together: Overcoming Domain Dependence
in Sentiment Tagging
Alina Andreevskaia
Sabine Bergler
Concordia University
ACL 2008
Domain Adaptation
in Sentiment Research
General word lists from Internet vs.
corpus-based classifiers: out of
domain training data, not sufficiently
large training data => comparable
performance.
 Domain adaptation: general vs.
corpus-based?

Factors
Affecting System Performance
Training set size
 Level of analysis (sentence or
document?)



Bo
Pang
Sentence: snippets/sentences
Document domain/genre?
Agreement:
92.5%, 95.9%
k=0.74, 0.75
99%,K=0.99
Bing
Liu
Baseline
Corpus-based System (CBS)
Supervised statistical methods: acc
85-90% for movie reviews: large
volume of labeled data available.
 SVM: p/r 0.50-0.52 on sentence
level sentiment classification;
acc=74.9% for subjectivity on the
MPQA
 Naïve Bayes: similar with SVM

System Performance
on Texts vs. Sentences (acc, NB)
System Performance
across Domains
System Performance
on Different Domains (acc, SVM)
Lexicon Based Approach
WordNet, human-annotated
adjectives as seeds, learn additional
unigrams from synsets and glosses.
Evaluated by GI, acc=88%
 Method: 58 system runs from
unique 58 seed sets. P: +1, N: -1;
Net score normalized to –1~1.

Baseline
Lexicon-Based System (LBS): acc

The lexicon-based approach is
characterized by a bounded but
stable performance when the
system is ported across domains.
Integrating Corpus-based
and Dic-based Approaches
Cascaded classifiers: Pang and Lee
(04) – sentiment->polarity
 Many classifiers vote: Das and Chen
(2004): market sentiment on Yahoo!
Postings. (ternary, acc=62%)

Classifier Integration Procedure (1)
Train/Test positive and negative
sentences separately
 Significant at alpha=0.01
 This difference is domain
independent

Classifier Integration Procedure (2)






(1) Small amount of training data for CBS.
(2) Performance (precision) of CBS and
LBS are evaluated on training instances.
(3) The performances subtracts 50%
(chance level) to get the weight of CBS
and LBS
(4) Weights normalized
(5) Using weights of CBS and LBS to
combine them
(6) The category of greater score is
assigned to the sentence
Classifier Integration Procedure
Example
89.3-50 = 39.3 => 0.671
 69.3-50=19.3 => 0.329
 SenP=
0.329*ScoreLBS+0.671*ScoreCBS

System Evaluation

Significant at
alpha=0.01
except
Movies (0.05)
Discussion
Integration of two fairly different
classifier learning approaches yields
substantial gains
 Gains are most likely when

Errors are complementary
 Errors are not fully random
 System can identify low-precision
segment and reduce the weights.

Conclusion and Future Work




Combine classifiers into a single ensemble
Precision-based technique for assigning
weights for classifier results
Proposing a new method in situations
where the annotated in-domain data is
scarse and insufficient
Deploy more advanced classifiers and
feature selection techniques