Sentiment Analysis and Classification with Feature
Reduction using Principal Component Analysis
Algorithm on Textual Reviews
Ms. Anju Tiwari
Ms. Rashmi Shrivas
PG Student, Dept. of CSE
School of Engineering & IT
MATS University,
Aarang, Raipur (C.G.), India
Assistant Professor, Dept. of CSE
School of Engineering & IT
MATS University,
Aarang, Raipur (C.G.), India
anjutiwari60@gmail.com
ABSTRACT
Text Mining is used to extract previously unknown
information from different written resources. A key
element is used to link together the extracted
information to form new facts or new hypotheses to be
explored further by more conventional means of
experimentation. Sentiment analysis is useful in social
media monitoring to automatically characterize the
overall feeling or mood of consumers as reflected in
social media toward a specific brand or company and
determine whether they are viewed on the web in
positive or negative form. In customer relation
management especially in the context of complaint
management this form of analysis has been widely
adopted. Document-level sentiment classification is
used for expressing sentiment (positive or negative) for
automating the task of dividing a single topic textual
review. By using multi-theme document analyzing
sentiment is very tough and the classification accuracy
is less. In Support Vector Machine (SVM) algorithm,
categorization of sentiments approximately done by
document level classification using large number of
words. In proposed work, a new algorithm called
Principal Component Analysis Algorithm with parts of
speech tags is used to improve the classification
accuracy on the benchmark dataset of Movies reviews
dataset.
Keywords
Principal Component Analysis algorithm, Sentiment
analysis, opinion mining, Text classification, parts of
speech tags, Support Vector Machine, Term weighting,
Sentiment Fuzzy Classification.
INTRODUCTION
Mining is used to help people to extract valuable
information from large amount of data. Sentiment
analysis or opinion mining is the field of computational
(or automatic) study of people’s opinion expressed in
written language or text. In the news article domain
work on sentiment analysis has thus far been limited.
This has mainly been caused by 1) the difficulty in
separating good and bad news from positive and
negative sentiment, 2) news articles lacking a clearly
defined target, 3) the seeming necessity of background
knowledge [1] [20].
Sentiment analysis is the process used to
determine the attitude/opinion/emotion expressed by a
person about a particular topic. To identify and extract
subjective information in source materials sentiment
analysis or opinion mining uses natural language
processing and text analytics. The rise of social media
such as blogs and social networks has fuelled interest in
sentiment analysis. In order to identify the new
opportunities and to manage the reputations, business
people usually view the reviews/
ratings/
recommendations and other forms of online opinion [2].
Sentiment analysis can be used as a complement to
other systems such as question answering systems,
recommendation systems, and information extraction
[3]. Both individuals and organizations can take
advantage of sentiment analysis and opinion mining [3].
Sentiment analysis is all about to get the real
voice of people towards specific product, services,
organization, movies, news, events, issues and their
attributes [8] [9]. Sentiment analysis can be done at
three different levels : document level, sentence level
and feature (aspect) level [1] [5]. Document level
sentiment analysis means classifying the overall
sentiments expressed by the author in the whole
document text in positive, negative or neutral classes
[3] [5] [7] [8]. The sentence level sentiment analysis is
used to identify whether the sentence is subjective or
objective and then only subjective sentences are
determine to be positive, negative or neutral [3] [5] [7]
[8] [17]. An aspect based opinion polling system takes
as input a set of textual reviews and some predefined
aspects, and identifies the polarity of each aspect from
each review to produce an opinion poll [7]. Aspect level
sentiment analysis performs finer grained analysis [8].
Sentiment analysis includes branches of
computer science like Text Mining, Natural Language
Processing, Information Theory, Machine Learning and
Coding [3] [8]. We can categorized our data may be in
form of movie reviews, news articles, blogs, tweets,
product reviews etc. into positive, negative or neutral
sentiment according to the sentiment is expressed in
them, by using methods, approaches, techniques and
models of defined branches, [8]. The Sentiment
learning uses Machine Learning or Lexicon based
learning [2]. The different learning types are:
Supervised learning: Learning classifier from training
data and assign class labels to test data. Semisupervised learning:- Amalgamate both labeled and
unlabeled training data and. Unsupervised learning:Learning without training data. Classification is a
supervised procedure that learns to classify new
instances based on learning from a training set of
instances that have been properly labeled with the exact
classes. The piece of input data is formally called an
instance, and the categories are called classes. Text
Classification (TC) is one of the prime techniques to
deal with the textual data. TC systems are used in a
number of applications such as, filtering email
messages, classifying customer reviews for large ecommerce sites, web page classification for an internet
directory, evaluating exams paper answers and
organizing document databases in semantic categories
[16] [18].
2. Removal of Stop Words: A stop-list is the name
commonly given to a set or list of stop words. It is
typically language specific, although it may contain
some words. Any search engine or other natural
language processing system may contain a variety of
stop-lists. Some of the more frequently used stop words
for English include "a", "I", "it", "of", "the", "you", and
”and” these are generally regarded as 'functional words'
which meaningless. By ignoring the functional words,
when assessing the contents of natural language, the
meaning can be conveyed more clearly. Hence it is
practical to remove those words which appear too often
that support no information for the task. If the stop
word removal is applied, all the stop words in the
particular text file will not be loaded. If the stop word
removal is not applied, the stop word removal algorithm
will be disabled when the dataset is loaded.
RELATED WORK
Sentiment analysis is conducted either at the word,
phrase, sentence paragraph or document level, and one
typically distinguishes between supervised or
unsupervised approaches [1] [10]. The Various research
groups are exploring the ways to use Text mining and
sentiment analysis as next generation’s paradigm shift
[1]. Document level classification is most promising
topic in Sentiment analysis [2] [19]. The sentiment
analysis is typically performed on one single level, such
as entity-level, sentence-level, and document-level. In
entity-level lexicon is build and then by distinguishing
between prior and contextual polarity important features
are extracted based on that features sentiment analysis
is performed. At sentence-level and document level,
documents are not classified by topic, classified by
overall sentiments [5] [11] [12]. Up till now, utmost of
the previous research is performed at the specific level.
Also the focus is on the binary classification in terms of
positive and negative class. The work include sentiment
analysis of movies review, electronics and stock posting
on an investor bulletin, in which comments are
classified into positive or negative [5] [13] [14] [15]. In
this paper, we focus on the reduction of the features by
using an algorithm called principal component analysis
algorithm.
METHODOLOGY
An overview of steps and techniques used in sentiment
classification approaches, as shown in Figure 1.
I). Text Preprocessing
Text pre-processing techniques are divided into two
subcategories.
1. Tokenization: Textual data comprises block of
characters called tokens. The documents are separated
as tokens and used for further processing.
Figure 1: Sentiment Analysis Algorithms and Techniques
II). Text Transformation
The score value of each sentence from the source
document is calculated by summation of weight of each
term in the respective sentences. The weight of each
term is calculated by product of TF and IDF of that
word based on adjective word taken out from Parts of
speech tags. The TF and IDF are defined as
𝑁𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑡𝑖𝑚𝑒𝑠 𝑡ℎ𝑒 𝑎𝑑𝑗𝑒𝑐𝑡𝑖𝑣𝑒
𝑡𝑒𝑟𝑚 𝑜𝑐𝑐𝑢𝑟𝑠 𝑖𝑛 𝑑𝑜𝑐𝑢𝑚𝑒𝑛𝑡 (𝑑)
𝑇𝐹 (𝑡) =
𝑇𝑜𝑡𝑎𝑙 𝑁𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑎𝑑𝑗𝑒𝑐𝑡𝑖𝑣𝑒
𝑖𝑛 𝑑𝑜𝑐𝑢𝑚𝑒𝑛𝑡(𝑑)
𝐼𝐷𝐹(𝑡) = log {
𝑁𝐷
}
𝐷𝐹(𝑡)
Here ND means total number of document in the
document collection and DF (t) is number of documents
in which adjective term (t) occurs in the document
collection.
III). Feature Selection
Many statistical feature selection methods for document
level classification can also be used for sentiment
analysis. The simplest statistical approach for feature
selection is to use the most frequently occurring words
in the corpus as polarity indicators. The approaches’
majority for sentiment analysis involves a two-step
process: • Identify the parts of the document to
contribute the positive or negative sentiments. • Join
these parts of the document in ways that increase the
odds of the document falling into one of these two polar
categories.
IV). Feature Reduction
For feature reduction using principal component
analysis algorithm. Principal component analysis (PCA)
is a statistical procedure that uses an orthogonal
transformation to convert a set of observations of
possibly correlated variables into a set of values of
linearly uncorrelated variables called principal
components. The numbers of principal components is
less than or equal to the numbers of original variables.
This transformation is well-defined in such a way that
the first principal component has the largest possible
variance (that is, accounts for as much of the variability
in the data), and each successive component in turn has
the highest variance possible under the constraint that it
is orthogonal to (i.e., uncorrelated with) the preceding
components. The principal components (derived from
method) are orthogonal because they are the
eigenvectors of the covariance matrix, which is
symmetric. Principal Component Analysis (PCA) is
sensitive to the relative scaling of the original variables.
PCA steps:
1.
2.
3.

Transform a 𝑁 × 𝑑 matrix 𝑋 into a 𝑁 × 𝑚 matrix 𝑌.
Centralized the data (subtract the mean).
Calculate the 𝑑 × 𝑑 covariance matrix:
1
𝐶=
𝑋𝑇𝑋
𝑁−1
1
 𝐶𝑖,𝑗 =
𝑁−1
∑𝑁
𝑞=1 𝑋𝑞,𝑖 . 𝑋𝑞,𝑗
 𝐶𝑖,𝑗 (diagonal) is the variance of variable i.
 𝐶𝑖,𝑗 (off-diagonal) is the covariance between variables i
and j.
4. Calculate the eigenvectors of the covariance matrix
(orthonormal).
5. Select ‘m’ eigenvectors that correspond to the largest
‘m’ eigenvalues to be the new basis.
V). Sentiment Fuzzy Classification
Sentiment polarity is vague with regard to its
conceptual extension. There is not a clear boundary
between the concepts of “positive”, “neutral” and
“negative”. To better handle such intrinsic fuzziness in
sentiment polarity, sentiment classification we apply the
fuzzy set theory. To do so, we first redefine sentiment
classes as three fuzzy sets, and then apply existing
fuzzy distributions to construct membership functions
for the three sentiment fuzzy sets. A fuzzy set is defined
by a membership function; these functions can be any
arbitrary shape but are typically triangular or
trapezoidal. In our formulation, the entire opinionated
documents under discussion are represented as a sorted
set, denoted by X, in terms of their opinion weight
(calculated by TF-IDF).
VI). Parameters for evaluation
In the context of classification, True Positives (TP),
True Negatives (TN), False Negatives (FN) and False
Positives (FP) are used to compare the class labels
assigned to documents by a classifier with the classes
the items actually belongs to. True positive means,
which are truly classified as the positive terms. True
positives (TP) are examples that the classifier correctly
labeled as belonging to the positive class. True negative
(TN) is examples that the classifier correctly labeled as
belonging to the negative class, True negative means,
which are truly classified as the negative terms. False
positive (FP) are examples which were not labeled by
the classifier as belonging to the positive class but
should have been. At last there is False Negative (FN),
which is an example which was not labeled by the
classifier as belonging to the negative class but should
have been. Other evaluation measures like precision,
recall, F-measure, specificity and accuracy can easily be
calculated from these four variables.
Table 1.Contegency table
Correct Labels
Classified
Labels
Positive
Negative
Positive
TP (True
Positive)
FP (False
Positive)
Negative
FN (False
Negative)
TN (True
Negative)
1. Accuracy:
A common measure for classification performance is
accuracy, or its complement error rate. Accuracy is the
proportion of correctly classified examples to the total
number of examples, while error rate uses incorrectly
classified instead of correctly. However, one should be
careful to use only accuracy when one is using skewed
data
𝑇𝑃 + 𝑇𝑁
𝐴𝑐𝑐𝑢𝑟𝑎𝑐𝑦 =
𝑇𝑃 + 𝑇𝑁 + 𝐹𝑃 + 𝐹𝑁
2. Precision and recall:
Precision is used to measure exactness, whereas recall
is a measure of completeness.
𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 =
𝑇𝑃
𝑇𝑃 + 𝐹𝑃
𝑇𝑃
𝑅𝑒𝑐𝑎𝑙𝑙 =
𝑇𝑃 + 𝐹𝑁
3. F-Measure
F-Measure is the harmonic mean of precision and
recall. This gives a score that is a balance between
precision and recall.
𝐹=
2 × 𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 × 𝑅𝑒𝑐𝑎𝑙𝑙
(𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 + 𝑅𝑒𝑐𝑎𝑙𝑙)
CONCLUSION
As we know today’s world is becoming a tapered. We
get reaction of people for particular products, events,
issues very fast on web. It is difficult for human to
predict the movie review. To resolve this, the
document-level sentiment classification is used in the
existing system. It can be approximately classifies the
sentiment using the Bag of words. To make the
classification accurate, parts of speech can be used. A
new algorithm called principal component analysis
algorithm is proposed to improve classification
accuracy and performance of sentiment analysis.
REFERENCES
[1] Pal-Christian, Lars S. Hoysaeter, Lars S. Hoysaeter, Jon
Atle Gulla and Wei Wei, ‘Evaluating Feature Sets and
Classifiers for Sentiment Analysis of Financial News’, In
IEEE, WIC/ACM International Joint Conferences on Web
Intelligence (WI) and Intelligent Agent Technologies (IAT)
2014.
[8] Jalaj S. Modha, Prof. & Head Gayatri, S. Pandi, Sandip, J.
Modha, ‘Automatic Sentiment Analysis for Unstructured
Data’, In International Journal of Advanced Research in
Computer Science and Software Engineering (IJARCSSE)
2013.
[9] Bing Liu, ‘Sntiment Analysis and Opinion Mining’, In
Morgan and Claypool Publishers, May 2012, p. 18-19, p. 2728, p. 44-45, p. 47, p. 90-101.
[10] R. Feldman, ‘Techniques and Applications for Sentiment
Analysis’, In Communications of the ACM, 2013, vol. 56, no.
4, p. 82-89.
[11] A. Abbasi, A. Salem and H. Chen, ‘Sentiment Analysis
in multiple languages : In Feature Selection for Opinion
Classification in Web Forums’, In ACM Transactions on
Information System, June 2008, Vol. 26.
[12] Pang Bo, Lillian Lee, and Shivakumar Vaithyanathan,
‘Thumbs Up ? Sentiment Classification using Machine
Learning Techniques’, In Conference of EMNLP, 2002.
[13] Dave, K., Lawrence, S., & Pennock, D. M., ‘Mining the
Peanut Gallery : Opinion Extraction and Semantic
Classification of Product Reviews’, In The Proceedings of
12th International WWW Conference, Budapest, Hungary,
May 2003, p. 519-528.
[14] Turney, P. D., ‘Thumbs Up or Thumbs Down ? Semantic
Orientation Applied to Unsupervised Classification of
Reviews , ” 40th Annual Meeting of the Association for
Computational Linguistics (ACL), Philadelphia, 2002, pp.
417-424.
[2] Ms. K. Mouthami, Ms. K. Nirmala Devi, Dr. V. Murli
Bhaskaran, ‘Sentiment Analysis and Classification Based On
Textual Reviews’, IEEE 2013.
[15] Chaovalit, P. and L. Zhou, ‘Movie Review Mining : A
Comparision between Supervised and Unsupervised
Classification Approaches’, In The Proceedings of 38th
Hawaii International Conference on System Science (HICSS),
2005.
[3] Eric Rolland, Mohsen Farhadloo, ‘Multiclass Sentiment
Analysis with Clustering and Score Representation’, In IEEE
13th International Conference on Data Mining Workshops
2013 .
[16] Wang Y., & Wang X., ‘New Approach to Feature
selection in Text Classification’, In The Proceedings of 4th
International Conference on Machine Learning and
Cybernetics. IEEE, 2005, pp. 145-189.
[4] Lizhen Liu, Hanshi Wang, Xinhui Nie, ‘Toward a Fuzzy
Domain Sentiment Ontology Tree for Sentiment Analysis’,
In IEEE 5th International Conference on Image and Signal
Processing (CISP) 2012.
[17] Whitelaw C., Garg N., & Argamon S., Using appraisal
groups for sentiment analysis, In The Proceedings of 14th
ACM International Conference on Information and
Knowledge Management, 2005, p. 625–631.
[5] Dr. M. S. Patwardhan, N. D. Valakunde, ‘Multi Aspect
and Multi Class Based Document Sentiment Analysis of
Educational Data Catering Accreditation Process’, In IEEE
International Conference on Cloud and Ubiquitous Computing
and Emerging Technologies.
[18] Yang. Y, X. Liu, ‘A re-examination of text categorization
methods’, In The Proceedings of ACM SIGIR Conference on
Research and Development in Information Retrieval (SIGIR),
ACM, New York, USA, 1999, p. 42–49.
[6] Mrs. Vijayalaxmi M, Mrs. Sangeeta Oswal, Mrs.
Deepshikha Chaturvedi, Mrs. Shalu Chopra, ‘The How,
When and Why of Sentiment Analysis’, In I. J. Computer
Technology and Applications (IJCTA), Vol 4 (4), p. 660-665.
[19] Yi, J. Nasukawa, T. Niblack, & Bunescu R., ‘Sentiment
analyzer: extracting sentiments about a given topic using
natural language processing techniques’, In The Proceedings
of 3rd IEEE International Conference on Data Mining
(ICDM), USA,2003, p. 427– 434.
[7] Asst. Prof. A Kowcika, Aditi Gupta, Karthik Sondhi,
Raunaq Kumar, Nishit Shivhre, ‘Sentiment Analysis for
Social Media’, In The International Journal of Advanced
Research in Computer Science and Software Engineering
(IJARCSSE) 2013.
[20] A. Abbasi, Z. Zhang and H. Chen, S. France, ‘Selecting
Attributes for Sentiment Classification using Feature relation
networks’, In Knowledge and Data Engineering IEEE
Transactions on, vol. 23, no. 3 pp. 447-462, 2011.