International Journal of Computer Application
(Special issue- Issue 5, Volume 2 (January 2015)
Available online on http://www.rspublication.com/ijca/ijca_index.htm
ISSN: 2250-1797
Twitter Sentimental Analysis using Hadoop
Kushal Sharma#1, Prashant Singh#2, Sachin Mote#3,
Sudarshan Patil#4, Vilas Khedekar#5
#1 GHRCEM, Wagholi, Pune, 8390371747 .
#2, GHRCEM, Wagholi, Pune, 9762447134 .
#3, GHRCEM, Wagholi, Pune, 7875817008 .
#4, GHRCEM, Wagholi, Pune, 8446425690 .
#5, GHRCEM, Wagholi, Pune, 9762801601 .
ABSTRACT
Twitter is a famous micro blogging service where users express their opinions on
different topics. The messages are called as tweets. There is a tremendous rise in social
networking where people share millions of thoughts daily. The aim of this project is to build
an algorithm that can accurately classify Tweets as positive or negative with respect to a
specific subject. The proposed system uses dictionary based approach to determine the
semantic orientation of Tweets. Twitter Sentiment Analysis is used to understand how the
public feels about something at a particular moment in time and also tracks how this opinion
changes over time. This type of sentiment analysis is useful for consumers to research a
product or service, or marketers to research public opinion of their organisation. The project
uses Hadoop framework for distributed storage and processing of twitter data.
Key words: Hadoop, ZooKeeper, Hive, MapReduce, Flume, Hbase, HDFS, Tweets, data
injection, Microblogging, Sentiment analysis.
Corresponding Author: Mr.Vilas Khehdekar, Assistant Professor in G.H.Raisoni
College of Engineering and Management, Wagholi, Pune.
INTRODUCTION
Social media is already a significant part of the information ecosystem. Also social
media platforms and applications gain widespread adoption with unprecedented reach to
users, clients, electorate, business, government, and non-profit organizations alike, interest in
social media from all walks of life has been skyrocketing from both application and research
perspectives. For-profit businesses are tapping into social media as both a rich source of
CONFERENCE PAPER
National level conference on
"Advances in Networking, Embedded System and Telecommunication 2015(ANEC-2015)"
On 6-8 Jan 2015 organized by
" G.H.Raisoni College of Engg. & Management, Wagholi, Pune, Maharashtra, India."
Page 234
International Journal of Computer Application
(Special issue- Issue 5, Volume 2 (January 2015)
Available online on http://www.rspublication.com/ijca/ijca_index.htm
ISSN: 2250-1797
information and a business- execution platform for product design and innovation, consumer
and stakeholder relations management, and marketing. For them, social media is an essential
component of the next-generation business intelligence platform. For politicians, political
parties, and governments, social media represents the ideal vehicle and information base to
gauge public opinion on policies and political positions as well as to build community
support for candidates running for public offices. Public-health officials could potentially use
social media as valuable, early clues about disease outbreaks and to provide feedback on
public-health policies and response measures. For home- land security and intelligence
analysis communities, social media presents immense opportunities to study terrorist group
behavior, including their recruiting and public relation schemes and the grounding social and
cultural contexts. Even think 14 tanks and social science and business researchers are
conceptually using social media as an unbiased sensor network and a laboratory for natural
experimentation, providing valuable indicators and helping test hypotheses about social
production and interactions as well as their economic, political, and societal implications.
For many individuals, social media has become a unique information source to deal
with information and cognitive-overload problems, find answers to specific questions, and
determine more precious opportunities for social and financial exchange. In addition, it has
become a platform for them to network and contribute to all kinds of dynamic dialogues by
sharing their expertise and opinions. It is safe to claim that social media has already
penetrated a spectrum of applications with remarkable impact. Given the continued interest
and the ever-growing information and meta-information generated through social media, it is
expected to continue enabling new exciting applications and revolutionizing many existing
systems.
RELATED WORK
The sentiment analysis has been of great interest since ages. Social media, being a
great source to express thoughts, opinions and views on almost every domain has brought a
new challenge to analyze the sentiments.
Twitter provides its conventions which makes it different from other social media
websites. Consider an example, 'RT @modi is our new #PM'. A user can post to another user
using @ as used in @modi. # describes a subject or domain as in #PM and RT is added at the
begining to show that the tweet is retweeted or reposted.
In the context of knowledge discovery, the two fundamental data mining tasks which
can be considered in conjunction with Twitter data are graph mining based on analysis of the
links amongst messages and text mining based on analysis of the messages' actual text.
Twitter graph mining has been used to tackle several interesting problems. There is
Measuring user influence and dynamics of popularity. There are three measures of inuence:
in degree, retweets and mentions. Cha et al. [5] show that popular users who have high in
degree are not necessarily inuential in terms of retweets or mentions, and that impact is
gained through concerted effort such as limiting tweets to a single topic. Next is Community
discovery and formation. Java et al. [1] found communities using HyperText Induced Topic
Search (HITS) [2], and the Clique Percolation Method [8]. Romero and Kleinberg [3] analyze
the formation of links in Twitter via the directed closure process. { Social information
CONFERENCE PAPER
National level conference on
"Advances in Networking, Embedded System and Telecommunication 2015(ANEC-2015)"
On 6-8 Jan 2015 organized by
" G.H.Raisoni College of Engg. & Management, Wagholi, Pune, Maharashtra, India."
Page 235
International Journal of Computer Application
(Special issue- Issue 5, Volume 2 (January 2015)
Available online on http://www.rspublication.com/ijca/ijca_index.htm
ISSN: 2250-1797
di_usion. De Choudhury et al. [7] study how data sampling strategies impact the discovery of
information di_usion. There are also a number of interesting tasks that have been tackled
using Twitter text mining: sentiment analysis, which is the application we reflect in this
paper, classi_cation of tweets into classes, clustering of tweets and trending topic exposure.
Considering sentiment analysis [4, 6], O'Connor et al. [9] found that surveys of
consumer con_dence and political opinion correlate with sentiment word occurrences in
tweets, and propose text stream mining as a substitute for traditional polling. Jansen et al.
[10] discuss the implications for organizations ofusing micro-blogging as part of their
marketing strategy. Pak et al. [11] used classification based on the multinomial na• _ve
Bayes classi_er for sentiment analysis.
Go et al. [12] compared multinomial naive Bayes, a maximum entropy classi_er, and
a linear support vector machine; they all exhibited broadly comparable accuracy on their test
data, but small di_erences could be observed depending on the features used.
EXISTING SYSTEM
Most of the systems are based on classifier algorithms which are trained using
annotated text data. Naive bayes, Support vector machines and K-nearest neighbours are
some of the popular algorithms. However, it is not clear that which of these algorithms are
most appropriate to perform sentiment analysis. In this proposed system we are using
dictionary based and corpus based methods to classify the tweets.
Figure 1: Proposed Architecture for sentiment analysis
CONFERENCE PAPER
National level conference on
"Advances in Networking, Embedded System and Telecommunication 2015(ANEC-2015)"
On 6-8 Jan 2015 organized by
" G.H.Raisoni College of Engg. & Management, Wagholi, Pune, Maharashtra, India."
Page 236
International Journal of Computer Application
(Special issue- Issue 5, Volume 2 (January 2015)
Available online on http://www.rspublication.com/ijca/ijca_index.htm
ISSN: 2250-1797
Opinion words are the words that people use to express their opinion which could be
positive, negative or neutral. The proposed system finds the semantic orientation of the
opinion words in tweets, using a novel hybrid approach involving both corpus-based and
dictionary-based techniques. The features like emoticons and capitalization are also
considered as they have recently become a major part of the cyber language.
To discover the opinion direction, we extract the opinion words in the tweets and then
find out their orientation which decides whether each opinion word reflects a positive
sentiment, negative sentiment or a neutral sentiment. In proposed system, we are considering
the opinion words as the combination of the adjectives along with the verbs and adverbs. We
find the semantic orientation of adjectives using corpus-based method and then employ
dictionary-based method to find the semantic orientation of verbs and adverbs. The overall
tweet sentiment is then calculated using a linear equation given in the upcoming section,
which includes emoticon scoring also.
PROPOSED SYSTEM
People use positive, negative or neutral words to express their opinions. These words
are called as Opinion words. To find the semantic orientation of the opinion words in tweets,
we propose a dictionary-based technique. We also consider features like emoticons and
capitalization as they have recently become a large part of the cyber language. Fig.1gives the
architectural overview of the proposed system.
To uncover the opinion direction, we will first extract the opinion words in the tweets
and then find out their orientation, i.e., to decide whether each opinion word reflects a
positive sentiment, negative sentiment or a neutral sentiment. In our work, we are considering
the opinion words as the combination of the adjectives along with the verbs and adverbs. The
corpus-based method is then used to find the semantic orientation of adjectives and the
dictionary-based method is employed to find the semantic orientation of verbs and adverbs.
The overall tweet sentiment is then calculated using a linear equation which incorporates
emotion intensifiers too.
The following sub-sections explain the details of the proposed system:
Tweets Pre-processing We prepare a separate transaction file which contains opinion
indicators, namely the adjective, adverb and verb along with emoticons. We have taken a
sample set of emoticons and manually assigned opinion strength to them. In addition, we also
identify some emotion intensifiers, such as, the percentage of the tweet in Caps, the length of
repeated sequences & the number of exclamation marks, amongst others. Thus, we preprocess all the tweets as follows:
Step 1 - Removal of all URLs (e.g. www.sample.com), hash tags (e.g. #subject),
targets (@uname) and special Twitter words (“e.g. RT”).
Step 2 - Replacement of all the emoticons with their sentiment polarity (Table 1).
c) Removal of all punctuations after counting the number of exclamation marks.
d) Using a Dictionary where we tag the adjectives, verbs and adverbs.
CONFERENCE PAPER
National level conference on
"Advances in Networking, Embedded System and Telecommunication 2015(ANEC-2015)"
On 6-8 Jan 2015 organized by
" G.H.Raisoni College of Engg. & Management, Wagholi, Pune, Maharashtra, India."
Page 237
International Journal of Computer Application
(Special issue- Issue 5, Volume 2 (January 2015)
Available online on http://www.rspublication.com/ijca/ijca_index.htm
ISSN: 2250-1797
Table 1. Scoring Module
Emotion
:D
BD
XD
\m/
:), =), :-)
:*
:|
:\
:(
</3
B(
:’(
X-(
Meaning
Big grin
Big grin with glasses
Laughing
Hi 5
Happy, Smile
Kiss
Straight face
Undecided
Sad
Broken heart
Sad with glasses
Cry
Angry
Strength
1
1
1
1
0.5
0.5
0
0
-0.5
-0.5
-0.5
-1
-1
Scoring Module The adjectives, verbs and adverbs are opinion carriers. Next we find the
semantic score of these opinion carriers. As mentioned in previous section, in our approach
we use corpus-based method to find the semantic orientation of adjectives and the dictionarybased method to find the semantic orientation of verbs and adverbs.
Semantic Scoring of Adverbs and Verbs Although, it is possible to compute the sentiment
of a certain texts based on the semantic orientation of the adjectives, but including adverbs is
imperative. The primary reason is that there are some adverbs in linguistics (such as “not”)
which are very essential to be taken into consideration as they would completely change the
meaning of the adjective which may otherwise have conveyed a positive or a negative
orientation.
Consider one example; One user says, “This is a good movie” and;
Other says, “This is not a good movie”
In this example, if we had not considered the adverb “not”, then both the sentences would
have given positive review. On the contrary, first sentence gives the positive review and the
second sentence gives the negative review. Further, the strength of the sentiment cannot be
measured by merely considering adjectives alone as the opinion words. In other words, an
adjective cannot alone convey the intensity of the sentiment with respect to the document in
question. Therefore, we take into consideration the adverb strength which modify the
adjective; in turn modifying the sentiment strength. Adverb strength helps in assessing
whether a document gives a perfect positive opinion, strong positive opinion, a slight positive
opinion or a less positive opinion.
For example; One user says, “This is a very good movie” and ;
Other says, “This is a good movie”
Some groups of verbs also convey sentiments and opinions (e.g. love, like) and are essential
to finding the sentiment strength of the tweet. As adverbs and verbs are not dependent on the
domain, we use dictionary methods to calculate their semantic orientation.
The seed lists of positive and negative adverbs and verbs whose orientation we know is
created and then grown by searching in WordNet [17]. Based on intuition, we assign the
CONFERENCE PAPER
National level conference on
"Advances in Networking, Embedded System and Telecommunication 2015(ANEC-2015)"
On 6-8 Jan 2015 organized by
" G.H.Raisoni College of Engg. & Management, Wagholi, Pune, Maharashtra, India."
Page 238
International Journal of Computer Application
(Special issue- Issue 5, Volume 2 (January 2015)
Available online on http://www.rspublication.com/ijca/ijca_index.htm
ISSN: 2250-1797
strengths of a few frequently used adverbs and verbs with values ranging from -1 to +1. We
consider some of the most frequently used adverbs and verbs along with their strength as
given below in table 2:
Table 2. Strength of Verbs and Adverbs
Verb
Love
Adore
Like
Enjoy
Smile
Impress
Attract
Excite
Relax
Reject
Disgust
Suffer
Dislike
Detest
Suck
Hate
Strength
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
-0.2
-0.3
-0.4
-0.7
-0.8
-0.9
-1
Adverb
Complete
Most
Totally
Extremely
Too
Very
Pretty
More
Much
Any
Quite
Little
Less
Not
Never
Hardly
Strength
1
0.9
0.8
0.7
0.6
0.4
0.3
0.2
0.1
0.2
-0.3
-0.4
-0.6
-0.8
-0.9
-1
The whole procedure for predicting adverb and verb polarity is given below:
The algorithm for determining the orientation Procedure “determine_orientation” takes the
target Adverb/Verb whose orientation needs to be determined and the respective seed list as
the inputs.
1. Procedure determine_orientation (target_Adverb/ Verb wi , Adverb/ Verb_ seedlist)
2. begin
3. if (wi has synonym s in Adverb/ Verb _ seedlist )
4. { wi’s orientation= s’s orientation;
5. add wi with orientation to Adverb/ Verb _ seedlist ; }
6. else if (wi has antonym a in Adverb/ Verb _ seedlist)
7. { wi’s orientation = opposite orientation of a’s orientation;
8. add wi with orientation to Adverb/ Verb _ seedlist; }
9. end
The procedure determine_orientation searches Word Net and the Adverb/ Verb seed list for
each target adjective to predict its orientation (line 3 to line 8). In line 3, it searches
synonym set of the target Adverb/ Verb from the Word Net and checks if any synonym has
known orientation from the seed list. If so, the target orientation is set to the same
orientation as the synonym (line 4) and the target Adverb/ Verb along with the orientation is
inserted into the seed list (line 5). Otherwise, the function continues to search antonym set
of the target Adverb/ Verb from the Word Net and checks if any Adverb/Verb have known
orientation from the seed lit (line 6). If so, the target orientation is set to the opposite of the
CONFERENCE PAPER
National level conference on
"Advances in Networking, Embedded System and Telecommunication 2015(ANEC-2015)"
On 6-8 Jan 2015 organized by
" G.H.Raisoni College of Engg. & Management, Wagholi, Pune, Maharashtra, India."
Page 239
International Journal of Computer Application
(Special issue- Issue 5, Volume 2 (January 2015)
Available online on http://www.rspublication.com/ijca/ijca_index.htm
ISSN: 2250-1797
antonym (line 7) and the target Adverb/ Verb with its orientation is inserted into the seed
list (line 8). If neither synonyms nor antonyms of the target word have known orientation,
the function just continues the same process for the next Adverb/ Verb since the word’s
orientation may be found in a later call of the procedure with an updated seed list.
Note:
1) For those adverbs/ verbs that Word Net cannot recognize, they are discarded as they may
not be valid words.
2) For those that we cannot find alignments, they will also be removed from the opinion
words list and the user will be notified for attention.
3) If the user feels that the word is an view word and knows its sentiment, he/she can update
the seed list.
4) For the case that the synonyms/antonyms of an adjective have different known semantic
orientations, we use the first found orientation as the orientation
for the given adjective.
Tweet Sentiment Scoring
As adverbs qualify adjectives and verbs, we group the corresponding adverb and adjective
together and call it the adjective group; similarly we group the corresponding verb and
adverb together and call it the verb group. The adjective group strength is calculated by the
product of adjective score (adji) and adverb (advi) score, and the verb group strength as the
product of verb score (vbi) and adverb score (advi). Sometimes, there is no adverb in the
opinion group, so the S (adv) is set as a default value 0.5. To calculate the overall sentiment
of the tweet, we average the strength of all opinion indicators like emoticons, exclamation
marks, capitalization, word emphasis, adjective group and verb group as shown below:
S(T) = [(1+(Pc+log(Ns)+log(Nx))/3) / |OI(R)|]*∑i=1|OI(R)| S(AGi)+S(VGi)+Nei*S(Ei)
Where,
|OI(R)| denotes the size of the set of opinion groups and emoticons extracted from the tweet,
Pc denotes fraction of tweet in caps,
Ns denotes the count of recurrent letters,
Nx denotes the count of exclamation marks,
S (AGi) denotes score of the ith adjective group,
S (VGi) denotes the score of the ith verb group,
S (Ei) denotes the score of the ith emoticon
Nei denotes the count of the ith
Pc, Ns and Nx represent emphasis on the sentiment to be conveyed so they can be
collectively called sentiment intensifiers. If the score of the tweet is more than 1 or less than
-1, the score is taken as 1 or -1 respectively.
CONCLUSION AND FUTURE WORK
Hadoop Ecosystem with integration of Hbase, hive and flume should be used in
various fields like telecommunications, banks, insurance, medical fields etc. to maintain
public details in an efficient manner and to avoid the fraudulent.Fulfillment of Lack of good
high level support for Hadoop. ClubCF method for big data applications is related to service
recommendation. The explosion of microblogging sites like Twitter offers an unprecedented
opportunity to create and employ theories & technologies that search and mine for
CONFERENCE PAPER
National level conference on
"Advances in Networking, Embedded System and Telecommunication 2015(ANEC-2015)"
On 6-8 Jan 2015 organized by
" G.H.Raisoni College of Engg. & Management, Wagholi, Pune, Maharashtra, India."
Page 240
International Journal of Computer Application
(Special issue- Issue 5, Volume 2 (January 2015)
Available online on http://www.rspublication.com/ijca/ijca_index.htm
ISSN: 2250-1797
sentiments.The integration of Hadoop Ecosystem is used to store and retrieve the huge
structured/unstructured datasets without any loss and the parallelization is achieved in
loading the datasets, building the indexes and evaluation of the queries.
REFERENCE
[1] A. Java, X. Song, T. Finin, and B. Tseng. Why we twitter: understanding microblogging
usage and communities. In Proceedings of the 9th WebKDD and 1st SNA-KDD 2007
Workshop on Web Mining and Social Network Analysis, pages 56-65, 2007.
[2] J. M. Kleinberg. Authoritative sources in a hyperlinked environment. J. ACM,
46(5):604{632, 1999.
[3] D. M. Romero and J. Kleinberg. The directed closure process in hybrid socialinformation
networks, with an analysis of link formation on Twitter. In Proceedings of the 4th
International AAAI Conference on Weblogs and Social Media, pages 138-145, 2010.
[4] B. Liu. Web data mining; Exploring hyperlinks, contents, and usage data. Springer,2006.
[5] M. Cha, H. Haddadi, F. Benevenuto, and K. P. Gummadi. Measuring User Inuence in
Twitter: The Million Follower Fallacy. In Proceedings of the 4th International AAAI
Conference on Weblogs and Social Media, pages 10{17, 2010.
[6] B. Pang and L. Lee. Opinion mining and sentiment analysis. Foundations and Trends in
Information Retrieval, 2(1-2):1-135, 2008.
[7] M. De Choudhury, Y.-R. Lin, H. Sundaram, K. S. Candan, L. Xie, and A. Kelliher. How
does the data sampling strategy impact the discovery of information di_usion in social
media? In Proceedings of the 4th International AAAI Conference on Weblogs and Social
Media, pages 34-41, 2010.
[8] I. Derenyi, G. Palla, and T. Vicsek. Clique percolation in random networks. Physical
Review Letters, 94(16), 2005.
[9] . B. O'Connor, R. Balasubramanyan, B. R. Routledge, and N. A. Smith. From tweets to
polls: Linking text sentiment to public opinion time series. In Proceedings of the International
AAAI Conference on Weblogs and Social Media, pages 122{129, 2010.
[10] B. J. Jansen, M. Zhang, K. Sobel, and A. Chowdury. Micro-blogging as online word of
mouth branding. In Proceedings of the 27th International Conference Extended Abstracts on
Human Factors in Computing Systems, pages 3859{3864, 2009.
[11] A. Pak and P. Paroubek. Twitter as a corpus for sentiment analysis and opinion mining.
In Proceedings of the Seventh Conference on International Language Resources and
Evaluation, pages 1320{1326, 2010.
CONFERENCE PAPER
National level conference on
"Advances in Networking, Embedded System and Telecommunication 2015(ANEC-2015)"
On 6-8 Jan 2015 organized by
" G.H.Raisoni College of Engg. & Management, Wagholi, Pune, Maharashtra, India."
Page 241
International Journal of Computer Application
(Special issue- Issue 5, Volume 2 (January 2015)
Available online on http://www.rspublication.com/ijca/ijca_index.htm
ISSN: 2250-1797
[12] A. Go, L. Huang, and R. Bhayani. Twitter sentiment classification using distant
supervision. In CS224N Project Report, Stanford, 2009.
[13] Ammar Hassan* and Ahmed Abbasi+ *Department of Systems and Information
Engineering +Department of Information Technology University of Virginia Charlottesville,
Virginia USA mah9tg@virginia.edu, abbasi@comm.virginia.edu, Daniel Zeng Department of
Information Systems University of Arizona, Tucson, Arizona, USA Institute for Automation,
Chinese Academy of Sciences Beijing, China zeng@email.arizona.edu “Twitter Sentiment
Analysis: A Bootstrap Ensemble Framework ”
[14] Vishal S Patil1, Pravin D. Soni2 1ME (CSE) Scholar, Department of CSE, P R Patil
College of Engg. & Tech. AmravatiAmravati-444605,, 2Assitantant Professor, Department
of CSE, P R Patil College of Engg. & Tech. Amravati Amravati-444605,India, “HADOOP
SKELETON & FAULT TOLERANCE IN HADOOP CLUSTERS” International Journal of
Application or Innovation in Engineering & Management (IJAIEM) Web Site:
www.ijaiem.org Email: editor@ijaiem.org, editorijaiem@gmail.com Volume 2, Issue 2,
February 2013 ISSN 2319 - 4847
[15] V.Nappinna lakshmi1, N. Revathi2* 1PG Scholar, 2Assistant Professor Department of
Information Technology, Sri Venkateswara College of Engineering, Sriperumbudur –
602105, Chennai, INDIA. 1Nappinnavenkat@gmail.com 2 revathi@svce.ac.in*
*Corresponding author “DATA MINING OVER LARGE DATASETS USING HADOOP
IN CLOUD ENVIRONMENT ” V Nappinna Lakshmi et al, International Journal of
Computer Science & Communication Networks,Vol 3(2), 73-78 ISSN:2249-5789
[16] Akshi Kumar and Teeja Mary Sebastian Department of Computer Engineering, Delhi
Technological University Delhi, India “Sentiment Analysis on Twitter” IJCSI International
Journal of Computer Science Issues, Vol. 9, Issue 4, No 3, July 2012 ISSN (Online): 16940814 www.IJCSI.org
CONFERENCE PAPER
National level conference on
"Advances in Networking, Embedded System and Telecommunication 2015(ANEC-2015)"
On 6-8 Jan 2015 organized by
" G.H.Raisoni College of Engg. & Management, Wagholi, Pune, Maharashtra, India."
Page 242