A
VI Semester
MINOR PROJECT REPORT
ON
“N ATURAL L ANGUAGE P ROCESSING W ITH D ISASTER
T WEETS ”
BY
Sarthak Chandrakar
Ankit Paul
Sidharth paliwal
18101039
18100007
18100056
UNDER THE GUIDANCE OF
Dr.N.Srinivas Naik
DEPARTMENT OF COMPUTER SCIENCE ENGINEERING
Dr. SPM INTERNATIONAL INSTITUTE OF INFORMATION TECHNOLOGY NAYA RAIPUR
ATAL NAGAR - 493661, INDIA
May 2021
Natural Language Processing with Disaster Tweets
Declaration
I declare that this written submission represents my ideas in my own words and where others’
ideas or words have been included, I have adequately cited and referenced the original sources. I
also declare that I have adhered to all principles of academic honesty and integrity and have not
misrepresented or fabricated or falsified any idea/data/fact/source in my submission. I understand
that any violation of the above will be cause for disciplinary action by the Institute and can also
evoke penal action from the sources which have thus not been properly cited or from whom proper
permission has not been taken when needed.
Dated: 11/05/2021
Sarthak Chandrakar
Ankit Paul
Sidharth Paliwal
Department of Computer Science Engineering, DSPM IIIT Naya Raipur
2
Natural Language Processing with Disaster Tweets
Certificate
This is to certify that the project titled “Natural Language Processing With Disaster Tweets” by
“Sarthak Chandrakar, Ankit Paul, Sidharth Paliwal” has been carried out under my/our supervision
and that this work has not been submitted elsewhere for a degree/diploma.
Dated: 11/05/2021
Dr.N.Srinivas Naik
Assistant Professor
Department of CSE
DSPM IIIT Naya Raipur-493661
Department of Computer Science Engineering, DSPM IIIT Naya Raipur
3
Natural Language Processing with Disaster Tweets
Acknowledgments
At the outset, I would like to express my whole hearted and deep sense of gratitude to my guide
Prof. N. Srinivas Naik for his guidance, help and encouragement throughout my research work. I
greatly admire his attitude towards research, creative thinking, hard work and dedication in work. I
am highly grateful to him for patiently checking all my manuscripts and thesis. This thesis would
not have been possible without his bounteous efforts. More than guide, he is my mentor for shaping
my personal and professional life, without whom I would not have been where I am today. I owe my
profound gratitude to Prof N. Srinivas Naik for his supports in all respects.
Sarthak Chandrakar Ankit Paul Sidharth Paliwal
Department of Computer Science Engineering, DSPM IIIT Naya Raipur
4
ABSTRACT
Microblogging sites such as Twitter has become a very important communication
medium in the times of emergency events such as hurricanes ,earthquakes. Convenience of smartphones has now enabled people to announce an emergency which
they are observing in real time. Thus, more and more agencies and organizations
have become interested in programatically analyzing twitter. This data can be used
for situational awareness and other disaster management efforts. But it is easier said
than done. It is not always clear if a tweet is announcing disaster. For example, a
user might metaphorically say that the sky is looking ablaze. This is quite clear to a
human, but not so much for a machine. Processing tweets pose multiple challenges
such as parsing noisy,brief and other informal messages.We will use Natural Language processing on the dataset. In this paper, we build a model that predicts which
Tweets are about real disasters and which ones are not.Before building the baseline model we have analysed the dataset and found it needed a lot of cleaning since
tweets can be very varying in language and size.We have used techniques such as
spellchecker to correct wrongly spelt words.We have performed data cleaning.After
studying many research papers we have come to conclusion that for keyphrase extraction LSTM would be a good baseline model.
Keywords: Twitter,Disasters,NLP,keyphrase extraction
Contents
1
Introduction
1
2
Literature Survey
2
3
Proposed Model
4
4
Data Preprocessing
4.1 Dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.2 EXPLORATORY DATA ANALYSIS . . . . . . . . . . . . . . . .
4.3 DATA CLEANING . . . . . . . . . . . . . . . . . . . . . . . . . .
5
5
7
10
5
Feature Engineering
5.1 Count Vectorizer . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.2 TF-IDF . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
11
11
12
6
Base Line Models
6.1 Train-Test Split . . . . . .
6.2 Logistic Regression . . . .
6.3 Naives Bayes . . . . . . .
6.4 Random Forest Regression
6.5 XGBoost . . . . . . . . .
13
13
13
13
13
14
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
7
Results and Analysis
15
8
References
17
List of Figures
4.1
4.2
4.3
4.4
4.5
Dataset . . . . . . . . . . . . . . . . . . . . . . . .
Count vs Target . . . . . . . . . . . . . . . . . . . .
Keyword vs Count . . . . . . . . . . . . . . . . . .
Keywords with highest percentage of disaster tweeets
Keyword with lowest percentage of disaster tweets .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
5
6
6
7
8
7.1
Output of twitter classification . . . . . . . . . . . . . . . . . . . .
16
Natural Language Processing with Disaster Tweets
Chapter 1
Introduction
Twitter has become an important medium to reach out for help in troubling times
like disaster.In times of natural calamities or other disasters when disaster relief
organisations are overwhelmed with responses and reaching them is almost impossible people now turn to twitter to send a call out for help.Sending the tweet is very
easy,fast and its popularity makes it just more favorable.But generally tweets contain a lot of noise and abstract vocabulary which can be hard to understand or filter
out.This noisiness of data must be removed before processing or it may lead to errors
in the future.
Tweets play an important role in communicating messages between two parties
and if there is any discrepancy, then the integrity of message will be lost and hence
the communication will not be successful.Therefore it becomes necessary to decode
the message correctly during emergency times.There are several mechanisms which
we can adopt to extract the information from the text.NLP is one those mechanisms
which helps us to extract several number of features from the given text message.
Although Twitter contains a lot of data which can help in rescue operations but
the data must be cleared of the noise and categorised correctly for it to be of any
help to the relief organisations.Tweets have slang,irregularites and may contain redundant information.To make the process of search and rescue fast the tweets must
be categorised whether its disaster related or a figure of speech which includes
some keywords related to disaster such as ablaze,fire,thunderstorm,police, hijacker
etc.Keyphrase extraction from the tweet is a very important step in Model prediction.But Since tweets can be very short methods such as term frequencies,word occurences and clustering methods dont work well because they are designed to work
on long text.This is the reason that recently researchers have started using deep learning models and nlp to explore keyword extraction.
Department of Computer Science Engineering, DSPM IIIT Naya Raipur
1
Natural Language Processing with Disaster Tweets
Chapter 2
Literature Survey
Before Classifying the tweets as disastrous or non disastrous,we have performed
literature survey to identify the existing methods to extract relevant information from
text using some of the existing techniques.Tweet classification is one the important
factor during an emergency to announce or aware the people.One of the papers for
identifying and categorizing disaster disaster related tweets focused on the data generated by the twitter before ,during and after the Hurricane Sandy which has a great
impact on New York.[1]They proposed an annotation schema for identifying relevant
tweets and developed rich feature classifiers for fine grained categorization.Different
methods for filtering tweets are used.Their baseline features includes the counting
of unigrams in tweets and then removes capitalization,punctuations,stopwords after
preprocessing. Data Sparsity caused them difficulty as many classes lack positive
samples which are necessary for the machine to reliably classify them which is observed as one of the limitation of this paper.Another limitation is because Information is extracted via batch processing only after events and thus this approach can be
extended to real time processing.
[2] They present Twitter corpora of more than 52 million crisis related messages
collected during 19 crises. They provide human annotations (volunteer and crowd
source workers) of two types. First, the tweets are annotated with a set of categories
such as displaced people,infrastructres etc. These annotations were built using formal crisis response agencies such as United Nations Office for the Coordination of
Humanitarian Affairs (UN OCHA). Secondly, tweets are annotated to find out of
vocabulary. The annotated data is used to train machine learning classifiers : NAive
Bayes, Random forest, SVM. They also train word2vec from 52 million tweets and
make them publicly available.
[3]Doina et al. used a Joint layer bidirectional LSTM model for keyword extraction from disaster related tweets.A joint layer RNN model can predict keywords of
any arbitrary length without much preprocessing therefore this model was explored
and LSTM which is special kind of RNN was used for extraction.The keyword extraction was applied on both general and diaster related tweets.The joint layer BiDepartment of Computer Science Engineering, DSPM IIIT Naya Raipur
2
Natural Language Processing with Disaster Tweets
LSTM serves as the baseline Model.Tweet converted into tokens was passed through
the model with windowsize 3.Embedding based measures were considered major
evaluation metric for the model.Model achieved a good accuracy and with some
more features such as POS tags and Elmo embeddings with the baseline model this
accuracy can be further increased.
The authors in [4] proposed a system which is based on the idea that there is
always a large stream of tweets from disaster struck area.The keyword from these
tweets are extracted via keyword specific to disaster.Data is concentrated around
different locations using the K nearest neighbour.Data is stored and the category of
disaster is determined from them.it is carried out by checking each tweet against a
set of predefined ewighed keywords.Disasters location is also labelled along with
tweets.Data is fed through sentiment analysis and users emotional status is found.It
is found that people with high emotional distress writing tweets from disaster struck
places are the most effected.
[5] Akash kumar et al. used crisisMMD dataset and performed a multimodal
analysis of disaster tweets on it.Data preprocessing was done on the dataset.The
models were classified as text only,image model and both text and image processing
model.It was found from the results that logistic regression based decision policy for
incorporating text and image feature is better than only text and only image models.
[6]Sridevi M et al. used a classifier model with ELMo embeddings to classify
crisis related data.Classifier with elmo embeddings were tested on three different
datasets.The accuracy came out to be 78.This accuracy can be improved using deep
learning architectures.
The main objective of [7] is to find the credibility of tweets during a disaster
crisis.A novel web solution was made to detect credibility called TweetCred.It will
assess credibility of tweets in real time and help to filter out disaster related tweets
which contain vital information which can help people in times of disaster.A dataset
of disaster whch contain 45 features was used and SVM model was used to give
them credibility scores.This model was deployed as a browser extension using web
and REST API.A lot of users donloaded the extension and credibility score of 5.4
million tweets was calculated by tweetcred.
Department of Computer Science Engineering, DSPM IIIT Naya Raipur
3
Natural Language Processing with Disaster Tweets
Chapter 3
Proposed Model
Tweet classification for disastrous and non disastrous tweet can be a significant
contribution towards the society as this may help people to escape at the time of
emergency.Even after having so many methods for feature extraction from the text
and also there are various ways for text mining, still the twitter classification remains remains unsolved many times.In this study we have first tried to extract many
features of the text so that we can classify them more accurately.We have tried to
find out the number of unique users who has tweeted.Also tracing location is also an
important step from where the tweet has come.For identifying whether the tweet is
disastrous or not,we have also tried to find the most usage of keywords which will
help us to identify the tweet from the given keywords only during tweet classification.
After feature extraction,text mining and also after removal of stopwords,punctuation
marks and other necessary steps of Natural Language Processing,we have tried to
find out the accuracy of tweet classification using different models like Random
Forest Regression,XGBoost etc.This comparison among different models helps us
to find the accuracy of our classification.Comparing accuracy of different models
along with their confusion matrix will give us a clear picture of this classification.
Department of Computer Science Engineering, DSPM IIIT Naya Raipur
4
Natural Language Processing with Disaster Tweets
Chapter 4
Data Preprocessing
4.1
Dataset
The dataset is acquired from Kaggle. Each sample in the data contains the following
information:
• The text from a tweet
• The keyword from the tweet (can be empty)
• The location from where the tweet was sent(can be empty)
Printing the head of our dataWe check the number of positive and negative samples.0 corresponds to a tweet
not being a real disaster whereas 1 means the tweet is a disaster.
Then we print the keywords in our tweets and sort them in descending order of
their frequency.
Figure 4.1: Dataset
Department of Computer Science Engineering, DSPM IIIT Naya Raipur
5
Natural Language Processing with Disaster Tweets
Figure 4.2: Count vs Target
Figure 4.3: Keyword vs Count
Department of Computer Science Engineering, DSPM IIIT Naya Raipur
6
Natural Language Processing with Disaster Tweets
Figure 4.4: Keywords with highest percentage of disaster tweeets
Now we will find the keywords with highest percentage of disaster tweets.
Similarly keywords with lowest percentage of disaster tweets.
4.2
EXPLORATORY DATA ANALYSIS
Before feeding the model input we have to check our input on various different
factors and after that we can perform data cleaning on it.
The factors considered for exploring the data are:
We can see that the most common stopwords in tweets related to non-disastrous
tweets and disastrous tweets.We can remove these stopwords which are common
from both the classes 0 and 1 from our tweets using nlp tools so that only quality
and somewhat unique keywords are left which our model can easily identify and
understand.
Department of Computer Science Engineering, DSPM IIIT Naya Raipur
7
Natural Language Processing with Disaster Tweets
Figure 4.5: Keyword with lowest percentage of disaster tweets
Number of characters in tweets
Department of Computer Science Engineering, DSPM IIIT Naya Raipur
8
Natural Language Processing with Disaster Tweets
Number of words in tweets
Common Stopwords in tweets with class 0
Common Stopwords in tweets with class 0
Department of Computer Science Engineering, DSPM IIIT Naya Raipur
9
Natural Language Processing with Disaster Tweets
4.3
DATA CLEANING
Since the tweets contain a lot of irregularity and noise we need a lot of data cleaning on our dataset.Common areas to get most errors removed from the tweets are
• Removing URLS
Tweets generally contain link to various other information which does not
help to clarify whether it is disaster related or not.So even if we remove these
urls no change occurs in the accuracy of our model.We will remove all the
• Spell checking on tweets.
We will use pyspellchecker to correctly spell any misspelt words in the
dataset.
• Cleaning HTML Tags.
• Removing punctuations.
Punctuations dont tell us a lot about our target variable.They must be cleaned
before modelling.
• Cleaning all emojis from tweets.
Our model will not be able to differentiate between different emojis.NLP
tools cannot understand emojis and they must be removed.The emojis dont
play a very important role in classifying the tweets so its safe to remove them.
Department of Computer Science Engineering, DSPM IIIT Naya Raipur
10
Natural Language Processing with Disaster Tweets
Chapter 5
Feature Engineering
5.1
Count Vectorizer
Count vector tokenizes (that means it breaks down a sentence or paragraph or
any text into words) the given text along with it performs some of the very basic
preprocessing operations such as removing punctuation marks and converting all the
words to lowercase etc.The vocabulary of known words is formed which can be used
for encoding unseen text later.Encoded vector is returned with a length of the entire
vocabulary and an interger count for the number of times each word appeared in the
document.Consider an example of a disaster tweet: ”Our Deeds are the reason of the
earthquake”.
keyword
Our
Deeds
are
the
of
reason
earthquake
Count
1
1
1
2
1
1
1
Table 5.1: Count of keywords
keyword
0
1
2
3
4
5
6
Count
1
1
1
2
1
1
1
Table 5.2: Count of keywords with indexes
Department of Computer Science Engineering, DSPM IIIT Naya Raipur
11
Natural Language Processing with Disaster Tweets
5.2
TF-IDF
TF-IDF stands for Term Frequency Inverse Document Frequency. It is a very
common algorithm to convert text into meaningful representation of numbers for
the machine learning algorithm. Count vectorizer gives number of frequency wrt
index of vocabulary whereas tf-idf considers overall document of weight of words.
Consider the example - d1: The sky is blue. d2: The sun is bright.Using count vector
and TF-IDF on the two documents, we get
Department of Computer Science Engineering, DSPM IIIT Naya Raipur
12
Natural Language Processing with Disaster Tweets
Chapter 6
Base Line Models
6.1
Train-Test Split
After splitting the dataset into training part and testing part,we have found the
the shape of each of the following.Training data has 5329 rows and 1700 columns
whereas testing data has 2284 rows and 1700 columns.
6.2
Logistic Regression
Logistic Regression model is fit on the training dataset and on predicting whether
the tweets are about real disasters or not, it gives an accuracy of 77.9 percent.
Positive Negative
Positive
1124
191
Negative
267
702
6.3
Naives Bayes
Naive Bayes model is fit on the training dataset and on predicting whether the tweets
are about real disasters or not, it gives an accuracy of 78.8 percent.
Positive Negative
Positive
1201
114
Negative
326
643
6.4
Random Forest Regression
Random Forest Regression model is fit on the training dataset and on predicting
whether the tweets are about real disasters or not, it gives an accuracy of 77.8 percent.
Department of Computer Science Engineering, DSPM IIIT Naya Raipur
13
Natural Language Processing with Disaster Tweets
Positive Negative
Positive
1121
194
Negative
302
667
6.5
XGBoost
XGBoost classifier is fit on the training dataset and on predicting whether tweets are
about real datasets or not,it gives accuracy of 77.02 percent.
Positive Negative
Positive
1115
200
Negative
292
677
Department of Computer Science Engineering, DSPM IIIT Naya Raipur
14
Natural Language Processing with Disaster Tweets
Chapter 7
Results and Analysis
We have used 4 models for predicting the tweets whether they belong to real
disaster or not.We have used Logistic Regression , Naive Bayes ,Random Forest
Regression, XGBoost models and their accuracies are calculated. Their confusion
matrices are already plotted in the previous section.Let’s have a quick look at the
accuracy table.
Model
Logistic Regression
Naive Bayes
Random Forest Regression
XGBoost
Accuracy
77.9
78.8
77.8
77.02
Out of these 4 models we can see that all models have performed pretty well but
we can see that there is a slight difference of accuracy between models and Naive
Bayes model has come out with highest percentage of accuracy. The confusion
matrix for the Naive Bayes Model is plotted
Positive Negative
Positive
1201
114
Negative
326
643
The output of the following model is given below.Tweets which are about real
disasters are labelled as 1 and which are not about real disasters are labelled as
0.Tweets having particular ids are classified as disaster tweets or not.
Department of Computer Science Engineering, DSPM IIIT Naya Raipur
15
Natural Language Processing with Disaster Tweets
‘
Figure 7.1: Output of twitter classification
Department of Computer Science Engineering, DSPM IIIT Naya Raipur
16
Natural Language Processing with Disaster Tweets
Chapter 8
References
1. Stowe, Kevin, et al. ”Identifying and categorizing disaster-related tweets.” Proceedings of The fourth international workshop on natural language processing
for social media. 2016.
2. Ray Chowdhury, Jishnu, Cornelia Caragea, and Doina Caragea. ”Keyphrase
extraction from disaster-related tweets.” The world wide web conference. 2019.
3. Imran, Muhammad, Prasenjit Mitra, and Carlos Castillo. ”Twitter as a lifeline:
Human-annotated twitter corpora for NLP of crisis-related messages.” arXiv
preprint arXiv:1605.05894 (2016).
4. Shekhar, H., Setty, S. (2015). Disaster analysis through tweets. 2015 International Conference on Advances in Computing, Communications and Informatics (ICACCI). doi:10.1109/icacci.2015.7275861
5. Gautam, A. K., Misra, L., Kumar, A., Misra, K., Aggarwal, S., Shah, R. R.
(2019). Multimodal Analysis of Disaster Tweets. 2019 IEEE Fifth International Conference on Multimedia Big Data (BigMM). doi:10.1109/bigmm.2019.0038
6. Madichetty, S., M, S. (2020). Improved Classification of Crisis-Related Data
on Twitter using Contextual Representations. Procedia Computer Science, 167,
962–968. doi:10.1016/j.procs.2020.03.395
7. Gupta, A., Kumaraguru, P., Castillo, C., Meier, P. (2014). TweetCred: RealTime Credibility Assessment of Content on Twitter. Social Informatics, 228–243.
Department of Computer Science Engineering, DSPM IIIT Naya Raipur
17