Deep Learning for Detecting Inappropriate Content in Text
Thesis submitted in partial fulfillment
of the requirements for the degree of
MS in CSE by Research
by
Harish Yenala
201407613
harish.yenala@research.iiit.ac.in
International Institute of Information Technology
Hyderabad - 500032, INDIA
March 2018
Copyright c Harish Yenala, 2017
All Rights Reserved
International Institute of Information Technology
Hyderabad, India
CERTIFICATE
It is certified that the work contained in this thesis, titled “Deep Learning for Detecting Inappropriate
Content in Text” by Harish Yenala, has been carried out under my supervision and is not submitted
elsewhere for a degree.
Date
Adviser: Prof. Manish Shrivastava
Co-Adviser: Dr. Manoj Chinnakotla
To all my Teachers
for their guidance, love and support.
Acknowledgements
My journey in IIIT- Hyderabad has been a wonderful experience. As I submit my MS thesis, I wish
to extend my gratitude to all those people who helped me in successfully completing this journey.
First of all, I want to thank my guides Dr. Manish Shrivastava, Dr. Manoj Kumar Chinnakotla and
Chakravarthi Jada for accepting me as their student and constantly guiding me. Their guidance has not
only helped me become a good researcher but also a better person. They never left a stone unturned to
help me out in any situation under the sun. They truely are my ”Godfathers”!
I could not have accomplished it without the support of my mother, brother and my uncle. Their
endless love and faith in me constantly motivates me.
I will be indebted to all my school and college teachers for making me the person I am today.
I thank my sisters Maheshwari and Nagalakshmi and my friend Vandana for supporting me in my
odds, taking care of me and keep on motivating me to reach a better place in my career.
I thank my friend Arpita Das for being almost as my third Masters guide. Thank you for inspiring me
to work better always. I would also like to thank all my labmates Avinash, Nirmal, Nausheen, Goutham,
Prasad, Praneeth for the wonderful interactions and fun work sessions.
Special thanks to Kranthi, Irfan, Anil and Mounika. I woud like to thank Subbareddy, Nayyar,
Yasaswi, Aditya for sharing your knowledge and for those fun-filled experiences in IIIT.
I thank Deepti Ma’am and Vasu sir for giving me internship opportunity at IIIT in my bachelors.
This opportunity made me to choose research field.
I thank Rajesh, Uday, Vinay, Ranjith, Harish Reddy, Srinivas Reddy for being such amazing friends.
Survival in Hyderabad would be very difficult without you people.
Last, but not the least, thanks to IIIT community for giving me an inspiring environment and loads
of opportunities to grow.
v
Abstract
A given piece of textual information produced by any user or agent is said to be inappropriate if the
expressed intent may cause anger, annoyance to certain users or exhibits lack of respect, rudeness, discourteousness towards certain individuals/communities or may be capable of inflicting harm to oneself
or others.
A search engine should regulate its query completion suggestions by detecting and filtering such
queries as it may hurt the user sentiments or may lead to legal issues thereby tarnishing the brand
image. Hence, automatic detection and pruning of such inappropriate queries from completions and
related search suggestions is an important problem for most commercial search engines. The problem
is rendered difficult due to unique challenges posed by search queries such as lack of sufficient context,
natural language ambiguity and presence of spelling mistakes and variations.
In this Thesis, we propose a novel deep learning based technique for automatically identifying inappropriate query suggestions. We propose a novel deep learning architecture called “Convolutional BiDirectional LSTM (C-BiLSTM)” which combines the strengths of both Convolution Neural Networks
(CNN) and Bi-directional LSTMs (BLSTM). Given a query, C-BiLSTM uses a convolutional layer for
extracting feature representations for each query word which is then fed as input to the BLSTM layer
which captures the various sequential patterns in the entire query and outputs a richer representation
encoding them. The query representation thus learnt passes through a deep fully connected network
which predicts the target class. C-BiLSTM doesn’t rely on hand-crafted features, is trained end-end as
a single model, and effectively captures both local features as well as their global semantics. Evaluating
C-BiLSTM on real-world search queries from a commercial search engine reveals that it significantly
outperforms both pattern based and other hand-crafted feature based baselines. Moreover, C-BiLSTM
also performs better than individual CNN, LSTM and BLSTM models trained for the same task.
Rapid growth of chatbots and Interactive Gaming Systems expect large, real time, clean (noninappropriate) human conversation data to train their models. High demand of social networking sites
provide a platform for users to interact and share opinions publicly. Inappropriate conversations and
posts in these platforms may also lead to loss of business and damaging company’s image. So we extended our idea to identify inappropriate conversations in chat data. We have applied various techniques
to solve this problem and observed that BLSTM performs better than LSTM and Boosted Decision Tree
approaches.
vi
Contents
Chapter
Page
1
Introduction : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : :
1.1 Inappropriate Text . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.2 Importance of Detecting and filtering inappropriate text . . . . . . . . . . . . . .
1.2.1 Importance of Identifying Inappropriate Queries in Web search . . . . . .
1.2.2 Importance of Identifying Inappropriate text in Conversations . . . . . .
1.3 Challenges involved in Identifying Inappropriate Text . . . . . . . . . . . . . . .
1.4 Our Contribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.5 Organization of the Thesis . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
: :
.
.
.
.
.
.
.
1
1
1
2
3
5
6
7
2
Literature Review : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : :
2.1 Literature Review on Inappropriate Text Detection . . . . . . . . . . . . . . . . . . .
2.2 Related work on Hybrid Deep Models . . . . . . . . . . . . . . . . . . . . . . . . . .
8
8
9
3
Inappropriate Text Detection in Web search queries : : : : : : : : : : : : : : : : : :
3.1 C-BiLSTM for Inappropriate Query Detection . . . . . . . . . . . . . . . . . . .
3.1.1 Input Query Embedding and Padding . . . . . . . . . . . . . . . . . . .
3.1.2 Learning Feature Representations using Convolution Layer . . . . . . . .
3.1.3 Capturing Sequential Patterns with Bi-Directional LSTM . . . . . . . . .
3.2 Model Training . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.3 Experimental Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.3.1 Dataset Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.3.2 Baseline Approaches . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.4 Results and Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.4.1 Qualitative Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.4.2 Query Autocompletion Filtering Task . . . . . . . . . . . . . . . . . . .
:
.
.
.
.
.
.
.
.
.
.
.
:
.
.
.
.
.
.
.
.
.
.
.
: : 11
.
11
.
12
.
12
.
13
.
13
.
14
.
14
.
15
.
16
.
18
.
18
4
Inappropriate conversation Detection in Chat Data : : : : : : : : : : : : : : : : : : :
4.1 Inappropriate Text Detection for Conversations . . . . . . . . . . . . . . . . . .
4.2 Bi-directional LSTM for Inappropriate Conversation Detection . . . . . . . . . .
4.2.1 Input Conversation Embedding . . . . . . . . . . . . . . . . . . . . . .
4.2.2 Bi-Directional LSTM . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.3 Model Training . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.4 Details of the Conversation Data Sets . . . . . . . . . . . . . . . . . . . . . . .
4.5 Quantitative Analysis on Conversation Sets . . . . . . . . . . . . . . . . . . . .
4.6 Error Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
:
.
.
.
.
.
.
.
.
:
.
.
.
.
.
.
.
.
: : 20
.
20
.
21
.
21
.
22
.
23
.
24
.
24
.
25
vii
:
.
.
.
.
.
.
.
:
.
.
.
.
.
.
.
viii
CONTENTS
5
Siamese for Similar Question Retrieval : : : : : : : :
5.1 Siamese Neural Network . . . . . . . . . . . . .
5.2 Architecture of SCQA . . . . . . . . . . . . . .
5.2.1 Input to SCQA . . . . . . . . . . . . . .
5.3 Training and Testing . . . . . . . . . . . . . . .
5.4 Siamese Neural Network with Textual Similarity
5.5 Experiments . . . . . . . . . . . . . . . . . . . .
5.5.1 Parameter Sharing . . . . . . . . . . . .
5.6 Results . . . . . . . . . . . . . . . . . . . . . . .
5.6.1 Quantitative Analysis . . . . . . . . . . .
5.7 Conclusion . . . . . . . . . . . . . . . . . . . .
:
.
.
.
.
.
.
.
.
.
.
:
.
.
.
.
.
.
.
.
.
.
:
.
.
.
.
.
.
.
.
.
.
: : : : : : : : : : : : :
. . . . . . . . . . . . . .
. . . . . . . . . . . . . .
. . . . . . . . . . . . . .
. . . . . . . . . . . . . .
. . . . . . . . . . . . . .
. . . . . . . . . . . . . .
. . . . . . . . . . . . . .
. . . . . . . . . . . . . .
. . . . . . . . . . . . . .
. . . . . . . . . . . . . .
:
.
.
.
.
.
.
.
.
.
.
:
.
.
.
.
.
.
.
.
.
.
: : 28
.
28
.
29
.
30
.
30
.
31
.
32
.
32
.
32
.
33
.
34
6
Conclusions : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 36
6.1 Identifying Inappropriate queries and conversations . . . . . . . . . . . . . . . . . . .
36
6.2 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
37
List of Figures
Figure
Page
1.1
1.2
1.3
1.4
Sample Inappropriate Query Suggestions from Popular Web Search Engines.
Sample Inappropriate Tweets from a Popular Chatbot. . . . . . . . . . . . . .
Inappropriate comments on a popular video sharing website . . . . . . . . .
Sample Inappropriate post in a popular social networking site. . . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
2
4
4
5
3.1
3.2
3.3
3.4
3.5
3.6
Architecture of Convolutional Bi-Directional LSTM (C-BiLSTM) Model. . . . .
Statistics of Inappropriate Categories in our Evaluation Dataset. . . . . . . . . .
Sample Patterns and Keywords used in PKF Baseline. . . . . . . . . . . . . . . .
Runtime performance of various models during Train and Test phases. . . . . . .
Qualitative Analysis of C-BiLSTM Results vis-a-vis other Baseline Approaches.
Results on QPBAD Dataset using top five Query Completion Suggestions. . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
12
15
16
17
19
19
4.1
4.2
Architecture of Bi-Directional LSTM (BLSTM) Model. . . . . . . . . . . . . . . . . .
Error Distribution Chart . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
22
26
5.1
5.2
Architecture of SCQA. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Testing phase of T-SCQA. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
29
33
ix
.
.
.
.