Eran Banin
Background
 User-generated content is critical to the success of any online
platforms (CNN, Facebook, StackOverflow).
 These sites engage their users by allowing them to contribute
and discuss content, strengthening their sense of ownership
and loyalty.
 While most users tend to be civil, others may engage in
antisocial behavior, negatively affecting other users and
harming the community.
Examples-
Mechanisms for preventing
 Moderator - person who is responsible for a newsgroup or
mailing list on the Internet and for checking the messages
before sending them to the group.
 Up & Down voting.
 Ability to delete or report posts.
 Blocking users.
Yet, antisocial behavior is a significant problem that
can result in offline harassment and threats of
violence.
Past research
 Despite its severity and prevalence, surprisingly little is known
about online antisocial behavior.
 Most research reports qualitative studies that focus on
characterizing antisocial studying the behavior of a small
number users in specific communities.
 In order to create new methods for identifying undesirable
users, large-scale and longitudinal analysis (of the
phenomenon) is required.
Comparison with related work
Past:
Present:
Antisocial behavior has been
widely discussed in past
literature in order to formally
define the behavior.
Rely on a community and its
moderators to decide who they
consider to be disruptive and
harmful
Comparison with related work
Past:
Present:
Research has tended to be largely
qualitative, generally involving
deep case study analyses of a
small number of manuallyidentified trolls.
Large-scale data-driven analysis
with the goal of obtaining insights
and developing tools for the early
detection of trolls.
Moreover, the research provide
prior study of the effects of
community feedback on user
behavior.
Comparison with related work
Past:
Present:
Focus on detecting vandalism with Predict whether individual users
respect for their language and
will be subsequently banned from
reputation.
a community based on their
overall activity.
Other works tried to identified
undesirable comments based on
Also show how our models
their relevance to the discussed
generalize across multiple
article and the presence of insults.
communities.
Questions
1. Are there users that only become antisocial later in their
community life, or is deviant behavior innate?
2. Does a community’s reaction to users’ antisocial behavior
help them improve, or does it instead cause them to
become more antisocial?
3. Can antisocial users be effectively identified early on?
Contents
 Characterizing antisocial behavior
 Longitudinal analysis
 Typology of antisocial users
 Predicting future banning
Source research
 18 months, 1.7 million users contributed nearly 40 million
posts and more than 100 million votes.
 Members that repeatedly violate community norms are
eventually banned permanently. Such individuals are clear
instances of antisocial users, and constitute “ground truth” in
our analyses.
Source research
List of posts in the
same article
Comments
and replies
•CNN - General news
•IGN – Computer gaming
•Breitbart – Political news
In addition, complete time-stamped trace of user activity from
March 2012 to August 2013, as well as a list of users that were
banned from posting in these communities.
Measuring undesired behavior
 On a discussion forum, undesirable behavior may be signaled in several
ways:
 down-vote
 comment
 report a post
 community moderators may delete the offending post
 ban a user from ever posting again in the forum.
 However, down-voting may signal disagreement rather than undesirability.
Further, one would need to define arbitrary thresholds needed to label a
user as antisocial.
 In contrast, we find that post deletions are a highly precise indicator of
undesirable behavior, as only community moderators can delete posts.
Measuring undesired behavior
 Bans are similarly strong indicators of antisocial behavior, as only
community moderators can ban users.
 Thus, we focus on users who moderators have subsequently banned
from a community .
 While such an approach does not necessarily identify every antisocial
user, this results in a more precise set of users who were explicitly
labeled as undesirable.
 Filter by At least 5 posts
 Exclude users who were multiple times
Matching FBUs and NBUs
 We note that FBUs tend to post more frequently than average users.
 For example, on CNN, a typical FBU makes 264 posts, but an average
user makes only 22 posts.
 To control for this large disparity, we use matching - statistical
technique used to support causality claims in observational studies,
to control for the number of posts a user made and the number of
posts made per day.
Measuring text quality
 Problems –
 Unbiased measure of the quality or appropriateness of a post
 Dictionary-based approaches may miss non-dictionary words
 classifier trained on the text of deleted posts may confound post
content and community bias
 Thus, we instead obtained human judgments of the appropriateness
of a specific post.
 Collect post of random FBU & NBU and asked workers for evaluate
how appropriate is a post (on a scale of 1 to 5). Each post was
labeled by three independent workers, and their ratings averaged.
 131 workers completed these tasks, and they rated deleted posts
significantly lower than non-deleted posts (2.4 vs. 3.0)
Measuring text quality
 Using these labeled posts, we then trained a logistic regression
model on text bigrams to predict the text quality of a post. Posts with
a rating higher than 3 were labeled as appropriate, and the other
labeled as inappropriate.
 Under ten-fold cross validation, the AUC attained by this classifier
was 0.70.
 This suggests that while the classifier is able to partially capture this
human decision making process and allow us to observe overall
trends in the data, other factors may play a significant role in
determining whether a post is appropriate.
Understanding Antisocial Behavior
 To understand antisocial behavior in the context of a discussion
community, we first characterize how users who are banned differ
from those who are not in the terms of how they write and how they
act in a community.
 Then, we analyze changes in behavior over the lifetimes of these
users to understand the effects of post quality, community bias, and
excessive censorship.
Differences in how FBUs & NBUs write
 The similarity of a post to previous posts in a same thread may reveal how
users are contributing to a community.
 Here we compare the average text similarity of a user’s post with the
previous three posts in the same thread
 FBUs make less of an effort to integrate or stay on-topic
 Post deletion is weakly negatively correlated with text similarity,
suggests that off-topic posts are more likely to be deleted
Differences in how FBUs & NBUs write
 Next, we measure each post with respect to several readability
tests, including the Automated Readability Index (ARI), which are
designed to gauge how understandable a piece of text is.
 FBUs appear to be less readable than those written by NBUs
Differences in how FBUs & NBUs write
 Prior research also suggests that trolls tend to make inflammatory
posts.
 FBUs are less likely to use positive words
 Use less conciliatory language
 More likely to swear
How do FBUs generate activity
around themselves
 Do FBUs purposefully try to create discussions, or opportunistically
respond to an on-going discussion?
 While FBUs may either create or contribute to (already existing)
discussions depending on the community, they generally get more
replies from other users, and concentrate on fewer threads.
Evolution over time
 how does FBU’s behavior and the community’s perception of them
change over time?
 The rate of post deletion increases over time for FBUs, but is
effectively constant for NBUs.
Evolution over time
 The increase in the post deletion rate could have two causes-
1) A decrease in posting quality
2) An increase in community bias
 Text quality is decreasing over time, suggesting that 1) may be
supported
Evolution over time
 Random Select of 400 FBUs & NBUs and from each user sampled a
random post from the first 10% or last 10% of their entire posting
history.
 The posts presented to examiners who rated the appropriateness off
each post
 FBUs start out writing worse than NBUs and worsen more than NBUs
over the course of their life.
Community tolerance over time
 For testing it, random posts with similar predicted text quality were
matched to pairs. Each pair contain one post from the first 10% of a
user’s life, and one from the final 10%.
 Wilcoxon Signed-rank performed for check which post is more likely
to be deleted.
 Results - among FBUs, found a significant effect of post time in all
communities, mean that a post was made in the last 10% of an
FBU’s life is more likely to be deleted in contrast to NBUs.
The meaning of excessive censorship
 Does draconian post deletion policy could exacerbate undesirable
behavior?
 For testing it, users with at least 10 posts were taken and divided into
2 groups : 4/5 post s deleted among their 5 first posts and the
others.
 Pairs were matched based on mean text quality of their 5 first posts
and the compared by the last 5.
 Here, a Wilcoxon Signed-rank test shows a significant effect of the
deletion rate – mean that “unfairly” deletion cause worsen writing.
Types of Antisocial Users
 The following figure suggest that there are 2 kinds of FBUs-
 Hi-FBU - proportion of deleted posts above 0.5
 Lo-FBU - proportion of deleted posts below 0.5
 Across all communities, the number of users in each population is
split fairly equally between the two groups.
Differences between types
 Hi-FBUs exhibit characteristics more strongly associated with
antisocial behavior:
 use language that is less accommodating
 receive more replies
 write more posts per thread
 Live shorter period time (obvious)
 Their post deletion rate starts
high and remains high
 In contrast, Lo-FBUs tend to
have a constant low rate,
until the second half of their life.
Increasing reasons
 The text similarity of these users’ posts with other posts in the same
thread is significantly lower across their last five posts.
 Additionally, they start to post more frequently, and in fewer threads
later in their life.
 Thus, a combination of a large number of less relevant posts in a
short period of time potentially makes them more visible to other
members of the community.
Antisocial Behavior in Two Phases
 Attempt to characterize this change over a user’s lifetime by splitting
it into two halves, with the goal of understanding how users may
change across them.
 Fit two linear regression lines to a
user’s post deletion rate over time,
one for each half of the user’s life
obtained by bucketing posts into tenths .
 Computing (m1,m2) –
the slope in first and
second half respectively.
Antisocial Behavior in Two Phases
 By plotting the points we can identify quadrants corresponding to
how the deletion rate of a user’s posts changes over time:
fraction of users who are
getting worse is higher for FBUs
fraction of users who are
improving is higher for NBUs
Antisocial Behavior in Two Phases
 When looking only on users with high initial deletion rates:
high proportion of NBUs many users who should be
banned are in fact not
fraction of users who are
improving is higher for NBUs
Main goal
 In the communities we studied, FBUs tend to live for a long
time before actually getting banned .
 communities response slowly to toxic users.
 For example , on CNN, FBU write an average of 264 posts (over
42 days), with 124 posts deleted before they are finally
banned.
 Therefore, our currently main goal, is to build tools for
automatic early identification of potently FBUs.
Factors for identify FBUs
 Motivated by our observations and insights, we could begin by
designing features for identification –
1. Post: Number of words, Readability metrics (ARI),
LIWC - content analysis and text-mining software.
2. Activity: number of posts per day/thread ,fraction of replied posts,
votes up & down.
3. Community: votes received per post, fraction of up-votes received,
fraction of posts reported, number of replies per post.
4. Moderator: fraction of posts deleted, slope and intercept of linear
regression lines (i.e. m1,m2).
Techniques used for classifier
 Building decision tree
 Random forest classifier
 K-fold cross validation
 Area under the ROC curve
Building decision tree
Building decision tree
outlook
rain
sunny
overcast
2 yes / 3 no
Split further
4 yes / 0 no
Pure subset
3 yes / 2 no
Spilt further
Building decision tree
outlook
sunny
rain
overcast
humidity
high
normal
4 yes / 0 no
Pure subset
0 yes / 3 no
Pure subset
2 yes / 0 no
Pure subset
3 yes / 2 no
Spilt further
Random forest classifier
 Rationale – the combination of learning models increase the
classification accuracy.
 Main idea – To average noisy and unbiased models to create a
model with low variance .
 Random forest tree works as a large collection of decorrelated
decision trees.
Random forest classifier
Algorithm- assume we have set of features (F1,…,Fn) and dataset
for training (X1,…,Xm)1. Create the following matrix :
F1(x1) F2(x1) … Fn(x1) X1
F1(x2) …
F1(xm) …
Fn(xm) Xm
2. Chose subset of rows from the matrix and create decision
tree.
3. For testing X’ , train on all the trees and average the results.
K-fold cross validation
 The original sample is randomly partitioned into k equal
sized subsamples.
 One of the k subsamples is retained as the validation data
for testing the model, and the remaining k − 1
subsamples are used as training data.
 The k results from the folds can
then
be averaged.
Area under the ROC curve
 In order to measure the classifier performance AUC technique
been used:
Classifier performance
CNN
IGN
Breitbart
Avg.
Prop. Deleted
post
0.74
0.72
0.72
0.73
Post
0.62
0.67
0.58
0.62
+Activity
0.73 (0.66) 0.74 (0.65)
0.66 (0.64)
0.71 (0.65)
+Community
0.83 (0.75)
0.79 (0.72)
0.75 (0.69)
0.79 (0.72)
+Moderator
0.84 (0.75)
0.83 (0.73)
0.78 (0.72)
0.82 (0.73)
Prediction performance
change over number of post
 Performance seem to peak near to 10 post.
 In case user make to mach posts, it’s difficult to predict.
Prediction performance change over
“distance” from getting banned
 It becomes increasingly difficult to predict whether a user will
get banned the further in time the examined posts are from
when the user gets banned
Generalize the classifier
Conclusions
 The paper presents a data-driven study of antisocial behavior in
online discussion communities by analyzing users that are eventually
banned from a community.
 Leads to characterization of antisocial users and to an investigation
of the evolution of their behavior and of community response
 Also, the paper proposes a typology of antisocial users based on post
deletion rates.
 Finally, introduces a system for identifying undesired users early on
in their community life.
Potentially improvements
 A more fine-grained labeling of users may reveal a greater
range of behavior and lead to better performance.
 A better analysis of the content of posts, and of the relation
between the posts in a thread.
 Expend the analysis to temporary banned users and different
communities.
 Research for understanding how antisocial users may
steer individual discussions
Pay attention
 While we present effective mechanisms for identifying and weeding
antisocial users, taking extreme action against small infractions can
exacerbate antisocial behavior.
 Though average classifier precision is relatively high (0.80), one in
five users identified as antisocial are nonetheless misclassified.
 Possible solutions  Trading off overall performance for lower rate of such mistake.
 Employ a human moderator for approve any bans.
 Giving antisocial users a chance to redeem themselves.