Don’t Follow me : Spam Detection in Twitter Alex Hai Wang The Pensylvania State University International Conference on Security and Cryptography, 2010 January 12, 2011 In-seok An SNU Internet Database Lab. Outline Introduction Social Graph model Features Data Set Spam Detection Experiments Evaluation Conclusion 2 / 37 Introduction Social Network Service ( SNS ) – An online service, platform, or site that focuses on building and reflecting of social networks or social relations among people – The most popular applications of Web 2.0 Twitter – Founded in 2006 – One of the fastest growing SNSs Surging more than 2,800% in 2009 – Social networking site and microblogging service 3 / 37 Introduction Twitter You can post your latest updates Messages(Tweets) from twitter that you are following( describing ) 4 / 37 Introduction Spammer in Twitter – The goal of Twitter Allow friends to communicate and stay connected through the exchange of short message – Spammer also use Twitter as a tool to post malicious links – More than 3% messages are spam on Twitter ( Analytics, 2009 ) – The offensive trending topic Attack on February 20 ( CNET, 2009 ) 5 / 37 Introduction Method to report spam – By clicking on the “report as spam” – To post a tweet in the “@spam @username” This report service is also abused by both hoaxes and spam Legitimate user can be mistakenly suspended by Twitter’s anti spam action 6 / 37 Outline Introduction Social Graph model Features Data Set Spam Detection Experiments Evaluation Conclusion 7 / 37 Social Graph model Twitter can be modeled as a directed graph – G = (V ,A ) – V : a set of nodes ( vertices ) – A : a set of arcs ( Edges ) Four types of relationships on Twitter can be defined – Follower Node v j is a follower of node vi if the arc a = ( j , i ) is contained in A – Friend Node v j is a friend of node vi if the arc a = ( i, j ) is contained in A – Mutual Friend Node vi and node v j are mutual friends if both arcs a = ( i , j ) and a = ( j , i ) are contained in A – Stranger Node vi and node vi are strangers if neither arcs a = ( I , j ) nor a = ( j , I ) is contained in A 8 / 37 Social Graph model A simple Twitter graph A follows B B follows C, C follows B A is follower of B B is friend of A B and C are Mutual friend A doesn’t follow C, C doesn’t follow A A and C are stranger 9 / 37 Social Graph model Twitter Social Graph 10 / 37 Outline Introduction Social Graph model Features – Graph-based features – Content-based features Data Set Spam Detection Experiments Evaluation Conclusion 11 / 37 Features Graph-based features Twitter’s spam and abuse policy – “if you have a small number of followers compared to the amount of people you are following, it may be considered as a spam account” Three features – The number of friends The indegree d I (vi ) of a node vi – The number of followers The outdegree d (v ) of a node vi O i – The reputation of a user R(vi ) d I (vi ) d I (vi ) d O (vi ) 12 / 37 Features Content-based features Duplicate Tweets – An account may be considered as a spam if you post duplicate content on one account – Detected by measuring the Levenshtein distance ( edit distance ) Minimum cost of transforming one string into another through a sequence of edit operations ( deletion , insertion and substitution of individual symbols ) Clean the data by stopping the words containing “@”, “#”, “http://” and “www.” – The number of duplicate tweets can be measurement In the user’s 20 most recent tweets Two tweets are considered as duplicate only when the are exactly the same 13 / 37 Features Content-based features Need for cleaning 14 / 37 Features Content-based features HTTP Links – It is considered as spam if your updates consist mainly of links and not personal updates – Twitter filters out the URLs linked to known malicious sites URL shorten services like bit.ly provides opportunity for attacker to spam – The number of tweets containing HTTP links can be measurement ?? http://bit.ly/ab3cd http://porno.com http://bit.ly/ab3cd ↓ http:// porno.com Tweet Tweet with with HTTP HTTP link link URL shorten service 15 / 37 Malicious MaliciousSite Site Features Content-based features Replies and Mentions – You can send a reply message to another user @username + message – You can also mention another @username anywhere in the tweet Message + @username + message – Twitter automatically collects all tweets containing your username – You can reply anyone no matter they are your friends/followers or not – Spammer abuses this feature – The number of Tweets containing mention or reply can be measurement 16 / 37 Features Content-based features Spam tweets using mention or reply 17 / 37 Features Content-based features Trending topic – The most-mentioned terms on Twitter at that moment, week, month – User can use the hashtag to a tweet #tagname – If there are many tweets containing the same term, It may become a trending topic – Twitter considers an account as spam If you post multiple unrelated updates to a topic using the # symbols 18 / 37 Outline Introduction Social Graph model Features Data Set Spam Detection Experiments Evaluation Conclusion 19 / 37 Data Set Data Set – – – – 3 weeks from January 3 to January 24, 2010 25,847 users 500k tweets 49M follower/friend relationships 20 / 37 Outline Introduction Social Graph model Features Data Set Spam Detection Experiments Evaluation Conclusion 21 / 37 Spam Detection Several classification algorithms – – – – – Decision tree Neural network Support vector machines K – nearest neighbers Naïve Bayesian Naïve Bayesian outperform all other method – Bayesian classifier is noise robust It uses posterior probability – A spam probability is calculated for each individual user based its behaviors, instead of giving a general rule 22 / 37 Spam Detection Naïve Bayesian classifier P( X | Y ) P(Y ) P(Y | X ) P( X ) – X : each Twitter account is considered as a vector X with feature values – Y : one of two classes, spam and non-spam – The features are conditionally independent 23 / 37 Outline Introduction Social Graph model Features Data Set Spam Detection Experiments Evaluation Conclusion 24 / 37 Experiments To evaluate the detection method – 500 Twitter user accounts are labeled manually to two classes( spam or not ) By reading the 20 most recent tweets Checking the friends and followers of the user – Result show that there are around 1% spam account in the data set Additional spam data are added to the data set To simulate the reality and avoid the bias in the crawling and label methods – The study in Analytics, 2009, shows there is 3% spam on Twitter Search @spam on Twitter and collect additional spam data – Only small number of result report real spam – The data set is mixed to contain around 3% spam data 25 / 37 Experiments Graph-based features – The number of friends for each Twitter account – Only 30% of spam accounts follow a large amount of user Spammer doesn’t need to follow other user 26 / 37 Experiments Graph-based features – The number of followers for each Twitter account – Usually the spam accounts do not have a large amount of followers Some spam accounts having a relatively large amount of followers 27 / 37 Experiments Graph-based features – The reputation for each Twitter account – The reputation of most legitimate users is between 30% to 90% Some spam accounts have a 100% reputation 28 / 37 Experiments Content-based Features – The number of pairwise duplication – Not all spam accounts post multiple duplicate tweets We can not only depend on this feature 29 / 37 Experiments Content-based Features – The number of mentions and replies – Most spam accounts have the maximum 20 “@” symbol This will lure legitimate users to read their spam messages or click their link 30 / 37 Experiments Content-based Features – The number of links – Some legitimate users also include links in all tweets, some companies join Twitter to promote their own web sites 31 / 37 Experiments Content-based Features – The number of Hash tag signs 32 / 37 Outline Introduction Social Graph model Features Data Set Spam Detection Experiments Evaluation Conclusion 33 / 37 Evaluation The evaluation of the overall process – Confusion matrix – Precision : P = a / ( a + c ) – Recall : R = a / ( a + b ) – F-measure : F = 2PR / ( P + R ) Each classifier is trained 10 times – Each time using the 9 out of the 10 partitions as training data – Computing the confusion matrix using the tenth partition as test data 34 / 37 Evaluation The evaluation results – Naïve Bayesian classifier has the best overall performance Finally, the Bayesian classifier learned from the labeled data is applied to the entire data set – Information about totally 25,817 users – Precision of the spam detection system 392 users are classified as spam 348 users are real spam account and 44 users are false alarms 89% precision 35 / 37 Conclusion The spam behavior in a popular online SNS, Twitter – To formalize the problem, social graph model is proposed Novel content-based and graph-based features are proposed – Graph-based features The number of friends The number of followers The reputation of the user – Content-based features The number of pairwise duplications The number of Mention and Replies The number of Links The number of Hashtags Analyze the data set and evaluate the performance of the detection system 36 / 37 Conclusion Among the graph-based features – The proposed reputation features has the best performance – No many spam follow large amount of users – Some spammers have many followers For the content-based features – Most spam accounts have multiple duplicate tweets – But not all spam account post multiple duplicate tweets We can not rely on this feature Several popular classification algorithms are studied and evaluated The naïve classifier achieve a 89% precision 37 / 37