Twitter, Big Data, and
Other Ramblings
Robert Dittmer
Perspective on those V words

Volume- 1% of the Twitter stream for roughly one month was about 68 million
Tweets. Now multiply that by 100. Facebook has the same problem.

Velocity- How do you analyze thousands of points of data in real-time? SQL
Server sure isn’t going to do that.

Variety- Social Media, Manufacturing, Sales, Financial, CRM, Web Traffic,
External


Think about what goes into Amazon recommending you a book or movie
Veracity- It all means nothing if it’s not at least somewhat clean
What do you do with a Tweet?

Sentiment Analysis is assigning a numerical value to a word



Positive, Negative, Neutral connotation
Methods for performing Sentiment Analysis

“Dumb” Method- Break down text into individual words and compare with a
sentiment dictionary. AKA “Bag of Words”

“Smart” Method- Use a natural language processing tool to analyze parts of speech
and calculate sentiment based on context
Example Tweet

“The Apple iPad sucks. The new Google Nexus 7 is awesome!”
Collecting Tweets

Twitter uses a RESTful service to stream Tweets

Steps to start streaming your own Tweets

Go to dev.twitter.com and create an application

Generate your OAuth credentials

Find an open-source Twitter library


Tweepy (Python)

Tweetinvi (C#)
Plug your credentials in and modify the example
The Tweet, the Whole Tweet, and
Nothing but the Tweet

JSON Format (Key-Value Pair)

Notable Fields

ID

CreatedAt

Text

Entities


Hashtags

URLs
Latitude, Longitude
What does a Tweet look like?

{"filter_level":"medium","contributors":null,"text":"Iron man 3 was awesome
=)","geo":{"type":"Point","coordinates":[50.73529254,4.00720746]},"retweeted":false,"in_reply_to_screen_name":null,"truncated":false,"lang":"en","entities":{"symbols":[],"urls":[],"ha
shtags":[],"user_mentions":[]},"in_reply_to_status_id_str":null,"id":330043889589288960,"source":"<a
href=\"http://twitter.com/download/android\" rel=\"nofollow\">Twitter for
Android<\/a>","in_reply_to_user_id_str":null,"favorited":false,"in_reply_to_status_id":null,"retweet_count":0,"created_at":"Thu
May 02 19:39:29 +0000
2013","in_reply_to_user_id":null,"favorite_count":0,"id_str":"330043889589288960","place":{"id":"0613276b16c0d59f","bounding_b
ox":{"type":"Polygon","coordinates":[[[-4.335135,50.429347],[-4.335135,50.874614],[-3.732303,50.874614],[3.732303,50.429347]]]},"place_type":"city","name":"West
Devon","attributes":{},"country_code":"GB","url":"http://api.twitter.com/1/geo/id/0613276b16c0d59f.json","country":"United
Kingdom","full_name":"West Devon,
Devon"},"user":{"location":"okehampton","default_profile":false,"statuses_count":1345,"profile_background_tile":true,"lang":"en",
"profile_link_color":"FC0AFC","profile_banner_url":"https://si0.twimg.com/profile_banners/503242961/1354459551","id":503242
961,"following":null,"favourites_count":492,"protected":false,"profile_text_color":"0084B4","description":"vicky pollards twin
sister ( the nice one )","verified":false,"contributors_enabled":false,"profile_sidebar_border_color":"FFFFFF","name":"vicki
phillips ","profile_background_color":"FA03DD","created_at":"Sat Feb 25 16:40:53 +0000
2012","default_profile_image":false,"followers_count":149,"profile_image_url_https":"https://si0.twimg.com/profile_images/30
83034337/fb9a8158c125dbb5a0650f58206880e0_normal.jpeg","geo_enabled":true,"profile_background_image_url":"http://a0.tw
img.com/profile_background_images/636836788/xb893f8eb29554020cb59540210c070b.jpg","profile_background_image_url_htt
ps":"https://si0.twimg.com/profile_background_images/636836788/xb893f8eb29554020cb59540210c070b.jpg","follow_request_
sent":null,"url":"http://www.facebook.com/vickipolard","utc_offset":0,"time_zone":"Casablanca","notifications":null,"profile_use
_background_image":true,"friends_count":1059,"profile_sidebar_fill_color":"DDEEF6","screen_name":"vixoakleophill","id_str":"503
242961","profile_image_url":"http://a0.twimg.com/profile_images/3083034337/fb9a8158c125dbb5a0650f58206880e0_normal.jp
eg","listed_count":0,"is_translator":false},"coordinates":{"type":"Point","coordinates":[-4.00720746,50.73529254]}}
My Tweet Collection


Collected for roughly one month

Lots of trial and error

Originally used Tweepy, but ran into errors

Switched to Tweetinvi and it worked
About 68 million Tweets

Apple

Amazon

Google

Microsoft

Netflix

Tesla

Ford (Probably should have used a different car company)
Yahoo! Finance Detour

Use an HTTP request to get stock data

http://fiance.yahoo.com/d/quotes.csv?s=AAPL+GOOG+MSFT+YHOO+NFLX+AM
ZN+TSLA+F&f=snb2b3opl1t1d1

Create a metric with stock data and compare the sentiment of a company to
their performance
Big Data (and regular data) Tools

Talend Open Studio

Hadoop

SAP HANA
Talend Open Studio

Open Source ETL Tool


Built on Eclipse
Data Quality and Format Issues

Even though I saved Tweets in delimited format, issues remained

Iterated through all 12,736 files with 5000 tweets each

Verified each row against a schema

Mapped to different output files


Tweet (Fact table)

Tracks

User Mentions

Hashtags

URLs
Demo Time!
Hadoop Overview

Based on the Hadoop Distributed File System and MapReduce

MapReduce is a way of parallelizing code using batch processing

Map finds the data you’re looking for

Reduce aggregates that data (count, sum, average)

Embarrassingly parallel processing

Each server in a Hadoop cluster is referred to as a Node


NameNode

DataNode
Blocks of data are replicated to three nodes

Extremely fault tolerant
More Hadoop

Open-source technology

Cloudera vs. Hortonworks

Intel, IBM, MapR, Amazon EMR

Cloudera and Hortonworks are the two biggest faces of Hadoop

Intel actively contributes to optimize it for Xeon Processors

IBM and MapR also involved

Big companies and entities use it
Hadoop Projects

Hive

Data Warehouse on top of Hadoop

Uses HiveQl (essentially SQL with a few extras) to query data




Abstracts MapReduce processes
Has an ODBC connector to allow it talk to anything that talks to databases
Pig

Uses a language called Pig Latin to analyze data

Data flow language abstracts MapReduce for easy use for data analysts
HBase

Billions of rows and millions of columns

Distributed column data store
Hadoop Trivia Time

Who created Hadoop?

Why is it called Hadoop?

Who developed the concept of MapReduce?

What does Facebook Messenger use to store its data?

Who created Hive?

What is Accumulo and who created it?
2nd Generation Hadoop


Much faster than previous versions

Hive 0.12 is up to 50X faster than previous versions

Hortonworks Stinger project aims for 100X performance improvement
Projects like Spark are moving towards real-time analysis

In-memory cluster compute analysis

Streaming processing with routines written in Python and Scala

Shark is an implementation of Hive using Spark instead of MapReduce
Hadoop Sentiment Analysis

Used the “Dumb” method of Sentiment Analysis

Import the data into HDFS and create Hive tables

Tweet

Sentiment Dictionary

Explode words in each tweet to create a view with TweetID and Word

Join with the Sentiment Dictionary on the word to get sentiment value

Demo Time!
SAP HANA

In-Memory, Column-Store database

Loads all data into main-memory


Analyze billion of rows with sub-second response time
Column-store table structure

Allows for much better compression and parallelization than row-store

Used for real-time analytics

Available with an on premise appliance or cloud-based VM
Why is SAP HANA Awesome?

Column-stores are naturally very good at parallelization

In-Memory means no waiting on IO from disks and is still hundreds of times
faster than SSD

Feature rich


Text analytics

Predicative Analytics Library

Application Server

It is an actual Database and does everything a database does
Demo Time
SAP HANA Sentiment Analysis

Sentiment is calculated when creating a full-text index on the text of the
tweet

Creates a sentiment value for each tweet

Analyze by my different dimensions

Aggregate sentiment by hour

Demo Time!
Other Text Analysis Options

Python Natural Language Toolkit


Analyze parts of speech and context
Should be possible to integrate with Hadoop (The Google did not help)
Other Big Data Problems

A GE Engine on a transatlantic flight generates 2TB of sensor data

There’s four engines on a 747

What does the LHC at CERN do with their 15 petabytes of data they create
annually?

How does the NSA store a yottabyte of data?

How does a small online gaming company analyze their customer base to
increase retention and margins?
How is Sentiment Analysis Being Used?

Companies ingest their social media feeds into these systems

If a Tweet or Facebook post meets a certain criteria, an automated or human
response can be requested
Hot vs. Cold Data



Hot data is the recent data you are most interested in

Keep this data in SAP HANA for real-time processing

Archive it after a period of time: 1 month, 3 months, 6 months, etc…
Cold Data is your historical data

Data warehouses that can handle massive volumes of data are EXPENSIVE!!!!

Use Hadoop and Hive as your data warehouse

It only costs the hardware
Still able to analyze cold data, store it cheaply, and integrate with SAP HANA