Introduction to Data Science
Data All Around
Lots of data is being collected
and warehoused
Web data, e-commerce
Financial transactions, bank/credit transactions
Online trading and purchasing
Social Network
How Much Data Do We have?
Google processes 20 PB a day (2008)
Facebook has 60 TB of daily logs
eBay has 6.5 PB of user data + 50 TB/day (5/2009)
1000 genomes project: 200 TB
Cost of 1 TB of disk: $35
Time to read 1 TB disk: 3 hrs
(100 MB/s)
A single Jet engine can generate 10+terabytes of data
in 30 minutes of flight time. With many thousand flights
per day, generation of data reaches up to many Petabytes.
Each day 500 million tweets are sent.
Amazon, in order to recommend products, on average, handles more than 15 million+
customer clickstreams per day.
Walmart an American Multinational Retail Corporation handle about 1 million+ customer
transactions per hour.
65 billion+ messages are sent on Whatsapp every day.
On average, everyday 294 billion+ emails are sent.
Modern cars have close to 100 sensors for monitoring tire pressure, fuel level, etc. , thus
generating a lot of sensor data.
Facebook stores and analyzes more than 30 Petabytes of data generated by the users
each day.
YouTube users upload about 48 hours of video every minute of the day.
Big Data
Big Data is any data that is expensive to manage and hard to extract value from
Volume
Velocity
The size of the data
The latency of data processing relative to the growing demand for interactivity
Variety and Complexity
the diversity of sources, formats, quality, structures.
Big Data
Types of Data We Have
Relational Data (Tables/Transaction/Legacy Data)
Text Data (Web)
Semi-structured Data (XML)
Graph Data
Social Network, Semantic Web (RDF), …
Streaming Data
You can afford to scan the data once
What is Data Science?
“Data Science is about extraction, preparation, analysis,
visualization, and maintenance of information. It is a crossdisciplinary field which uses scientific methods and
processes to draw insights from data. ”
What is Data Science?
An area that manages, manipulates, extracts, and
interprets knowledge from tremendous amount of data
Data science (DS) is a multidisciplinary field of study
with goal to address the challenges in big data
Data science principles apply to all data – big and small
https://hbr.org/2012/10/data-scientist-the-sexiest-job-of-the-21st-century/
What is Data Science?
Theories and techniques from many fields and disciplines are used
to investigate and analyze a large amount of data to help decision
makers in many industries such as science, engineering, economics,
politics, finance, and education
Computer Science
Mathematics
Pattern recognition, visualization, data warehousing, High performance
computing, Databases, AI
Mathematical Modeling
Statistics
Statistical and Stochastic modeling, Probability.
Data Science
Data Science
Applications of data science
Augmented realities
Self driving cars
Robots
Data Scientists
Data Scientist
The Sexiest Job of the 21st Century
They find stories, extract knowledge. They are not
reporters
Data Scientist Roles and Responsibilities
Collect data and identify data sources
Analyze huge amounts of data, both structured and
unstructured
Create solutions and strategies to business problems
Work with team members and leaders to develop data
strategy
To discover trends and patterns, combine various
algorithms and modules
Present data using various data visualization techniques
and tools
Investigate additional technologies and tools for
developing innovative data strategies
Create comprehensive analytical solutions, from data gathering to
display; assist in the construction of data engineering pipelines
Supporting the data scientists, BI developers, and analysts team as
needed for their projects Working with the sales and pre-sales team
on cost reduction, effort estimation, and cost optimization
To boost general effectiveness and performance, stay current with
the newest tools, trends, and technologies
collaborating together with the product team and partners to provide
data-driven solutions created with original concepts
Create analytics solutions for businesses by combining various tools,
applied statistics, and machine learning
Lead discussions and assess the feasibility of AI/ML solutions for
business processes and outcomes
Architect, implement, and monitor data pipelines, as well as conduct
knowledge sharing sessions with peers to ensure effective data use
1
2
CRISP-DM is the most popular data mining
process model
3
Daimla Benz , ISL , NCR & OHRA
Founded at 1996
Non proprietary , documented , freely
available
4
5
1. Determine the business question and
objective:
What to solve from the business perspective,
what the customer wants, and define the
business success criteria
2. Situation Assessment:
assess the resources availability,
project requirements,
risks, and cost-benefit from this project.
6
3. Determine the project goals:
4. Project plan:
7
understand the scope and depth of the problem ,if we
make a mistake ,we end up spending a lot of time.
Key Questions must be asked in framing the problem:
•
•
•
•
•
What kind of a system would the company like to build?
What kind of data is available for us to use?
How many movies are there in the library?
How many movies should be there in a recommendation?
How are these recommendations going to be used?
8
9
10
Collect Data:
Describe data:
Explore data:
Verify data quality:
11
faulty, incorrect data is insufficient to solve
the problem
collect needs from reliable sources
Get data directly from customers with their
knowledge
websites using web scraping
12
• missing values in several rows or columns -fill them with zero or
fill them with the average
• could have many outliers, incorrect values, or values in timestamps
with different time zones,
• issues related to date ranges
• e.g., if the data is collected from multiple thermometers and any of
those are faulty
13
14
15
16
17
we can extract some patterns from our data, which can lead us to solve
our problem.
exploration can be performed using the visualizations and the numerical
summaries of the data and its columns.
For example, imagine we are analyzing data from an e-commerce
platform to help devise a meaningful strategy to attract more customers
to each product.
18
use statistical and numerical methods to draw inferences about the data.
identify the relationship between multiple columns in our dataset.
summarize the data using images, graphs, charts, plots, etc.
how different columns are related to each other by finding out their
correlation.
19
20
21
22
23
24
25
26
27
consolidate the results so that they can be analyzed and understood
by stakeholders.
create documents that justify our conclusions by describing the
insights and visualizations.
28
29
30
31
Similarity & dissimilarity
What is data?
• Data denotes a collection of objects and their attributes.
• An attribute (feature, variable, or field) is a property or characteristic
of an object.
• A collection of attributes describe an object (individual, entity, case,
or record).
3
4
5
6
7
8
• Proximity refers to either a similarity or dissimilarity
Similarity might be used to identify
• duplicate data that may have differences due to typos.
• equivalent instances from different data sets. E.g. names and/or
addresses that are the same but have misspellings.
• groups of data that are very close (clusters)
Dissimilarity might be used to identify
• outliers
• interesting exceptions, e.g. credit card fraud
• boundaries to clusters
Proximity measures for
• Nominal attributes
• Binary attributes
• Ordinal attributes
• Numerical attributes
• Mixed attributes
• Why Proximity measures ?
For clustering
Outlier analysis
Nearest neighbor classification
Mahalanobis distance:
• Measures distance between a point and a distribution
• Used for anomaly detection , classification
• Euclidean will work when dimensions are same
Area (sq ft)
Price ($1000)
Area (acre)
Price ($M)
2400
156000
0.0550944
156
1950
126750
0.0447642
126.75
2100
105000
0.0482076
105
1200
78000
0.0275472
78
2000
130000
0.045912
130
900
54000
0.0206604
54
• Euclidean distance gives different values even though distances are
same .
• This can be overcome by scaling.
• But still Euclidean distance between point and center of the
distribution point can give misleading information about how close a
point to the cluster.
• So Mahalanobis distance.
• It is introduced by Prof. P.C. Mahalanobis in 1936.
Supremum distance :
• d(x,y)=maxni=1 |xi-yi|
point
xi
yi
P1
0
2
d(p2-p1)=max(|0-2|,|0-2|}
2
0
= {2,2}
3
1
=2
5
1
•
P2
•
P3
•P4
L
p1
p2
p3
P1
0
P2
2
0
P3
3
1
0
p4
5
3
2
p4
0
Bhattacharya distance :
• Measures the similarity of two probability distributions
• Developed by Anil Kumar Bhattacharya.
• More reliable than mahalanobis distance
• It is a generalization of mahalanobis distance
• Bhattacharya distance =-log(BC(P,Q))
• Where BC-bhattacharya coefficient
• BC(P,Q)=sum((P(x)*Q(x))^0.5
Similarity measures for symmetric and
asymmetric binary data
• Binary Attributes: Binary data has only 2 values/states. For Example
yes or no, affected or unaffected, true or false.
• Symmetric: Both values are equally important (Gender).
• Asymmetric: Both values are not equally important (test Result).
• SMC would say all transactions are very similar.
Hamming distance :
• Compare binary strings
• Calculates distance between two binary vectors
• Compute by counting the no of features which have different values .
• Used for error detection or error correction while transfer on
network.
• Eg D1=1010010
• D2=0011001
• H. distance=1+0+0+1+0+1+1
•
=4
Similarity measure for textual data :
Text similarity : how close the two pieces of text
Applications :
• Search engines
• Legal matter
• Customer services
Jaccard Similarity
• The Jaccard similarity (Jaccard coefficient) of two sets S1,
S2 is the size of their intersection divided by the size of
their union.
• JSim (C1, C2) = |C1C2| / |C1C2|.
3 in intersection.
8 in union.
Jaccard similarity
= 3/8
• Extreme behavior:
• Jsim(X,Y) = 1, iff X = Y
• Jsim(X,Y) = 0 iff X,Y have no elements in common
• JSim is symmetric
40
Disadvantage :
• JS neither able to capture semantic similarity nor lexical semantic of
two sentences .
• As the size of document increase the no of common words tend to
increase even if document talk about different topic.
Cosine Similarity
• Sim(X,Y) = cos(X,Y)
• The cosine of the angle between X and Y
• If the vectors are aligned (correlated) angle is zero degrees and
cos(X,Y)=1
• If the vectors are orthogonal (no common coordinates) angle is 90
degrees and cos(X,Y) = 0
• Cosine is commonly used for comparing documents, where we
assume that the vectors are normalized by the document length.
• Given two objects represented by tuples (22,1,42,10) and (20,0,36,8)
• Compute the Euclidian distance between two objects
• Compute the Manhattan distance between two objects .
• Compute the Minkowski distance between two objects using q=3.
1 . 6.708
2. 11
3. 6.1534
How similar are two strings?
• Spell correction
• The user typed “graffe”
Which is closest?
• graf
• graft
• grail
• giraffe
• Also for Machine Translation, Information Extraction, Speech Recognition
Edit Distance
• The minimum edit distance between two strings
• Is the minimum number of editing operations
• Insertion
• Deletion
• Substitution
• Needed to transform one into the other
Minimum Edit Distance
• Two strings and their alignment:
Minimum Edit Distance
• If each operation has cost of 1
• Distance between these is 5
• If substitutions cost 2 (Levenshtein)
• Distance between them is 8
The Edit Distance Table
N
9
8
9
10
11
12
11
10
9
8
O
8
7
8
9
10
11
10
9
8
9
I
7
6
7
8
9
10
9
8
9
10
T
6
5
6
7
8
9
8
9
10
11
N
5
4
5
6
7
8
9
10
11
10
E
4
3
4
5
6
7
8
9
10
9
T
3
4
5
6
7
8
7
8
9
8
N
2
3
4
5
6
7
8
7
8
7
I
1
2
3
4
5
6
7
6
7
8
#
0
1
2
3
4
5
6
7
8
9
#
E
X
E
C
U
T
I
O
N
Jaro Similarity
Jaro Similarity is the measure of similarity between two strings. The value of Jaro
distance ranges from 0 to 1. where 1 means the strings are equal and 0 means no
similarity between the two strings.
Examples:
Input: s1 = “CRATE”, s2 = “TRACE”;
Output: Jaro Similarity = 0.733333
Input: s1 = “DwAyNE”, s2 = “DuANE”;
Output: Jaro Similarity = 0.822222
Jaro distance :
0
