Stinger Initiative: Deep Dive
Interactive Query on Hadoop
Chris Harris
E-Mail : charris@hortonworks.com
Twitter : cj_harris5
© Hortonworks Inc. 2013
Page 1
Agenda
• Key Hive Use Cases
• Brief Refresher on Hive
• The Stinger Initiative: Interactive Query for Hive
© Hortonworks Inc. 2013
Page 2
Key Hive Use Cases
• RDBMS / MPP Offload
– More data under query.
– Database unable to keep up with SLAs.
• Analysis of semi-structured data.
• ETL / Data Refinement
• +++ Increasingly: Business Intelligence and
interactive query
© Hortonworks Inc. 2013
Page 3
BI Use Cases
Enterprise Reports
Dashboard / Scorecard
Visualization
© Hortonworks Inc. 2013
Parameterized Reports
Data Mining
Page 4
Organize Tiers and Process with Metadata
Access
Tier
Conform, Summarize, Access
• Organize data
based on
source/derived
relationships
• Allows for fault
and rebuild
process
Pig
HiveQL
Gold
Tier
Transform, Integrate, Storage
Pig
MapReduce
Work
Tier
Standardize, Cleanse, Transform
Pig
MapReduce
HCat
Provides unified
metadata access
to Pig, Hive &
MapReduce
Raw
Tier
Extract & Load
© Hortonworks Inc. 2013
WebHDF
S
Flume
Sqoop
Page 5
Hive Current Focus Area
Real-Time
• Online systems
• R-T analytics
• CEP
NonInteractive
Batch
• Data preparation
• Incremental batch
processing
• Dashboards /
Scorecards
• Operational batch
processing
• Enterprise
Reports
• Data Mining
Interactive
• Parameterized
Reports
• Drilldown
• Visualization
• Exploration
Current Hive Sweet Spot
0-5s
1m – 1h
5s – 1m
1h+
Data Size
© Hortonworks Inc. 2013
Page 6
Stinger: Extending Hive’s Sweetspot
Real-Time
• Online systems
• R-T analytics
• CEP
NonInteractive
Batch
• Data preparation
• Incremental batch
processing
• Dashboards /
Scorecards
• Operational batch
processing
• Enterprise
Reports
• Data Mining
Interactive
• Parameterized
Reports
• Drilldown
• Visualization
• Exploration
Future Hive
Expansion
0-5s
Current Hive Sweet Spot
1m – 1h
5s – 1m
1h+
Data Size
Improve Latency & Throughput
• Query engine improvements
• New “Optimized RCFile” column store
• Next-gen runtime (elim’s M/R latency)
© Hortonworks Inc. 2013
Extend Deep Analytical Ability
• Analytics functions
• Improved SQL coverage
• Continued focus on core Hive use cases
Page 7
The top BI vendors support Hive today
© Hortonworks Inc. 2013
Page 8
Agenda
• Key Hive Use Cases
• Brief Refresher on Hive
• The Stinger Initiative: Interactive Query for Hive
© Hortonworks Inc. 2013
Page 9
Brief Refresher on Hive
The State of Hive Today (0.10)
© Hortonworks Inc. 2013
Page 10
Hive’s Origins
Hive was originally developed at Facebook.
More data than existing RDBMS could handle.
60,000+ Hive queries per day.
More than 1,000 users per day.
100+ PB of data.
15+ TB of data loaded daily.
Hive is a proven solution at extreme scale.
© Hortonworks Inc. 2013
Page 11
Hive 0.10 Capabilities
• De-facto SQL Interface for Hadoop
• Multiple persistence options:
– Flat text for simple data imports.
– Columnar format (RCFile) for high performance processing.
• Secure and concurrent remote access
• ODBC/JDBC connectivity
• Highly extensible:
– Supports User Defined Functions and User Defined Aggregation
Functions.
– Ships with more than 150 UDF/UDAF.
– Extensible readers/writers can process any persisted data.
• Support from 10+ BI vendors
© Hortonworks Inc. 2013
Page 12
HDP 1.2: ODBC Access for Popular BI Tools
• Seamless integration with BI
tools such as Excel, PowerPivot,
MicroStrategy, and Tableau
Applications &
Spreadsheets
Visualization &
Intelligence
• Efficiently maps advanced SQL
functionality into HiveQL
ODBC
Hortonworks
Data Platform
– With configurable pass-through of
HiveQL for Hive-aware apps
• ODBC 3.52 standard compliant
• Supports Linux & Windows
High quality ODBC driver developed in partnership with Simba.
Free to download & use with Hortonworks Data Platform.
© Hortonworks Inc. 2013
Page 13
0 to Big Data in 15 Minutes
Hands on tutorials
integrated into
Sandbox
© Hortonworks Inc. 2013
HDP environment for
evaluation
Page 14
Agenda
• Brief Refresher on Hive
• Key Hive Use Cases
• The Stinger Initiative: Interactive Query for Hive
© Hortonworks Inc. 2013
Page 15
The Stinger Initiative
Interactive Query on Hadoop
© Hortonworks Inc. 2013
Page 16
Stinger Initiative: 2-Pronged Approach
Making Hive Best for Interactive Query
Improve Latency and Throughput
Extend Deep Analytical Ability
Tez
Analytics Functions
• New primitives move beyond map-reduce
and beyond batch
• Avoid unnecessary persistence of
temporary data
• Hive, Pig and others generate Tez plans
for high perf
• SQL:2003 Compliant
• OVER with PARTITION BY and ORDER
BY
• Wide variety of windowing functions:
• RANK
• LEAD/LAG
• ROW_NUMBER
• FIRST_VALUE
• LAST_VALUE
• Many more
• Aligns well with BI ecosystem
Query Engine Improvements
•
•
•
•
Cost-based optimizer
In-memory joins
Caching hot tables
Vector processing
State-of-the-art Column Store
• “Optimized RCFile” or ORCFile
• Minimizes disk IO and deserialization
Improved SQL Coverage
• Non-correlated Subqueries using IN in
WHERE
• Expanded SQL types including
DATETIME, VARCHAR, etc.
Tez Service
• Always-on service for query interactivity
© Hortonworks Inc. 2013
Page 17
Hive: Performance Improvements
© Hortonworks Inc. 2013
Page 18
Stinger Initiative At A Glance
© Hortonworks Inc. 2013
Page 19
Base Optimizations: Intelligent Optimizer
• Introduction of In-Memory Hash Join:
– For joins where one side fits in memory:
– New in-memory-hash-join algorithm.
– Hive reads the small table into a hash table.
– Scans through the big file to produce the output.
• Introduction of Sort-Merge-Bucket Join:
– Applies when tables are bucketed on the same key.
– Dramatic speed improvements seen in benchmarks.
• Other Improvements:
– Lower the footprint of the fact tables in memory.
– Enable the optimizer to automatically pick map joins.
© Hortonworks Inc. 2013
Page 20
Dimensionally Structured Data
• Extremely common pattern in EDW.
• Results in large “fact tables” and small “dimension
tables”.
• Dimension tables often small enough to fit in RAM.
• Sometimes called Star Schema.
© Hortonworks Inc. 2013
Page 21
A Query on Dimensional Data
• Derived from TPC-DS Query 27
SELECT col5, avg(col6)
FROM fact_table
join dim1 on (fact_table.col1
join dim2 on (fact_table.col2
join dim3 on (fact_table.col3
join dim4 on (fact_table.col4
GROUP BY col5
ORDER BY col5
LIMIT 100;
=
=
=
=
dim1.col1)
dim2.col1)
dim3.col1)
dim4.col1)
• Dramatic speedup on Hive 0.11
© Hortonworks Inc. 2013
Page 22
Star Schema Join Improvements in 0.11
© Hortonworks Inc. 2013
Page 23
Hive: Bucketing
• Bucketing causes Hive to physically co-locate rows
within files.
• Buckets can be sorted or unsorted.
CREATE EXTERNAL TABLE IF NOT EXISTS test_table
(
Id INT, name String
)
PARTITIONED BY (dt STRING, hour STRING)
CLUSTERED BY(country,continent) SORTED BY(country,continent) INTO n BUCKETS
ROW FORMAT DELIMITED FIELDS TERMINATED BY '|'
LOCATION '/home/test_dir';
© Hortonworks Inc. 2013
Page 24
ORCFile - Optimized Column Storage
• Make a better columnar storage file
– Tightly aligned to Hive data model
• Decompose complex row types into primitive fields
– Better compression and projection
• Only read bytes from HDFS for the required columns.
• Store column level aggregates in the files
– Only need to read the file meta information for common queries
– Stored both for file and each section of a file
– Aggregates: min, max, sum, average, count
– Allows fast access by sorted columns
• Ability to add bloom filters for columns
– Enables quick checks for whether a value is present
© Hortonworks Inc. 2013
Page 25
Performance Futures - Vectorization
• Operates on blocks of 1K or more records, rather than
one record at a time
• Each block contains an array of Java scalars, one for
each column
• Avoids many function calls, virtual dispatch, CPU pipeline
stalls
• Size to fit in L1 cache, avoid cache misses
• Generate code for operators on the fly to avoid branches
in code, maximize deep pipelines of modern processers
• Up to 30x faster processing of records
• Beta possible in 2H 2013
© Hortonworks Inc. 2013
Page 26
Performance Futures – Cost-Based
Optimizer
• Generate more intelligent DAGs based on properties of
data being queried, e.g. table size, statistics, histograms,
etc.
© Hortonworks Inc. 2013
Page 27
Performance Futures - Buffering
• Query workloads always have hotspots:
– Metadata
– Small dimension tables
• Build into YARN or Tez Service ways of buffering
frequently used data into memory so it is not always read
from disk.
• Part of the “last mile” of latency efforts.
© Hortonworks Inc. 2013
Page 28
Yarn
Moving Hive and Hadoop beyond MapReduce
© Hortonworks Inc. 2013
Page 29
Hadoop 2.0 Innovations - YARN
• Focus on scale and innovation
– Low latency, Streaming, Services
– Do more with a single Hadoop cluster
© Hortonworks Inc. 2013
Other
• Supports new frameworks beyond
MapReduce
Graph Processing
– Improves MapReduce performance
Tez
• Next generation execution
MapReduce
– Support 10,000+ computer clusters
– Extensible to encourage innovation
YARN: Cluster Resource Management
HDFS
Redundant, Reliable Storage
Tez
Moving Hive and Hadoop beyond MapReduce
© Hortonworks Inc. 2013
Page 31
Tez
• Low level data-processing execution engine
• Use it for the base of MapReduce, Hive, Pig,
Cascading etc.
• Enables pipelining of jobs
• Removes task and job launch times
• Hive and Pig jobs no longer need to move to the end
of the queue between steps in the pipeline
• Does not write intermediate output to HDFS
– Much lighter disk and network usage
• Built on YARN
© Hortonworks Inc. 2013
Page 32
Tez - Core Idea
Task with pluggable Input, Processor & Output
Input
Processor
Output
Task
Tez Task - <Input, Processor, Output>
YARN ApplicationMaster to run DAG of Tez Tasks
© Hortonworks Inc. 2013
Page 33
Tez – Blocks for building tasks
MapReduce ‘Map’
HDFS
Input
Map
Processor
MapReduce ‘Reduce’
Sorted
Output
Shuffle
Input
Reduce
Processor
HDFS
Output
MapReduce ‘Map’ Task
MapReduce ‘Reduce’ Task
Intermediate ‘Reduce’
for
Map-Reduce-Reduce
Shuffle
Input
Reduce
Processor
Sorted
Output
Intermediate ‘Reduce’ for MapReduce-Reduce
© Hortonworks Inc. 2013
Page 34
Tez – More tasks
Special Pig/Hive ‘Map’
HDFS
Input
Map
Processor
In-memory Map
Pipeline
Sorter
Output
HDFSIn
put
Map
Processor
Inmemory
Sorted
Output
Tez Task
Tez Task
Special Pig/Hive
‘Reduce’
Shuffle
Skipmerge
Input
Reduce
Processor
Sorted
Output
Tez Task
© Hortonworks Inc. 2013
Page 35
Pig/Hive-MR versus Pig/Hive-Tez
SELECT a.state, COUNT(*), AVERAGE(c.price)
FROM a
JOIN b ON (a.id = b.id)
JOIN c ON (a.itemId = c.itemId)
GROUP BY a.state
Job 1
Job 2
I/O Synchronization
Barrier
I/O Synchronization
Barrier
Single Job
Job 3
Pig/Hive - MR
© Hortonworks Inc. 2013
Pig/Hive - Tez
Page 36
FastQuery: Beyond Batch with YARN
Tez Generalizes Map-Reduce
Always-On Tez Service
Simplified execution plans process
data more efficiently
Low latency processing for
all Hadoop data processing
© Hortonworks Inc. 2013
Page 37
Tez Service
• MR Query Startup Expensive
– Job launch & task-launch latencies are fatal for short queries (in
order of 5s to 30s)
• Solution
– Tez Service
– Removes task-launch overhead
– Removes job-launch overhead
– Hive/Pig
– Submit query-plan to Tez Service
– Native Hadoop service, not ad-hoc
© Hortonworks Inc. 2013
Page 38
Tez Service Delivers Low Latency
SELECT a.state, COUNT(*), AVERAGE(c.price)
FROM a
JOIN b ON (a.id = b.id)
JOIN c ON (a.itemId = c.itemId)
GROUP BY a.state
Existing Hive
Hive/Tez
Tez and Tez Service
Parse Query
0.5s
Parse Query
0.5s
Parse Query
0.5s
Create Plan
0.5s
Create Plan
0.5s
Create Plan
0.5s
Launch Map-Reduce
20s
Launch Map-Reduce
20s
Submit to Tez Service
0.5s
Process MapReduce
10s
Process MapReduce
2s
Process Map-Reduce
2s
Total
31s
Total
23s
© Hortonworks Inc. 2013
Total
3.5s
* Numbers for illustration only
Page 39
Recap and Questions: Hive Performance
© Hortonworks Inc. 2013
Page 40
Improving Hive’s SQL Support
© Hortonworks Inc. 2013
Page 41
Stinger: Deep Analytical Capabilities
• SQL:2003 Window Functions
– OVER clauses
– Multiple PARTITION BY and ORDER BY supported
– Windowing supported (ROWS PRECEDING/FOLLOWING)
– Large variety of aggregates
– RANK
– FIRST_VALUE
– LAST_VALUE
– LEAD / LAG
– Distrubutions
© Hortonworks Inc. 2013
Page 42
Hive Data Type Conformance
• Data Types:
– Add fixed point NUMERIC and DECIMAL type (in progress)
– Add VARCHAR and CHAR types with limited field size
– Add DATETIME
– Add size ranges from 1 to 53 for FLOAT
– Add synonyms for compatibility
– BLOB for BINARY
– TEXT for STRING
– REAL for FLOAT
• SQL Semantics:
– Sub-queries in IN, NOT IN, HAVING.
– EXISTS and NOT EXISTS
© Hortonworks Inc. 2013
Page 43
Questions?
© Hortonworks Inc. 2013
Page 44
Thank You!
Questions & Answers
© Hortonworks Inc. 2013
Page 45