Les Conférences
Groupe des Utilisateurs SQL Server
Juin 2013 – SQL Server in-memory
Alexandre Chemla – Masao
Frédéric Pichaut – Microsoft
Client App
TDS Handler and Session Management
Natively Compiled
SPs and Schema
Proc/Plan cache for ad-hoc TSQL and SPs
Parser,
Catalog,
Algebrizer,
Optimizer
Hekaton
Compiler
Non-durable
Table
Interpreter for TSQL, query plans,
expressions
Access Methods
Query
Interop
Hekaton Engine: Memory_optimized Tables
& Indexes
Key
T1
T2
T3
T4 Tables
T1
T2
T3
T4 Indexes
Buffer Pool for Tables & Indexes
SQL Server.exe
Checkpoint & Recovery
Memory-optimized Table Filegroup
T1
T2
T3
T4
T1
T2
T3
T4
T1
T2
T3
T4
T1
T2
T3
T4
Transaction Log
Data Filegroup
Existing SQL
Component
Hekaton
Component
Generated
.dll
Client App
TDS Handler and Session Management
No improvements in
communication stack,
parameter passing,
result set generation
10-30x more efficient
Reduced log bandwidth
& contention. Log
latency remains
Checkpoints are
background sequential
IO
Hekaton
Compiler
Natively
Compiled SPs
and Schema
Hekaton Engine for
Memory_optimized Tables
& Indexes
Parser,
Catalog,
Algebrizer,
Optimizer
Key
Proc/Plan cache for ad-hoc
T-SQL and SPs
Interpreter for TSQL, query
plans, expressions
Access Methods
Query
Interop
Buffer Pool for Tables &
Indexes
SQL Server.exe
Memory-optimized
Table Filegroup
Transaction Log
Data Filegroup
Existing SQL
Component
Hekaton
Component
Generated
.dll
Operation
Factor faster (slower) than
regular SQL
Interop
Select
count(*)1
Comments
Native
No clustered index scan in
Hekaton
(2.5)
=
(1.3)
N/A
Uses index scan
Nested-loop Join1
4.0
N/A
Probes into hash index
Single-row selects1
1.3
40
SP doing selects in loop
Single-row selects1
1.2
17
Native compiled SP calls SQL’s
rand()
N/A
10
SP doing update in loop
Hash Join1
Single-row updates1
Bwin Session State
6
Version M4
(1) 1 million rows accessed in single query or SP
Interop targets app
migration, not perf
Advantage of
pushing work to SPs
Expectation for
OLTP workloads
Same SQL Server Optimiser
•
Storage
ALTER DATABASE ContosoOLTP ADD FILEGROUP [ContosoOLTP_hk_fs_fg] CONTAINS MEMORY_OPTIMIZED_DATA;
ALTER DATABASE ContosoOLTP
ADD FILE (NAME = [ContosoOLTP_fs_dir],
FILENAME = 'H:\MOUNTHEAD\DATA\CONTOSOOLTP_FS_DIR') to FILEGROUP [ContosoOLTP_hk_fs_fg];
•
Table
CREATE TABLE Customers (
CustomerID nchar (5) NOT NULL PRIMARY KEY NONCLUSTERED HASH WITH (BUCKET_COUNT=100000),
CompanyName nvarchar (40) NOT NULL INDEX IX_CompanyName HASH(CompanyName) WITH (BUCKET_COUNT=65536),
ContactName nvarchar (30) NOT NULL ,
ContactTitle nvarchar (30) NOT NULL ,
Address nvarchar (60) NOT NULL ,
City nvarchar (15) NOT NULL INDEX IX_City HASH(City) WITH (BUCKET_COUNT=1024),
Region nvarchar (15) NOT NULL INDEX IX_Region HASH(Region) WITH (BUCKET_COUNT=1024),
PostalCode nvarchar (10) NOT NULL INDEX IX_PostalCode HASH(PostalCode) WITH (BUCKET_COUNT=100000),
Country nvarchar (15) NOT NULL ,
Phone nvarchar (24) NOT NULL) WITH (MEMORY_OPTIMIZED=ON, , DURABILITY = SCHEMA_AND_DATA)
•
Native procedure
CREATE PROC InsertCustomers (@CustomerID nchar(5),@CompanyName nvarchar(40),
@ContactName nvarchar(30),@ContactTitle nvarchar(30), @Address nvarchar(60),
@City nvarchar(15),@Region nvarchar(15),@PostalCode nvarchar(10),
@Country nvarchar(15),@Phone nvarchar(24))
WITH NATIVE_COMPILATION, SCHEMABINDING, execute as owner as
BEGIN ATOMIC WITH (TRANSACTION ISOLATION LEVEL = SNAPSHOT, language = 'english')
INSERT INTO [dbo].[Customers] VALUES(@CustomerID,@CompanyName,@ContactName,@ContactTitle,@Address,
@City,@Region,@PostalCode,@Country,@Phone,@Fax);
END
CREATE TABLE DDL
Table code generated
Compiler invoked
Table DLL produced
Table DLL loaded
Timestamps
Hash index
on Name
J
S
Chain ptrs
200, ∞
Name
John
City
Garbage Collection
Removes Unused Rows
Susan
B
P
Beijing
100, 200
90,150
Hash index
on City
John
Paris
Jane
Prague
Bogota
50, ∞
Hekaton Memory
XID
RowID Name(PK) Airport
234 | 001 | George | LAX
237 | 001 | George | SEA
235 | 002 | Fred
| CHI
Transaction log
Add
Add
Del
234 | 001 | George | LAX
Add 237 | 001 | George | SEA
Checkpoint File
234 | 001 | George | LAX
235 | 002 | Fred
| CHI
237 | 001 | George | SEA
Delta File
Create
Delete
XID RowID XID
234 | 001 | 237
Interpreted T-SQL Access
Natively Compiled Stored Procedures
Access both memory- and disk-based tables
Less performant
Virtually full T-SQL surface area
Access only memory optimized tables
Maximum performance
Limited T-SQL surface area
When to use
When to use
Native Compiled
Stored Procedures
Non-Native
Compilation
High. Significantly less
No different than T-SQL
calls in SQL Server today
Migration Strategy
Application changes –
development overhead
Easier app migration as
can still access Memory
Optimized (MO) tables
Access to objects
Can only interact with
Memory Optimized
tables
All Objects. Access for
transactions across MO
and b-tree tables
Support for T-SQL
Constructs
Limited.
T-SQL surface area (limit
on MO interaction)
Optimization/Stats and
Query Plan
Statistics utilized at
CREATE -> Compile time
Statistics updates can be
utilized to modify plan at
runtime
Flexibility
Limited (e.g., no ALTER
procedure, compile-time
isolation level)
Ad-hoc query patterns
Drivers
Hekaton Tech Pillars
Customer
Benefits
Performance
Efficient, business-logic
processing
T-SQL Compiled to
Machine Code
• T-SQL compiled to machine
code via C code generator
and VC
• Invoking a procedure is just
a DLL entry-point
• Aggressive optimizations @
compile-time
Hardware trends
Stalling CPU clock rate
instructions to go
through
Time
Transaction T1 (SNAPSHOT)
1
BEGIN
Transaction T2 (SNAPSHOT)
2
BEGIN
3
UPDATE t SET c1=‘bla’ WHERE c2=123
4
UPDATE t SET c1=‘bla’ WHERE
c2=123 (write conflict)
First writer
wins
CREATE TABLE dbo.Orders( OrderID int NOT NULL PRIMARY KEY NONCLUSTERED HASH WITH
(BUCKET_COUNT=1000000),
CustomerID int NOT NULL INDEX IX_CustomerID HASH WITH (BUCKET_COUNT=1000000),
OrderDate datetime NOT NULL, OrderDescription nvarchar(1000) ) WITH (MEMORY_OPTIMIZED=ON)
• Assume the Orders table has 1M rows, and the average length of OrderDescription 78 characters.
• Index size:
• The bucket_count 1000000. This is rounded up to the nearest power of 2: 1048576.
• 8 * 1048576 + 8 * 1048576 = 16777216 bytes
• Table data size
• [row size] * [row count] = [row size] * 8379
• [row size] = [row header size] + [actual row body size]
• [row header size] = 24 + 8 * [number of indices] = 24 + 8 * 2 = 40 bytes
• [actual row body size]
• SUM([size of shallow types]) = 4 [int] + 4 [int] + 8 [datetime] = 16
• 2 + 2 * [number of deep type columns] = 2 + 2 * 1 = 4
•
•
NULL array = 1 + NULL array padding = 1
Size so far is 16 + 4 + 1 + 1 = 22. Padding to Nearest multiple of 8 is
• [actual row body size] = 24 + 2*78 = 180 bytes. So [row size] = 40 + 180 = 220 bytes
• [table size] = 16777216 + 220 * 1000000 = 236777216 bytes ~= 230Mb
Memory Optimized Data Filegroup
Range 500-
Range 400-499
Range 300-399
Range 200-299
A transaction with a commit
timestamp of 600 inserts one
new row and deletes rows
inserted by transactions with a
commit timestamp of 150, 250
and 420
Range 100-199
offline checkpoint Thread
Memory Optimized Data Filegroup
Range 500-599
Memory Optimized Data Filegroup
Key
Data file with rows generated
in timestamp range a-b
Range 400-499
Range 200-399
Range 300-399
Range 200-299
Range 100-199
Range 400-499
Range 300-399
Range 200-299
Range 100-199
Merge
200-399
Delta file with IDs of Deleted
Rows
Deleted Files
sys.dm_db_xtp_checkpoint
Returns database that has one or more IM objects
sys.dm_db_xtp_checkpoint_files
Displays information about checkpoint files
sys.dm_db_xtp_hash_index_stats
Useful for understanding and tuning the bucket counts and duplicates for index key
sys.dm_db_xtp_index_stats
Reports statistics about scans on an index
sys.dm_db_xtp_memory_consumers
Reports the database-level memory consumers in the IM database engine.
sys.dm_db_xtp_object_stats
Reports statistics about operations on a memory optimized object.
sys.dm_db_xtp_table_memory_stats
Returns memory usage statistics for each IM table (user and system) in the current
database.
sys.dm_db_xtp_transactions
Reports the active transactions in the IM database engine.
sys.dm_xtp_consumer_memory_usage
Reports memory usage for all memory consumers including @database level and
@system level.
sys.dm_xtp_gc_stats
Reports information about the current behavior of the IM garbage-collection process.
sys.dm_xtp_system_memory_consumers
Reports information about memory usage.
sys.dm_xtp_threads
Reports the threads that the IM database engine has started internally.
sys.dm_xtp_transaction_stats
Reports statistics about transactions that have run since the server started.
New or updated property, system
view, stored procedures, or DMV
Change
OBJECTPROPERTYEX
New property: TableIsMemoryOptimized.
SERVERPROPERTY
New property: IsXTPSupported.
sys.data_spaces
The following columns display additional values: type and type_desc
sys.indexes
The following columns display additional values: type and type_desc.
sys.parameters
New column: is_nullable.
sys.all_sql_modules
New column: uses_native_compilation.
sys.sql_modules
New column: uses_native_compilation.
sys.table_types
New column: is_memory_optimized.
sys.tables
New columns: durability, durability_desc, and is_memory_optimized.
sys.hash_indexes
New: Shows the current hash indexes and the hash index properties
sp_xtp_merge_checkpoint_files
New stored procedure: Merges all data and delta files in the transaction range specified.
BEGIN
Establish System
Performance Baseline
Run workload
Is MDW Set up?
Run AMR Reports
Configure Management
Data Warehouse
Migrate
Configure Data
Collection
Run Workload and
collect performance
metrics
Compare to Baseline
and set as new baseline
COMPLETE
Space Used in GB
101 million row table (Table + index space)
20.0
15.0
10.0
5.0
0.0
Table with
Table with
customary indexing customary indexing
(page compression)
Table with no
indexing
Table with no
indexing (page
compression)
Table with
columnstore index
Clustered
columnstore
Partition
• CREATE CLUSTERED COLUMNSTORE
Column Store (CS)
• BULK INSERT
• INSERT
• DELETE
• UPDATE
Not intended for OLTP applications,
but great for read-mostly data
warehouses!
Row Store (RS)
Deleted
Bitmap
•
CREATE TABLE <table> ( … )
•
CREATE CLUSTERED COLUMNSTORE INDEX <name> on <table>
•
BULK INSERT, SELECT INTO <name> on <table>
•
INSERT/UPDATE
Converts entire table to CS format
Take care of memory needed and parallelism (MAXDOP 1)
Creates new CS row groups
Store in Row Store
Tuple Mover
When RS reaches 1M rows, convert to a CS row group
Runs every 5 minutes by default
Started explicitly by ALTER INDEX <name> ON <table> REORGANIZE
Partitioning works on clustered columnstores
Just like any other table
The motivation is manageability more than performance
• SQL Server 2012
• SQL Server 14
http://www.cio.com/article/734462/SQL_Server_2014_Hastens_Transaction_Processing
http://research.microsoft.com/en-us/news/features/hekaton-122012.aspx
http://research.microsoft.com/apps/pubs/default.aspx?id=193594
Les Conférences
Groupe des Utilisateurs SQL Server
GUSS.fr