DBA Fundamentals :: Down Under
SQL Server on Flash: Re-Thinking Best Practices
Jimmy May, MCM
SQL Server Solutions Architect | Fusion ioMemory™ Solutions
c
1
Preamble
2
Legalese
During our meeting today, we may make forward-looking statements.
Any statement that refers to expectations, projections or other characterizations of future events or circumstances is a
forward-looking statement, including those relating to market position, market growth, product sales, industry trends,
supply chain, future memory technology, production capacity, production costs, technology transitions, construction
schedules, production starts, and future products. This presentation contains information from third parties, which
reflect their projections as of the date of issuance. Actual results may differ materially from those expressed in these
forward-looking statements due to factors detailed under the caption “Risk Factors” and elsewhere in the documents
we file from time to time with the SEC, including our annual and quarterly reports. We undertake no obligation to
update these forward-looking statements, which speak only as of the date hereof.
©2014 SanDisk Corporation. All rights reserved. SanDisk is a trademark of SanDisk Corporation. Lightning, Lightning Eco, Lightning Ascend, Lightning Ultra, Optimus, Optimus Eco, Optimus Ascend,
Optimus Ultra, Optimus Extreme, CloudSpeed, CloudSpeed Eco, CloudSpeed Ascend, CloudSpeed Ultra, CloudSpeed Extreme, FlashSoft, ZetaScale, Guardian Technology, FlashGuard, DataGuard,
EverGuard, SanDisk ION Accelerator, Fusion ioMemory, Fusion ioSphere, Fusion ioTurbine, Fusion ioVDI, and others are trademarks of SanDisk Enterprise IP LLC. Other brand names mentioned herein are
for identification purposes only and may be the trademarks of their respective holder(s).
3
Introduction


Flash storage is a game-changer for virtually every app in which it
is used. Whether used locally—SATA, SAS, or PCIe; or for tiered
storage or all-flash arrays—flash is changing what we thought we
knew about SQL Server performance.
This presentation provides a brief survey of history and a highlevel, vendor-agnostic review of options for implementing flash in
the enterprise.
4
Bio
Jimmy May, MCM
• SanDisk SQL Server Solutions Architect
Formerly:
• Sr. Program Manager, SQL CAT
• SQL Server Customer Advisory Team
• MSIT Principal Architect: Database
Microsoft Certified Master: SQL Server (2009)
MS IT Gold Star Recipient (2008)
Microsoft Oracle Center of Excellence (2008)
SQL Server MVP Nominee (2006)
Indiana Windows User Group www.iwug.net
• Founder & Board of Directors
Indianapolis Professional Association for SQL Server www.indypass.org
• Founder & Member of Executive Committee
SQL Server Pros: Founder & Visionary-in-Chief
SQL Innovator Award Recipient (2006)
• Contest sponsored in part by Microsoft
Formerly Chief Database Architect for high-throughput OLTP VLDB at
ExactTarget (recent IPO)
Senior Database Administrator for OpenGlobe/Escient
www.twitter.com/aspiringgeek jimmy.may@sandisk.com
http://sqlblog.com/blogs/jimmy_may
jimmymay@outlook.com
5
Data Propulsion Laboratory — DPL
6
C:\>sqlservr.exe /faster
7
Acknowledgements







Brian Walters, SanDisk
Niall MacLeod, SanDisk
John Gatch, SanDisk
Peter Plamondon, SanDisk
David Reeves, SanDisk
Grant Jacobson, SanDisk
Sean Hehir, SanDisk











Kevin Kline, MVP
Brent Ozar, MCM, MVP
Thomas LaRock, MCM, MVP
Denny Cherry, MCM, MVP
Gavin Payne, MCA, MCM
Chuck Suter, Pure Storage
Mike Anderson, Disney
Mike Ruthruff, Bungie Studios
Thomas Kejser
Tim Radney, MVP
Mindy Curnutt, MVP
8
Navigating this deck
1.
On the left, right-click on any section name
2.
Click “Collapse All”)
3.
Review the section names, then click to expand desired content
9
PASS Summit 2015 Pre-Con
The Complete Primer to SQL Server Virtualization
Learn More:
http://www.sqlpass.org/summit/2015/Sessions/Details.aspx?sid=8149
10
SanDisk Speaking Sessions

Title: The Complete Primer to SQL Server Virtualization (Full Day Session)
–
–
–
–
–

Title: SQL Server 2016 on Flash
–
–
–
–

Speaker: Jimmy May, MCM, SQL Server Solutions Architect, SanDisk
Date: Wednesday, October 28
Time: 4:45 p.m. – 6:00 p.m. PST
Location: 6A
Learn More: www.sqlpass.org/summit/2015/Sessions/Details.aspx?sid=8623
Title: SQL Server on Flash: Re-Thinking Best Practices [DBA-210]
–
–
–
–
–

Speaker: Brian Walters, Solution Architect, SanDisk
Date: Wednesday, October 28
Time: 10:15 a.m. PDT
Location: TCC101
Title: Columnstore Indexes in SQL Server: Flipping the DW Faster Bit [AD-206]
–
–
–
–
–

Speakers:
• Jimmy May, MCM, SQL Server Solutions Architect, SanDisk
• David Klee, MVP, Founder & Chief, Heraflux Technologies
• Argenis Fernandez, MVP, MCM, SQL Server Solutions Architect, Pure Storage
Date: Monday, October 26
Time: 9 a.m. – 6:00 p.m. PST
Location: 611 – 614
Learn More: www.sqlpass.org/summit/2015/Sessions/Details.aspx?sid=8149 [DBA-299-P]
Speaker: Jimmy May, MCM, SQL Server Solutions Architect, SanDisk and Brian Walters, Solution Architect, SanDisk
Date: Friday, October 30
Time: 8:00 – 9:15 a.m. PST
Location: TCC 304
Learn More: www.sqlpass.org/summit/2015/Sessions/Details.aspx?sid=8601
SanDisk Theater
–
Visit the SanDisk booth (#22) for brief sessions—throughout the day, throughout the week—for an opportunity to learn & to win!
11
Agenda
12
Agenda
History and Introduction to Enterprise Flash
 Flash Best Practices for SQL Server
 SQL Server Use Cases and Implementation Details
 Flash Myths, Misinformation,
& Re-visiting Best Practices
 PCIe Application Accelerators
 Microsoft SQL Server 2014 Fast Track (DWFTRA)
 Hardware for Nothing, Get Your Flash for Free

13
Enterprise Flash Unlocks
Microsoft SQL Server 2014 Business Value
•
Increase business productivity – more transactions, faster
•
Improve customer experience – faster response times
•
Reduce costs – scale more users and workloads per server
•
Increase SLAs – faster SQL database startup, failover, & maintenance
14
Flash: Don’t be this guy. Don’t be in this org.
Hat tip to Bob Pusateri (@SQLBob)
via Jason Horner (@jasonhorner) & Argenis Fernandez (@DBArgenis )
15
Introduction to Enterprise Flash
16
Flash is Revolutionizing Enterprise Storage
17
May’s I/O Mantra
X capacity (GB)
at Y throughput (MB/sec)
at Z IOPs (transfers/sec)
at <=10ms latency for OLTP data files
at <=30ms latency for DW data files
at 0ms-2ms for log files
19
What is I/O? Performance Monitor: Logical Disk and PhysicalDisk

Latency
– Measured in milliseconds (ms)
– Measured in microseconds (µs)
– Performance Monitor: Logical Disk
• Avg. disk sec/transfer
• Avg. disk sec/read
• Avg. disk sec/write

IOPS
• Disk transfers/sec
• Disk read/sec
• Disk writes/sec

Throughput
• Disk bytes/sec
• Disk read bytes/sec
• Disk write bytes/sec

Capacity
– Measured in GB/TB
– The easy one!
20
What is the business impact of flash on I/O?
Google found an extra half-second in search-page generation time
dropped traffic by 20%.
Amazon found every 100ms of latency cost them 1% in sales.
If a trading platform is 5 milliseconds behind the competition, it
could lose millions in revenues per millisecond.
Performance is paramount: Asking users to wait is like asking them to leave.
If it is fast and ugly, they will use it and curse you; if it is slow, they will not use it.
—Computer science professor, entrepreneur, & billionaire David Cheriton
Application performance is not an event or something you do once—it's a practice.
—Mark Kremer, CEO, Precise Software Solutions Inc.
21
Spinning Media vs. PCIe
How many conventional hard drives to match the IOPs of a single PCIe flash device?
22
23
Spinning Media vs. PCIe: Some Math

Note: We could do lots of calculations with various types of I/O
–
–
–

Random per 4K Read IOPs
–
–

1,906 * 1” height/HDD * 1 foot/12” * 1 story/10 foot = 15.8 stories
Weight
–

340,000 IOPs/PCIe device ÷ 181 IOPs/15K HDD = 1,906 HDDs/PCIe device
Height
–

600GB 15K HDD: 181
3.2TB PCIe Flash: 345,000
Devices for same Random Read IOPs
–

4KB IOPs are one of the most commonly published
Numbers will vary for I/O characteristic of SQL Server
The ratios are nonetheless equally remarkable
1,906 HDDs/PCIe * 1.51 pound/HDD = 2,878 pounds
Sources
–
–
–
https://www.sandisk.com/business/datacenter/products/flash-devices/pcie-flash/sx350
http://www.seagate.com/files/docs/pdf/datasheet/disc/cheetah-15k.7-ds1677.3-1007us.pdf
http://archive.benchmarkreviews.com/index.php?option=com_content&task=view&id=375&Itemid=60&limitstart=5
24
Relative Performance of Conventional HDD vs. Flash




There are many ways to look at disk performance:
–
Read/Write
–
Random/Sequential
–
IOPs/Throughput
–
Block size impacts IOPs, throughput, & latency
–
Let’s look briefly at one perspective: IOPs vs. latency
Spinning Media vs. Flash
180x – 1,000x+ IOPs
Spinning media (15K Enterprise SAS)
–
Random 4K reads: 210 IOPs @2000 µs
–
Random 4K writes: 190 IOPs @2000 µs
30x – 70x faster
Flash vendor 1:
–
Random 4K reads: 75,000 IOPs @50µs
–
Random 4K writes: 36,000 IOPs @65µs
Flash vendor 2:
–
Random 4K reads: 252,000 IOPs @29µs
–
Random 4K writes: 236,000 IOPs @29µs
25
Flash Best Practices for
SQL Server
38
Flash Best Practices for SQL Server

Though flash can seem like a magic bullet …
–
Flash can mask many performance, configuration, and design challenges
Best practices remain fundamental
 There is no substitute for engineering discipline in terms of
application & database design

If you don't have time to do it right, when will you have time to do it
over?
—John Wooden
39
Workload Characteristics and Pain Points
OLTP
Data Warehouse
Small block, random I/O
 High IOPS
 Low throughput
 Low Latency


Large block, sequential I/O
 Lower IOPS
 High throughput
 High Latency
40
Flash Addresses Pain Points
Pain Points


Overprovisioned Shared
Storage – random I/O
Introduces I/O Bottleneck:
Flash Liberation

Solid State minimizes seek time
–
Ultra Low latency (µs)
–
Small & Large Block Size
–
High Latency
–
High Throughput (29 GB/s @ 2 ms)
–
Low IOPS
–
Low cost / High return
–
Low CPU utilization
–
High Cost / Depressed Return
41
Flash Best Practices: SQL Server Configuration

tempdb Configuration
–
–
–
–
–

-T1117 (to enable consistent AutoGrow)
-T1118 (to eliminate SGAM contention)
Multiple, equisized tempdb data files
Note: These are baked into SQL Server 2016
Consider making these mandatory components for all SQL Server deployments
Local Security Policy
secpol.msc > Local Policies > User Rights Assignment
– Add SQL Server service account to the following:
–
• Perform volume maintenance tasks (to enable Instant File Initialization).
• Lock pages in memory (to prevent paging).
42
Best Practice – SQL Server
Dispelling some myths...

Mitigate memory latch contention:
• User DBs: Multiple equally-sized data files per data file filegroup
• Tempdb: Multiple equally-sized data files
• 8 files is the sweet spot for most implementations

Consider segregating log files from user and tempdb data files if
demanded by performance

Configure 'optimize for ad hoc workloads' – mitigate single use
statements from overly consuming plan cache & from
monopolizing excessive CPU
43
Flash Best Practices: Windows Power Configuration

Power options
–
Windows power options:
Control Panel > Hardware > Power Options >
High Performance
PoSh: Powercfg -SETACTIVE SCHEME_MIN
44
Flash Best Practices: BIOS

Logical processor / hyper-threading: Enable
–
Contemporary hyper-threading vs. the original incarnation

Fan: high or increased cooling

C-States: Disable all “power saving” options
–
Configure for maximum clock speed & voltage
–
Note: Sometimes called P-States
45
Flash Best Practices: Storage Devices



Logical volume configuration
–
Disk Partition Alignment remains important
–
Windows-level format (FAUS): 64KB
Drivers:
–
Update drivers
–
More important for flash devices than for conventional spinning media
Vendor-specific recommendations:
–
Default power settings for device may not be optimal
• This is in addition to physical server & Windows power settings
–
“Low-level” formatting: 4KB (not 512B) recommended
46
Power Optimization Summary

Maximize power settings in three different locations:
–
BIOS
• Disable C-State settings
• Maximize fan or cooling, if available
–
Windows: High Performance
–
PCIe configuration
47
Flash Best Practices: Bandwidth


Maximum performance is limited by system bottlenecks
2.5” flash drives in server front-end
–
–
It doesn’t take much flash to overwhelm last gen buses
Contemporary buses can accommodate 20 – 24 disks
• Up to 100TB!
• Lenovo X3650 w/Optimus Max 100TB storage (100sec)
www.twitter.com/aspiringgeek/status/615909659916898304
–

Be aware of potential thermal issues
Validate PCIe slots best-suited for cards
–
–
–
PCIe version
Bus speed: 4x, 8x, 16x
Bus power
48
Best Practice – SQL Server Backup

Design a User & System Database Backup strategy compliant
with SLOs & SLAs

Backup Types: FULL, DIFFERENTIAL, LOG, COPY_ONLY

Test & Validate:
–
Full, Diff, Log—including point-in-time
–
Integrity: DBCC CHECKDB
You don’t have a backup until it’s been restored—& validated.
—Kimberly Tripp
49
Best Practice – SQL Server Maintenance

Fragmentation
–

SQL Server “read ahead” relies on logically contiguous pages
SORT_IN_TEMPDB Index Build Option
–
Significant reduction in I/O within SQL Server data files
–
Especially important for AlwaysOn
50
Backup & Restore Performance Options

Backup
–
–
–
Backup File Count (multiple streams)
COMPRESSION
MAXTRANSFERSIZE
• 4,194,304 (4MB) is max permitted block size
–
BUFFERCOUNT
• 2x to 4x logical core count

Restore
–
–

Presumably multiple files & compression were implemented during backup
Leverage MAXTRANSFERSIZE & BUFFERCOUNT
Note: Consider enabling these options via sp_configure
–
–
backup checksum default
backup compression default
51
Prototypical T-SQL BACKUP Syntax
DECLARE @bkpname sysname = 'MyFastBackup‘
-- NAME is exposed via RESTORE HEADERONLY FROM DISK = 'path to first bkp file‘
-- & also in SELECT * FROM msdb.dbo.backupset
BACKUP DATABASE MyDB
TO DISK = '<local or \\UNC path>\DBbkps\MyDB_20150909_#1.bak'
, DISK = '<local or \\UNC path>\DBbkps\MyDB_20150909_#2.bak'
, DISK = '<local or \\UNC path>\DBbkps\MyDB_20150909_#3.bak'
, DISK = '<local or \\UNC path>\DBbkps\MyDB_20150909_#4.bkp'
WITH NAME = @bkpname
, STATS = 1
, COMPRESSION
, MAXTRANSFERSIZE = 4194304 -- 4MB is max permitted block size
, BUFFERCOUNT
= 96
-- 2x to 4x logical core count
, STOP_ON_ERROR
-- CHECKSUM error
52
Prototypical T-SQL RESTORE Syntax
DECLARE @bkpname sysname = 'MyFastBackup‘
-- NAME is exposed via RESTORE HEADERONLY FROM DISK = 'path to first bkp file‘
-- & also in SELECT * FROM msdb.dbo.backupset
RESTORE DATABASE MyDB
FROM DISK = '<local or \\UNC path>\DBbkps\MyDB_20150909_#1.bkp'
, DISK = '<local or \\UNC path>\DBbkps\MyDB_20150909_#2.bkp'
, DISK = '<local or \\UNC path>\DBbkps\MyDB_20150909_#3.bkp'
, DISK = '<local or \\UNC path>\DBbkps\MyDB_20150909_#4.bkp'
WITH REPLACE , RECOVERY , STATS = 1
, MAXTRANSFERSIZE = 4194304 --4MB is the max allowed
, BUFFERCOUNT
= 112
--up to 4xcorecount
, STOP_ON_ERROR
-- CHECKSUM error
, MOVE 'tpch_300_Primary_SystemData01' TO 'C:\mount\data\MyDB_Primary_SystemData01.mdf'
, MOVE 'tpch_300_Primary_SystemData02' TO 'C:\mount\data\MyDB_Primary_SystemData02.mdf'
, MOVE 'tpch_300_Primary_SystemData03' TO 'C:\mount\data\MyDB_Primary_SystemData03.mdf'
, MOVE 'tpch_300_Primary_SystemData04' TO 'C:\mount\data\MyDB_Primary_SystemData04.mdf'
, MOVE 'tpch_300_Secondary01'
TO 'C:\mount\data\MyDB_Secondary01.ndf‘
, ...
, MOVE 'tpch_300_Log'
TO 'C:\mount\data\MyDB_Log.ldf'
53
Automated Backup Tuning

What’re the best options for your environment?

Sir SQL—our knight in shining armor—with a PoSh-based solution!
Nic Cain @SirSQL
Automated Backup Tuning
https://bit.ly/SirSQLBackupTuning
54
Data Compression

Long a best practice for SQL Server data

Yet it’s often not implemented

Typically, relatively small CPU cost (3%) in exchange for storage
savings of ~2x – 5x

Corollary: If extant data is not compressed, compression while
migrating to flash can effectively discount your new storage costs
by half or more.

Note: Some flash vendors provide compression or de-dupe by
default.
55
Comparing Space Savings
19.7GB
10.9GB
6.9GB
5.0GB
TABLE WITH
CUSTOMARY
INDEXING
4.0GB
TABLE WITH
TABLE WITH NO TABLE WITH NO
TABLE WITH
CUSTOMARY
INDEXING
INDEXING (PAGE COLUMNSTORE
INDEXING (PAGE
COMPRESSION)
INDEX
COMPRESSION)
1.8GB
CLUSTERED
COLUMNSTORE
556
100
9,000
90
8,000
80
7,000
70
6,000
60
5,000
50
4,000
40
3,000
30
2,000
20
1,000
10
0
0
RS Scan Rate
Physical
(MB/sec)
CS Scan Rate
Physical
(MB/sec)
Max User
Capacity
(TB)
Query Thruput
4000
100
3500
90
80
3000
(Queries/Hr/TB
10,000
TB
MB/sec
I/O Thruput
70
2500
60
2000
50
1500
40
TB
Row Store vs. Columnstore:
I/O vs. Query Throughput
30
1000
20
500
10
0
0
RS Measured
Thruput
(Queries/Hr/TB)
CS Measured
Thruput
(Queries/Hr/TB)
Max User
Capacity
(TB)
Copyright © 2015 SanDisk. All rights reserved.
57
Best Practice – Data Warehouses

New deployments should consider Data Warehouse Fast Track
Reference Architectures (DWFTRA)

Columnstore Indexes
–
Best, underutilized feature in SQL Server 2014
–
Microsoft recommends that columnstore be the default choice for
warehouse tables
–
Leverage table partitioning
58
Stay Tuned for Guidance

-E & -T834
These options direct SQL Server to use Windows large-page memory
allocations for the buffer pool & large physical page allocations,
respectively.
– We are investigating interesting memory and I/O behaviors when
these traces flags are enabled with certain SQL Server features
–
Indirect Checkpoint
 Delayed Durability
 TDE (& Always Encrypted)

59
Flash Myths
60
Flash Myths, Misinformation, & Re-visiting
Best Practices

Log files & data files

Fragmentation

OLTP vs. OLAP/DW

Cost

Durability/Reliability

Availability & RAID for PCIe Cards

CPU0 bottlenecks
61
Fragmentation

Concept of “sequential” access has no physical analog in flash.

Defragmentation remains a nominal best practice, especially for
OLTP/DW workloads
–

SQL Server readahead requires logically contiguous data
Yet even within Microsoft, benchmarks on flash are often run
without the benefit of defragging(!)
62
Log Files & Data files

Once upon a time...
–
Physical segregation of indexes from data files was a best
practice
• Storage abstraction rendered this largely unnecessary
–
We still often needed to segregate log files from data files
–
However, this may no longer be necessary—test!
• Bottom line: Look for consistent, low (<1ms) log file latency
63
OLTP vs. OLAP/DW

Once upon a time...
–
We didn’t intermingle workloads
–
Today? Flash may render this moot
–
Depends on server resources & application workload
64
Availability & RAID for SAS, SATA, & PCIe

Not all flash is created equal!
–

Consumer vs. Enterprise
Contemporary Enterprise Flash is robust
–
–
Massively over-provisioned, internal error checking, etc.
Example warranty: 50PBW (petabytes written), i.e., 50 * 1024TB of writes
• 512GB each day for 6 years
• Doable, yet...

Is it necessary to RAID enterprise flash?
–
–
–
It depends
Be clear: I am not advocating an SPOF!
Perhaps a better use of resources to focus on server/system HA
65
CPU0 bottlenecks



Not a myth, but antiquated
Windows Software RAID leverages CPU0
Some flash drivers used to do so as well
–
–
–

For example: 16 core server, CPU0 at 99%, CPU1-15 at 10%
–

Result, high I/O could result in one core with little headroom
Even though other cores were relatively unused
Must examine core-by-core % Processor Time
Average CPU <16%
Contemporary flash drivers render this moot because they either:
–
–
On-board device processor, or
Balance CPU across all cores
66
Cost

Once upon a time...
–
Flash was expensive
–
If you think flash is expensive, try pricing a SAN
• And supporting it...
–
Yet time is money
• Remember the latency slide?
–
Today:
• Glacial SATA is ~50¢/GB
• Depending on format, vendor, re-seller, etc.,
Enterprise Flash has recently been as economical as $3/GB & quickly approaching $1/GB
• As of March 2015: InfiniFlash is as low as $2/GB!
• SAS & SATA SSD offerings as low as 50¢ - $1/GB
67
Raw Cost/Capacity vs. Access Density


Raw Cost/Capacity
–
$/TB is as simplistic as classifying storage simply in terms of capacity
–
Remember the I/O Mantra: latency, IOPs, thruput, capacity
Consider: Access Density
AD = work ÷ capacity ÷ time
• e.g., IOs (work) / GB (unit of storage) / second (unit of time)

Related:
–
Warehouse workloads rated in terms of Queries/Hr/TB
68
TCO & TCA: Flash is Fast—& Less Expensive than HDDs
15K SAS Enterprise HDD vs. 4TB Optimus Max
The Accelerating Economics of Flash and the Retreat of Hard Disk Drives
http://itblog.sandisk.com/the-accelerating-economics-of-flash-and-the-retreat-of-hard-disk-drives
69
SQL Server Use Cases
70
SQL Server Use Cases:

Single-Instance Options

tempdb as a Cluster Shared Resource

AlwaysOn HA using Readable Replicas

AlwaysOn Multi-site HA with DR

Windows Scale Out File Server (SOFS)

Compression

Fragmentation

Flash for Development Environments
71
SQL Server 2014 Architectural Configurations
Entire Database on
In-Server Flash
Parts of DB on
Virtualized Database Servers for
Data Warehouse Fast Track
In-Server Flash
Consolidation, Upgrade, Mixed
Reference Architectures (FTDWRA)
(e.g., “hot files” or tempdb)
Workloads
FlashSoft®
AlwaysOn Availability Groups
Cache data read from SAN
on In-Server Flash
ION Appliance
as Shared Storage for FCI
72
AlwaysOn Multi-site HA with DR
Disaster Recovery DC
Primary DC
Primary (server 1)
Asynchronous
Synchronous
Remote Replica
Availability Group
Local Replica
76
Windows Scale Out File Server (SOFS)

Software-defined Storage
https://en.wikipedia.org/wiki/Software-defined_storage
–
Software-defined storage (SDS) is evolving concept for computer data storage software to manage policy-based provisioning and management of data
storage independent of hardware. Software-defined storage definitions typically include a form of storage virtualization to separate the storage
hardware from the software that manages the storage infrastructure. The software enabling a software-defined storage environment may also provide
policy management for feature options such as deduplication, replication, thin provisioning, snapshots and backup.
–
By consensus and early advocacy, SDS software is separate from the hardware it is managing.

Windows implementation of a virtual SAN (VSAN) aka SANless storage

JBODs on steroids

–
Storage Spaces
–
Storage Spaces Direct
Features
–
Redundancy
–
Performance
–
Amazingly cheap—er, ah inexpensive

Introduced in Windows Server 2012, significant enhancements in Windows Server 2016

Be aware—pay attention—we’ll be hearing more-&-more about Software-defined Storage & SOFS

Windows Server 2012 Scale-Out File Server for SQL Server 2012 - Step-by-Step Installation by Jose Barreto
77
Windows Scale Out File Server (SOFS)
78
SanDisk: Test Scenario
Enhanced AlwaysOn
Workload scenario
 High volume transactional workload
 I/O
 Small block reads & writes (8K – 16K)
 Random reads & writes




Continuous writes to transaction log
Synchronous availability mode
High volume log transport across the wire
Likewise high volume for redo on secondary
79
79
SanDisk: Hardware Configuration
Enhanced AlwaysOn
Systems Under Test (SUT)

2 Lenovo ThinkServer RD650 Servers each
with:

SanDisk Fusion ioMemory


6.4TB

3.2TB

256GB DRAM
16 cores/32 LP
Windows Server 2012 R2
10Gbe NIC
LI
ID
RD650
Load Drivers
2 socket, 56 cores ~2.6Ghz Haswell-EP
Network






Processors


Load Injector (LI)
SUTs
2x Mellanox ConnectX-3 Ethernet Adapter
56Gbs (7MB/s)
BIOS set to high performance

Fan offset increased

C-States disabled
ID
ID
RD650
RD650
Workload: OLTP Hekaton
HA: AlwaysOn
Virtualization disabled
80
80
SanDisk: OS & SQL Server Configuration
Windows Server 2012 R2 Datacenter
Windows High Performance
(Powercfg -SETACTIVE SCHEME_MIN)
 secpol.msc



SQL Server 2016 CTP2.2
+
SQL Server 2014 SP1

Lock pages in memory
Perform volume maintenance tasks
Server configuration options:



AlwaysOn configuration



Availability mode:
SYNCHRONOUS_COMMIT
Failover mode: AUTOMATIC
Workload: TPCC Hekaton, assigned to a
Resource Governor resource pool:

81
max server memory: 128000
max worker threads: 2016 (default: 1344)
CREATE RESOURCE POOL Pool_Hekaton
WITH (MIN_MEMORY_PERCENT = 50,
MAX_MEMORY_PERCENT = 90);
81
AlwaysOn Testing: SQL Server 2014 vs. 2016
The engine & the hardware are liberated to do what they’re
supposed to do
Processor: ~4x CPU
More transactions: >5x XTP transactions
More flash: ~3x throughput; >4x IOPs; 0ms – 1ms latency
Source of Improvement
AlwaysOn Transport: >4x Log Bytes Received
82
AlwaysOn Log Synchronization
83
Application Performance
500
467
450
~4x CPU Utilisation
>5x XTP Transactions
>4x Log Bytes Received
400
350
300
250
200
150
120
117
110
109
100
52
50
12
22
0
CPU (%)
XTP Transactions/sec Log Bytes Received
(x1000)
(MB/s)SP1
SQL2014
Redone Bytes/sec
(MB/s)
84
SanDisk Flash Performance
1,000.000
873
900.000
~3x IOPs
800.000
>4x Throughput
600.000
0ms – 1ms Latency
700.000
500.000
400.000
300.000
200
200.000
100.000
18
53
0
0.000
Disk: IOPs (x1000)
Disk: Thruput (MB/s)
SQL2014 SP1
1
Disk: Latency (ms)
SQL2016 CTP2.2
85
Performance Pattern
Correlation:
Processor
AlwaysOn Transport
XTP Transactions
Flash
86
Flash for Development Environments

Consider using flash for dev environments

I’ve always been a fan of providing developers with a robust
environment

Providing a flash platform (& plenty of all the right resources)
encourages experimentation & innovation
87
Microsoft SQL Server 2014 Fast Track
(DWFTRA)
88
Data Warehouse Fast Track Reference Architectures
(DWFTRA)

Perfectly balanced hardware solutions (CPU, memory, storage)

Official Microsoft Certification issued for each RA

OEMs often provide BoMs for the solutions, significantly
simplifying ordering (e.g., Lenovo’s doc_id)

Integral for our upcoming SQL Server 2005 EOS consolidation story

Numerous reference architectures published publicly
89
DWFTRA
OEM
Server
(cont.)
Rack Protocol DB Measured Microsoft RA
Units
Size Thruput Certification
Date
OEM
Documentation
9/3/2014
9/3/2014
9/3/2014
10/15/2014
3/10/2015
8/11/2014
5/5/2015
7/14/2015
link
link
link
link
link
link
link
link
(TB) (Queries/Hr/TB)
HP
HP
HP
Lenovo
Lenovo
Lenovo
Lenovo
Lenovo
DL380 G8
DL380 G8
DL580 G8
x3650 M5
x3850 X6
x3850 X6
x3850 X6
x3650 M5
2
2
4
2
4
4
4
2
PCIe
PCIe
PCIe
PCIe
PCIe
PCIe
PCIe
SAS
28
45
90
20
55
60
95
37
202
198
366
265
360
371
433
192
#2014-003*
#2014-003*
#2014-002
#2014-009
#2014-023
#2014-001
#2014-028
#2014-032
*#2014-003 reflect two certifications issued on the same day & available in the same document.
90
DWFTRA

Internal resources
–
–

DPL: Sprocket >> MSSQL2014FTRA
EPS: Sprocket >> SQL Server > Reference Architectures and Tech Guides
Pros
–
–
–
–

(cont.)
These RAs are elegant solutions & the DPL strongly endorses them
DWFTRA are inexpensive & compact compared to spinning media
The Fast Track program is mature—Microsoft is currently on v.3, work on v.4 underway
The DPL collaborates closely with the Microsoft Fast Track team
Cons
–
–
Customer awareness is low
Microsoft incents sales for APS (formerly PDW) & Azure, so DWFTRA has a low profile
91
HP 28TB Data Warehouse Fast Track Certification
Breakthrough Performance
- 5.5GB/s throughput
- 1414 queries / Hr / TB
- 98% average CPU utilization
SanDisk-based 2.6TB HP Workload Accelerators
DWFT Certification
#2014-003
HP ProLiant DL380 Gen8 with HP PCIe LE Workload
Accelerator 28TB reference architecture for Microsoft SQL
Server 2014 Data Warehouse Fast Track
DWFT Rev. 5.4
System Provider
Report Date
9/3/2014
System Name
Processor Type
Memory
HP DL380 G8
Intel Xeon E5-2697v2
2.7 GHz (12/24/48)
768 GB
Operating System
SQL Server Edition
Windows Server 2012 R2
SQL Server 2014 Enterprise Edition
Storage Provider
Storage Information
4 x 2.6TB HP PCIe LE Workload Accelerator for data and tempdb
2 x 300GB HDD for OS (RAID 1)
8 x 15K 300GB HDD for log (RAID 10)
Primary Metrics
Rated User Data Capacity1
(TB)
Row Store Relative
Throughput2
28
170
Column Store Relative
Throughput3
Maximum User Data Capacity
(TB)
218
32
Row Store
Relative
Throughput2
Measured
Throughput
(Queries/Hr/TB)
Measured Scan
Rate Physical
(MB/Sec)
Measured Scan
Rate Logical
(MB/Sec)
Measured I/O
Throughput
(MB/Sec)
Measured CPU
(Avg.)
(%)
170
202
4,508
5,500
5,004
96
Column Store
Relative
Throughput2
Measured
Throughput
(Queries/Hr/TB)
Measured Scan
Rate Physical
(MB/Sec)
Measured Scan
Rate Logical
(MB/Sec)
Measured I/O
Throughput
(MB/Sec)
Measured CPU
(Avg.)
(%)
218
1,414
1,389
N/A
N/A
98
HP DL380 Server
92
Lenovo 60TB SQL 2014 Data Warehouse - Ref Architecture
SanDisk-based 2400GB Lenovo io3 Flash Adapters
System x 3850 X6 Server
Breakthrough Performance – 11.4GB/s scan rate
93
HP 45TB Data Warehouse Fast Track Certification
Breakthrough Performance
- 5.2GB/s throughput
- 1476 queries / Hr / TB
- 45TB compressed data in 2U
DWFT Certification
#2014-003
HP ProLiant DL380 Gen8 with HP PCIe LE Workload
Accelerator 45TB reference architecture for Microsoft SQL
Server 2014 Data Warehouse Fast Track
DWFT Rev. 5.4
System Provider
System Name
Processor Type
Memory
HP DL380 G8
Intel Xeon E5-2697v2
2.7 GHz (12/24/48)
768 GB
Operating System
SanDisk-based 5.2TB HP Workload Accelerators
Report Date
9/3/2014
SQL Server Edition
Windows Server 2012 R2
SQL Server 2014 Enterprise Edition
Storage Provider
Storage Information
4 x 5.2TB HP PCIe LE Workload Accelerator for data and tempdb
2 x 300GB HDD for OS (RAID 1)
8 x 15K 300GB HDD for log (RAID 10)
Primary Metrics
Rated User Data Capacity1
(TB)
Row Store Relative
Throughput2
45
161
Column Store Relative
Throughput3
Maximum User Data Capacity
(TB)
227
72
Row Store
Relative
Throughput2
Measured
Throughput
(Queries/Hr/TB)
Measured Scan
Rate Physical
(MB/Sec)
Measured Scan
Rate Logical
(MB/Sec)
Measured I/O
Throughput
(MB/Sec)
Measured CPU
(Avg.)
(%)
161
198
4,220
5,240
4,730
97
Column Store
HP DL380 Server
Relative
Throughput2
Measured
Throughput
(Queries/Hr/TB)
Measured Scan
Rate Physical
(MB/Sec)
Measured Scan
Rate Logical
(MB/Sec)
Measured I/O
Throughput
(MB/Sec)
Measured CPU
(Avg.)
(%)
227
1,476
1,443
N/A
N/A
99
94
HP 90TB Data Warehouse Fast Track Certification
Breakthrough Performance
- 10.1GB/s throughput
- 2721 queries / Hr / TB
- 90TB compressed data in 4U
DWFT Certification
#2014-002
HP ProLiant DL580 Gen8 with HP PCIe LE Workload
Accelerator 90TB reference architecture for Microsoft SQL
Server 2014 Data Warehouse Fast Track
DWFT Rev. 5.4
System Provider
SanDisk-based 5.2TB HP Workload Accelerators
Report Date
9/3/2014
System Name
Processor Type
Memory
HP DL580 G8
Intel Xeon E7-4890 v2
2.8 GHz (4/60/120)
1536 GB
Operating System
SQL Server Edition
Windows Server 2012 R2
SQL Server 2014 Enterprise Edition
Storage Provider
Storage Information
6 x 5.2TB HP Workload Accelerator PCIe Flash devices for data and tempdb
2 x 500GB 7200RPM HDD for OS (RAID 1)
2 x 1.3TB HP Workload Accelerator PCIe Flash devices for log (RAID 1)
Primary Metrics
Rated User Data Capacity1
(TB)
Row Store Relative
Throughput2
90
314
Column Store Relative
Throughput3
Maximum User Data Capacity
(TB)
419
108
Row Store
Relative
Throughput2
Measured
Throughput
(Queries/Hr/TB)
Measured Scan
Rate Physical
(MB/Sec)
Measured Scan
Rate Logical
(MB/Sec)
Measured I/O
Throughput
(MB/Sec)
Measured CPU
(Avg.)
(%)
314
366
8,351
10,139
9,245
92
Column Store
Relative
Throughput2
Measured
Throughput
(Queries/Hr/TB)
Measured Scan
Rate Physical
(MB/Sec)
Measured Scan
Rate Logical
(MB/Sec)
Measured I/O
Throughput
(MB/Sec)
Measured CPU
(Avg.)
(%)
419
2,721
3,392
N/A
N/A
96
HP DL580 Server
95
Hardware for Nothing,
Get Your Flash for Free
96
SQL Server Licensing Labyrinth
97
Licensing Variables









Cores & Sockets
SQL Server Edition
Virtualized vs. Bare Metal
Host vs. Guest
Software Assurance (SA) or not
Active or Passive
Legacy Licensing
Is your sales rep under quota & is it near end-of-quarter or better yet near end-offiscal year?
Which highly-trained professional sales rep you talk to
–
Ask 2 people, get 3 opinions
98
Weasel Clause

Licensing is complex

Directions on Microsoft has a multi-day boot camp costing
thousands of dollars dedicated to Microsoft licensing
–
Windows & Office portion: 4 – 6 hours
–
SQL licensing: several more hours

Failure to respect the complexities risks leaving out so many
important details that we’ll be “wrong” in some aspects

Yet the case studies which follow are eminently defensible
99
Data Center Economics
Hardware for Nothing, Get Your Flash for Free
SanDisk Flash isn’t transformative merely because of performance
- the economics are disruptive
5 SQL Server Licensing/Hardware scenarios
–
#1. Using Flash to eliminate AlwaysOn Read-only Secondaries & their
licenses
–
#2. The Simple Upgrade: Spare compute & budget
–
#3. Application-level: Scaling up to big money
–
#4. Data Center Scale: Literally saving millions
–
#5. ...
100
#1. AlwaysOn Readable Secondaries
3 Sustained Simultaneous Workstreams Scenario
⁻
⁻
⁻
⁻
⁻
2 HP DL580 4-socket servers, 64 logical cores per server
Highly transactional TPCC-like environment – 1,000,000 New Orders per Minute
4 AlwaysOn Ags in synchronous availability mode, no queuing
Database backups at 4GB/s
I/O Latency: < 0.0ms, i.e., µs range
Recommended Configuration
⁻
⁻
Readable Secondary used for backup stream
This requires licensing all cores on both servers
Configuration on SanDisk Fusion ioMemory Flash
⁻
⁻
Backup offloaded from Secondary, so the Read-Only bit can be turned off
This requires only one server to be licensed
SQL Server License Savings:
⁻
32 EE licenses (64 cores)
• @$13,473.00 USD = $ 431,136 (non-SA)
• @$20,209.00 USD = $ 646,688 (SA)
101
Hardware for Nothing, Get Your Flash for Free
I want my...
I want my...
I want my SSDs!
103
#2. The Simple Upgrade Case Study
Scenario
⁻
Four SQL Server instances
⁻
Windows Server 2003 & SQL Server 2005 EOL approaching
⁻
Four physical servers
⁻
SAN attached
Solution
₋
Four SQL Server instances
₋
Windows Server 2012 R2 + SQL Server 2014
₋
Three physical servers
₋
SanDisk PCIe Fusion ioMemory Flash
₋
...with lots of compute & storage headroom
104
104
#2. The Simple Upgrade (cont.)
Consider:
⁻
⁻
⁻
Dual socket quad-core to E5-2699 v3 octo-core, 128GB RAM @$9,943.50
Each 2-core SQL Server EE license @$13,472.50
• 32 logical cores per server >> 16 SQL licenses per server >> $215,560.00
Fusion ioMemory SX 6.4TB card @$39,779
Solution
⁻
Classic hardware upgrade:
•
4 servers, each requiring 16 SQL licenses
•
4 * ($9,944 + $215,560) = $ 902,016
⁻
SanDisk PCIe Application Accelerator solution:
•
3 servers , again each requiring 16 SQL licenses + 3 Fusion ioMemory cards
•
3 x ($9,944 + $215,560 + $39,779) = $ 795,849
⁻
Δ: Savings of $ 106,167
⁻
…with lots of compute & storage headroom
⁻
…enough to buy two more servers & Fusion ioMemory cards
⁻
And a leftover SAN!
105
105
Hardware for Nothing, Get Your Flash for Free
I want my...
I want my...
I want my SSDs!
106
#3. US Air Force & TEKSOUTH
THE CHALLENGE
• Reduce power, cooling, and footprint
• Near real time reporting for 15,000
users running up to 1.2 million queries
per month
• Meet failover requirements
RESULTS
• 3x more concurrent users
• 3x more queries
• 2x greater workload
• Eliminated maintenance overhead for 27
disk arrays and over 400 disks
• 16:1 footprint consolidation
• 1/16th power and cooling
Source: Case Study - TekSouth Fights US Air Force Datacenter Sprawl with ioMemory; http://www.fusionio.com/case-studies/teksouth
USAF Financial Data Warehouse
TEKSOUTH
#3. US Air Force System Changes (cont.)
108
#3. US Air Force System Changes Summary (cont.)
Before
⁻
Database Servers (20RU)
⁻ 5 x 4U servers, four quad-core AMD Opteron™ 822 SE processors @ 3.0GHz, 32GB RAM
⁻ 80 cores, 48 active
⁻
Storage
⁻ Hard disks: 5 x 15K RPM SAS drives, 146GB Storage (81RU)
⁻ 27 x 3U disk arrays
⁻ Hard disks: 15 x 15K RPM SAS drives
After
⁻
Database Servers (6RU)
⁻ 3 x 2U database servers, four hex-core Intel Xeon E5-4610 processors @ 2.40GHz, 64GB RAM
⁻ 72 cores, 36 active
⁻
Storage
⁻ Data stored on one ioDrive2 Duo 2.4TB and one ioDrive2 1.2TB in each server
Savings: (72-48) cores ÷ (2 cores/EE license) ÷ $13,472.50/license = $161,670
Case Study: http://www.fusionio.com/case-studies/teksouth
109
Hardware for Nothing, Get Your Flash for Free
I want my...
I want my...
I want my SSDs!
110
#4. Data Center Scale: Literally saving millions
Scenario
–
–
–
–
Windows Server 2003 & SQL Server 2005 EOL approaching
SAN-attached
Core SQL Server EE licenses @$13,472.50 each
1,500 servers
Solution
–
–
–
–
–
–
–
–
–
Windows Server 2012 R2 leveraging Hyper-V
SQL Server 2014 leveraging AlwaysOn
Fusion ioMemory cards
150 servers
Hundreds of fewer servers
Thousands of fewer licenses
Millions of dollars saved
...with lots of compute & storage headroom
And leftover SANs & shelves & fiber
111
Hardware for Nothing, Get Your Flash for Free
I want my...
I want my...
I want my SSDs!
112
#5. <your company name here>
Scenario
–
–
–
–
Windows Server 2003 & SQL Server 2005 EOS approaching
New hardware likely to accompany software upgrades
SQL Server upgrades require per core licensing
2-Core SQL Server EE license MSRP $13,472.50 (~$6,500/core)
Solution
–
–
–
–
Contemporary multi-core servers
Windows Server 2012 R2 leveraging Hyper-V
SQL Server 2014/2016 leveraging AlwaysOn, In-Memory, Columnstore, etc.
SanDisk Flash options:
• SanDisk Fusion ioMemory cards
• SanDisk front-side SSDs
• SanDisk FlashSoft caching software, SanDisk ION Accelerator™ shared storage,
etc.
113
Hardware for Nothing, Get Your Flash for Free
I want my...
I want my...
I want my SSDs!
114
Hardware for Nothing, Get Your Flash for Free
Takeaways
Challenges:
–
–
–
How to handle millions of IOPs/device, GBs of I/O throughput, µsec latency
What to do with the leftover spinning media
What to do with the leftover budget?
Scenarios such as described here aren’t unusual
They’re typical, & they’re becoming more-&-more common
I reiterate: Actual customers today are literally saving
hundreds of thousands of dollars per server
115
Upgrade for Nothing & the Flash is Free

Customer after customer: Hundreds of thousands, even millions in
savings

Licensing exercise / Consolidation story

EOL: Windows Server 2003 & SQL Server 2005
–
Upgrade h/w
–
Virtualize & Consolidate
116
SanDisk PCIe Application Accelerators
117
Accelerate your time-to-insight
Leveraging Fusion ioMemory Flash

Microsoft certified SQL 2014 solutions

9.2GB/s of I/O Throughput

700% improvement in query execution
times when using Clustered Columnstore
Indexes

Database consolidation made easy.

3TB- 90TB size warehouses
118
Leading the Industry: New! Improved! Gen 3.5
Fusion ioMemory™ PX600
Industry leading reliability
Strongest ECC using LDPC
Fusion ioMemory™ SX350
Capacities up to 6.4TB
Fusion ioMemory
Blade Mezzanine Cards
Capacities up to 1.6TB
From Cisco and HP
Unprecedented performance
119
Industry Leading Performance
4K Random
Read IOPS
4K Random
Write IOPS
Read
Bandwidth
Write Performance in the Industry
Previous Fusion ioMemory™
Read Performance
Write
Bandwidth
Write
Latency
120
Customer Testimonial: Plenty of Fish
At any given time, we can have
over 700,000 users active,
30,000 messages a minute,
500,000 SQL statements per
second across 30 servers.
The ROI is ridiculous. These
cards pay for themselves in two
weeks.
—Owen Morley
Director of Infrastructure
Plenty of Fish
https://youtu.be/zIWgmWEsvpM (2:13)
121
Thank You!
122