BIE07-INT
●
●
●
●
●
SQL Fast Track DW Overview
Fast Track DW Implementation Key Principles (Server)
Fast Track DW Implementation Key Principles (Storage Layer)
Fast Track DW Implementation Key Principles (Data Loading)
Q&A
Fast Track DW Overview
●
●
●
•
●
Traditional SQL DW
Architecture
Shared Infrastructure
Enterprise Shared SAN
Storage
Fast Track SQL DW Architecture
Dedicated DW Infrastructure
Architecture modeled after DW Appliances
1TB – 48TB Pre-Tested
Shared Network
Bandwidth
Dedicated Network
Bandwidth
SQL 2008 Data Warehouse
4 Processor 16 Core Server
OLTP Applications
Benefits:
-More System Predictability Thus User Experience
-Pretested Configurations Lowers TCO
-Balanced CPU to I/O Channel Optimized for DW
-Modular Building Block Approach
-Scale Out or Up within limits of Server and SAN
Dedicated Low Cost
SAN Arrays 1 for every
4 CPU Cores
EMC AX4 – HP MSA2312
method
configurations
Best practices
•
•
•
Software:
• SQL Server 2008 Enterprise
• Windows Server 2008
Configuration guidelines:
• Physical table structures
• Indexes
• Compression
• SQL Server settings
• Windows Server settings
• Loading
Hardware:
• Tight specifications for servers, storage and
networking
• ‘Per core’ building block
All databases contain both scans and seeks among with other
types of reads and writes, DW workload indicate that the vast
majority of reads are sequential – not all
Fast Track DW Implementation Key
Principles
Server Layer
CPU Feed Rate
A
B
FC
HBA
SQL Server
Read Ahead Rate
A
B
A
B
DISK
HBA Port Rate
Current Fast Track Architectures are rated at 200
MB/s per CPU core
DISK
A
A
B
STORAGE
CONTROLLER
CACHE
B
FC
HBA
FC SWITCH
A
CACHE
SQL SERVER
WINDOWS
CPU CORES
SERVER
SQL Server 2008 Potential Performance Bottlenecks
LUN
A
B
DISK
DISK
B
LUN
Switch Port Rate
SP Port Rate
LUN Read Rate
Disk Feed Rate
•
•
300 MB/s
500 MB/s
300 MB/s
300 MB/s
MCR 1.6 GB/s
Fiber Switch
500 MB/s
Windows Server OS
HBA
HBA
Server
Storage Enclosure
300 MB/s
500 MB/s
Min
2
GB/s
300 MB/s
300 MB/s
300 MB/s
Min 2 GB/s
500 MB/s
300 MB/s
Storage Enclosure
•
•
150 MB/s
300 MB/s
150 MB/s
150 MB/s
BCR 1.2 GB/s
Fiber Switch
300 MB/s
SQL Server OS
1.2
GB/s
HBA
Server
Storage Enclosure
150 MB/s
300 MB/s
HBA
150 MB/s
150 MB/s
150 MB/s
1.2 GB/s
300 MB/s
150 MB/s
Storage Enclosure
Fast Track DW Implementation Key
Principles
SQL Server 2008 Minimum Server Configuration
SMP Core-Balanced Architecture using Dual Read on HP MSA 2312
S
P
HBA FC 1
4Gb/s or 400MB/s x 2
200MB/s per
Core*
200MB/s per
Core*
HBA FC 2
4Gb/s or 400MB/s x 2
Quad Core CPU
A
01
RAID GP02
02
LUN2
S
P
B
Each SP port rated at 4Gb/s
or 400MB/s and 1600MB/s for all 4 SP ports.
05
LUN5
LUN6
03
LUN3
RAID GP05
04
09
10
LUN0
(Logs)
LUN4
RAID GP03
* Compressed Data
DAE = Disk Array Enclosure
HBA = Host Bus Adapter
SP = Storage Processor
FC = Fibre Channel
Ports = 4Gbs FC
RAID GP01
LUN1
SWITCH
200MB/s per
Core*
200MB/s per
Core*
Using 300GB 15k FC drives
each LUN rated at 125MB/s
each SP controls 4 LUN’s at 500MB/s or 1000MB/s per MSA
DAE
Each SP rated at 500MB/s or 1000MB/s
for both SP’s
Each HBA port rated at 4Gb/s
or 400MB/s and 1600MB/s for all 4 HBA ports.
RAID GP04
06
07
LUN7
08
LUN8
Per MSA2312 Drive Details
• Each MSA can hold 12 drives, this configuration requires 11
• MSA is 2U in total (capacitor eliminates need for battery)
• Each MSA SP port controls 4 LUNs, SP-A also controls LOG LUN
• Each pair of LUNs consists of (2) 300GB 15k FC drives RAID1
HS
LUN 1
LUN 2
LUN 3
LUN16
TempDB
Stage
Database
Permanant_DB
Permanent FG
Permanent_1.ndf
Permanent_2.ndf
Permanent_16.ndf
Permanent_3.ndf
Stage FG
Stage_1.ndf
Local Drive 1
TempDB.mdf (25GB)
Stage_2.ndf
TempDB_02.ndf (25GB)
Stage_3.ndf
Stage_16.ndf
TempDB_03ndf (25GB)
TempDB_16.ndf (25GB)
Log LUN 1
Log LUN 2
Permanent DB Log
Permanent DB Log
Stage DB Log
Stage DB Log
Fast Track DW Implementation Key
Principles
•
1:31
1:31
1:32
1:36
1:33
1:32
Key Order of Index
1:34
1:37
1:35
1:33
1:38
1:34
1:39
1:35
1:40
•
•
•
•
•
•
•
•
Step 2
“Stage Insert”
Step 1
“Base Load”
Step 3
“Transform”
Target Database
Step 4
“Final Append”
Destination Partitioned CI Table
Step 1
“Base Load”
Target Database
Destination Partitioned CI
Table
http://www.microsoft.com/sqlserver/2008/en/us/fasttrack.aspx
http://msdn.microsoft.com/en-us/library/dd459178.aspx
www.microsoft.com/teched
www.microsoft.com/learning
http://microsoft.com/technet
http://microsoft.com/msdn
“At the end of the day, IT operations is really
It’s a free download!
Go to www.microsoft.com/ipd
about running your business as efficiently as you
can so you have more dollars left for innovation.
IPD guides help us achieve this.”
Peter Zerger, Consulting Practice Lead for Management
Solutions, AKOS Technology Services
Fast Track DW Implementation Key
Principles
Core I/O 2 USB, 1 serial, 1 video port,
3 RJ-45 PS2 keyboard/mouse support
Power Supplies 3+3 redundant power
supplies
I/O slots
11 PCIe slots std.,
Option to upgrade to 2 HTx and 7
PCIe
Fans
6 hot plug redundant fans, 3 shown
No single points of failure
in Failover Clustering!
Fast Track DW
Case Study
Current Environment
Proposed Microsoft Platform
Teradata
SQL Server
Fast Track DW
Loading
Subject Area 1
5:10:21 total time
0:51:31 total time
Loading
Subject Area 2
4:36:08 total time
Query times
Subject Area 1
3:03 avg query time
(using 9 benchmark queries)
0:15 avg query time
(using 9 benchmark queries)
Query times
Subject Area 2
56:44 avg query time
(using 4 benchmark queries)
8:09 avg query time
(using 4 benchmark queries)
Comparison
R
6x faster
1:50.01 total time
R
2.5x faster
R
12x faster
R
7x faster
Situation
Business
Needs
Solution
Benefit
Large Retailer with limited capabilities because of their legacy based business
intelligence solution. The solution has capacity for 212 users at the cost of ~1 million in
annual maintenance. Competition – Netezza & Oracle
1) Lower their maintenance cost
2) They wanted to address the business needs (POS data, etc)
3) They also wanted to proliferate the advantages of Business Intelligence across their
enterprise.
Full MS BI stack Fast Track , SSRS, Excel Services , PPS & Office 2007
Our solution will replace and extend the existing DB2 AS400 system
SSIS will replace existing COBOL ETL (including ODI)
1)
2)
ARY01D1v01
•
ARY04D1v07
4MB
4MB
4MB
DB1-1.ndf
DB1-3.ndf
DB1-5.ndf
DB1-7.ndf
ARY01D2v02
ARY02D2v04
ARY03D2v06
DB1-2.ndf
•
•
ARY03D1v05
4MB
4MB
•
ARY02D1v03
ARY04D2v08
4MB
4MB
4MB
DB1-4.ndf
DB1-6.ndf
DB1-8.ndf
Fast Track SMP RA for SQL Server 2008 CPU Core Calculator v2.4
Updated 10/09/2009 - uw
This spreadsheet can be used to estimate the number of cores required to support a user workload and workload mix.
Enter your factors into the green fields and the results will be calculated in the pink cells.
The spreadsheet uses a weighted average to determine the number of cores required based on your inputs.
User Variable Input
Anticipated total number of users expected on the
system
Estimated percent of actual query concurrency
Fast Track DW CPU max core consumption rate
(MCR) in MB/s of page compressed data per core
Estimated compression ratio (default = 2.5:1)
Estimated drive serial throughput speed in
compressed MB/s
Number of data drives in single storage array
Usable capacity per drive
Space Reserved for TempDB
Adjust for
workload mix
Estimated % of
workload
3,000 users
1% concurrency
200 MB/s
2.5 :1
Estimated %
data found in
SQL Server
cache
Estimated Query
Data
Scan Volume MB
(Uncompressed)
Desired Query
Response Time
(seconds)
(under load)
Estimated Disk
Scan volume MB
(Uncompressed)
Simple
70%
10%
8,000
25
7,200
Average
20%
0%
75,000
180
75,000
Complex
10%
0%
450,000
1,200
450,000
100%
100 MB/s
8 drives
272 GB
26%
Calculations and Results
% of core
consumption
rate achieved
Simple
Average
Complex
100%
50%
25%
Arrays
Required based
on throughput
5
Suggested Fast Track RA Server
Requirements
©2009 Microsoft Corporation
No of CPU
cores
32
Expected per
CPU core
consumption
rate (MB/s)
Calculated Single
Query Scan
Volume in MB
(compressed)
200
100
50
Single Array
Throughput in
MB/s
800
2,880
30,000
180,000
Calculated
Target
Concurrent
Queries
Estimated
Target Queries
per Hour
8
3,024
120
9
2,419
1,000
450
12.10
10.00
9.00
30
3,153
3,869
32.00
Throughput in
MB/s for All
Required Arrays
4,000
16
Estimated
Number of Cores
Required
21
6
3
Max
Total Compressed Max achievable
achievable CPU
Data Capacity
IO Throughput
consumption in
(TB)
in MB/s
MB/s
Number of
arrays
Required IO
Throughput in
MB/s
6,400
6,400
Required IO
Throughput in
MB/s
57
3,869
Estimated Single
Query Run Time
(seconds)
0.5
9.4
112.5