SQL Server 2008 R2 Parallel Data Warehouse Steve Busby Data Warehouse Technology Specialist Microsoft Corporation Steve.Busby@microsoft.com Facebook: msstevebus Twitter: @BamaSteveB About Steve 14 year MSFT veteran MCS, then MTC Appdev/SQL background (OLTP) But Infrastructure too Just moved back “home” Graysville, Minor High (‘89) BSEE – UAB Former Southern Company Services Some caveats This presentation is about V1 of PDW V2 is releasing December’ish New HW and SW will be launched at SQLPASS Can’t talk publicly about V2 (NDA only) Significant performance increases and cost decreases Incentive to buy now though (procs vs cores) One more caveat! • I’m not new to Microsoft, but new to role (and I’m rusty on DW) • You guys get to be my lab rats MANAGE ANY DATA, ANY SIZE, ANYWHERE 010101010101010101 1010101010101010 01010101010101 101010101010 A Complete Business Analytic Platform Integration with SQL Server BI Complementary tools Non-Microsoft BI & ETL Tools SQL Server Integration Services, Analysis Services PowerPivot & Power View Hadoop StreamInsight Master Data Services Informatica Business Objects Microstrategy and SAS 6 Microsoft & Data Warehouse Alerts, Notifications SQL Server StreamInsight Big Data Sources (Raw, Unstructured) Data & Compute Intensive Application Business Insights SQL Server FTDW Data Marts Sensors Load Summarize & Load Devices Fast SQL Server Reporting Services Hadoop on Windows Azure Bots Hadoop on Windows Server Interactive Reports SQL Server Parallel Data Warehouse Historical Data (Beyond Active Window) Integrate/Enrich SQL Server Analysis Server Crawlers Performance Scorecards Enterprise ETL with SSIS, DQS, MDS ERP CR M LOB Source Systems APPS MICROSOFT DATA WAREHOUSING SOLUTIONS Reference Architectures offering best price performance Appliance for Data Warehousing requiring highest scalability, performance or complexity Ideal for data marts or small to mid-sized DWs with scan centric workloads Offers flexibility in hardware and architecture Software only Reference Architectures (Software and Hardware) DW Appliance (Fully integrated Software and Hardware) Scale-Up DW Scale-Up DW Scale-Out DW with MPP Scalable and reliable platform Ideal for data marts or small to mid-sized EDWs Tier 1 Services and Support Some SQL Data Warehouses today Big SAN Big 64-core Server Connected together What’s wrong with this picture? Answer: system out of balance • This server can consume 16 GB/Sec of IO, but the SAN can only deliver 2 GB/Sec − Even when the SAN is dedicated to the SQL Data Warehouse, which it often isn’t • System is typically IO bound • Queries are slow Result: significant investment, not delivering performance The Alternative: A Balanced System • Design a server + storage configuration that can deliver all the IO bandwidth that CPUs can consume when executing a SQL Relational DW workload • Avoid sharing storage devices among servers Fast Track Data Warehouse Components Balanced across all components CPU Feed Rate FC HBA A B FC HBA SQL Server Read Ahead Rate A B A B DISK HBA Port Rate DISK A A B STORAGE CONTROLLER CACHE A B FC SWITCH CPU CORES WINDOWS SQL SERVER CACHE SERVER SQL Server 2008 R2 Potential Performance Bottlenecks LUN A B DISK DISK B LUN Switch Port Rate SP Port Rate LUN Read Rate Disk Feed Rate Two SQL DW Infrastructure Options: SQL Classic DW or Fast Track SQL DW SQL Classic DW Architecture Leverages Shared SAN Enterprise Shared SAN Storage Fast Track SQL DW Architecture Architecture modeled after DW Appliances “ Appliance Like” solutions Uses Dedicated SAN arrays and Network Shared Network Bandwidth Dedicated Network Bandwidth Dedicated SAN SQL 2008 Data Warehouse SMP Server OLTP Applications SQL Fast Track DW supports “Scan Centric” DW workloads that are index light Balanced System Scale UP with Fast Track v3 Peak Scan rates 14GBs/sec – Proliant DL980 CPU Socket (8 Core) CPU Socket (8 Core) CPU Socket (8 Core) CPU Socket (8 Core) CPU Socket (8 Core) CPU Socket (8 Core) CPU Socket (8 Core) Fiber Switch Storage Processor Storage Processor Storage Enclosure Storage Processor Dual 8GB Ports X4 per P2000 CPU Socket (8 Core) Storage Processor Storage Processor Storage Processor Storage Processor Storage Processor Storage Processor Storage Processor RAID-1 RAID-1 RAID-1 RAID-1 RAID-1 RAID-1 RAID-1 RAID-1 Storage Enclosure HBA Storage Processor Storage Processor RAID-1 RAID-1 RAID-1 RAID-1 RAID-1 Storage Enclosure HBA HBA RAID-1 RAID-1 Storage Enclosure HBA Server RAID-1 RAID-1 RAID-1 RAID-1 RAID-1 Storage Enclosure Storage Processor HBA RAID-1 RAID-1 RAID-1 RAID-1 RAID-1 Storage Enclosure HBA HBA RAID-1 RAID-1 RAID-1 RAID-1 RAID-1 Storage Enclosure Storage Processor HBA RAID-1 RAID-1 RAID-1 RAID-1 RAID-1 Storage Processor Storage Processor RAID-1 RAID-1 RAID-1 RAID-1 RAID-1 Storage Enclosure SQL Server Parallel Data Warehouse A data warehouse appliance with massive scalability What do you do when the combination of your analytic workload and data volumes exceed the capabilities of SQL Server 2008 R2 and Fast Track Reference Architecture? PDW - Massive Scale and Performance Complete Business Analytics Platform Massive Scale & Performance Simplicity & Value Benefits 16 Parallel Data Warehouse An appliance experience All hardware from a single vendor Multiple vendors to chose from Orderable at the rack level Vendor will: Assemble appliances Image appliances with OS, SQL Server, and PDW software Appliance installed in 1 – 2 days Support: Microsoft provides first call support Hardware partner provides onsite break/fix support Linear Scalability 1 Data Rack • • • • 10 Servers 20 Procs 240 Cores 1.28TB RAM Control Rack DataRack Expand to 4 data racks and quadruple your performance or capacity! Dell/Microsoft PDW Architecture Scales for Resilience and High Performance, with Low Cost of Entry Control Rack Data Racks (up to 4) PowerVault MD3620f Storage Nodes PowerEdge R610 Database Servers Control Nodes (R710) Active / Passive Data Center Monitoring Dual Infiniband Management Servers (R610) Dual Fibre Channel Client Drivers Landing Zone (R510) ETL Load Interface Backup Node (R710 and MD3600f w/MD1200’s) Corporate Backup Solution Spare Database Server Corporate Network Private Network PDW 1 Data Rack Appliance Control Rack Data Rack Compute Nodes Designed Specifically for DW Workloads Control Nodes Storage Nodes SQL Active / Passive SQL SQL SQL True MPP shared nothing architecture SQL Linear scalability by adding additional data racks. Landing Zone Dual Infiniband SQL SQL SQL SQL High Availability Built in Backup Node Dual Fiber Channel Management Servers SQL SQL Spare Compute Node Control Rack– Designated Server Workloads Control Rack An Appliance = 1 Control Rack and 1 to 4 Data Racks Control Nodes Active / Passive Control Node Takes submitted statement and creates D-SQL plan SQL Landing Zone Backup Node Dual Infiniband Management Servers Management Server Patch Management, Active Directory, HPC monitoring Landing Zone Parallelizes loads. Tested to 1-2 TB an hour. LZ has 6.5 TB storage Backup Node Has enough storage to match data configuration. Backup Speeds 4-6 TB an hour Power of 10 Compute Nodes Data Rack Compute Nodes Each Compute Node complete independent SQL server. True MPP architecture. 24 cores, 128GB Ram, per node Storage Nodes SQL SQL SQL Data Rack: 1280 GB RAM 240 Cores Rack Capacity: up to 153 TB Dual Infiniband SQL SQL SQL SQL Scan Rates: up to 3.6 TB min Dual Fiber Channel SQL SQL SQL Spare Compute Node PDW Benefits – Massive Parallel Processing Storage Nodes Compute Nodes Active / Passive SQL Management Servers Landing Zone Backup Node Dual Infiniband Query 1 ? ? ? ? ? ? ? ? ? ? SQL SQL SQL SQL SQL SQL SQL SQL Dual Fiber Channel Control Nodes Query 1 is submitted to SQL Server on Control Node SQL SQL Spare Compute Node Query is executed on all 10 Nodes Results are sent back to client Built In High Availability Failover Clusters Mirrored drives Dual networks Hot swap drive Dual power supplies Dual cooling fans Price your appliance knowing you have no hidden costs for HA. More secret sauce…. MPP Data Layout Options Replicated • A table structure that exists as A full copy within each discrete DBMS instance. Distributed • A table structure that is hashed on a single column and uniformly distributed across all nodes on the appliance. Each distribution is A separate physical table in the DBMS. Basic Physical DB Design in PDW Time Dim Product Dim Date Dim ID Calendar Year Calendar Qtr Calendar Mo Calendar Day Prod Dim ID Prod Category Prod Sub Cat Prod Desc SQL Sales Facts Date Dim ID Store Dim ID Prod Dim ID Mktg Camp Id Qty Sold Dollars Sold Store Dim Store Dim ID Store Name Store Mgr Store Size SQL SQL Mktg Campaign Dim Mktg Camp ID Camp Name Camp Mgr Camp Start Camp End SQL Basic Physical DB Design in PDW Distributed Tables Time Dim Product Dim Date Dim ID Calendar Year Calendar Qtr Calendar Mo Calendar Day Prod Dim ID Prod Category Prod Sub Cat Prod Desc Store Dim Store Dim ID Store Name Store Mgr Store Size SQL SQL Sales Facts Date Dim ID Store Dim ID Prod Dim ID Mktg Camp Id Qty Sold Dollars Sold Larger Fact Table is Hash Distributed Across All Compute Nodes SQL Mktg Campaign Dim Mktg Camp ID Camp Name Camp Mgr Camp Start Camp End SQL SS SS FFFF11 11 SS SS FFFF- 1 1 21 SS SS FFFF- 1 1 31 Create Table Syntax CREATE TABLE FactSales ( ProductKey INT NOT NULL , OrderDateKey INT NOT NULL , DueDateKey INT NOT NULL , ShipDateKey INT NOT NULL , ResellerKey INT NOT NULL , EmployeeKey INT NOT NULL , PromotionKey INT NOT NULL , CurrencyKey INT NOT NULL , SalesTerritoryKey INT NOT NULL , SalesOrderNumber VARCHAR(20) NOT NULL, … ) WITH ( DISTRIBUTION = HASH(ProductKey), CLUSTERED INDEX(OrderDateKey) , PARTITION (OrderDateKey RANGE RIGHT FOR VALUES ( 20010601, 20010901, … ) ) ); Control Node Create table metadata on Control Node Send Create Table SQL to each compute node Create Table FactSales_A Create Table FactSales_B Create Table FactSales_C …… Create Table FactSales_H Compute Node 1 Compute Node 2 … Compute Node 10 FactSale FactSale FactSale FactSale FactSale FactSale s_A s_A s_A s_A s_A s_A FactSale FactSale FactSale FactSale FactSale FactSale s_A s_A s_A s_A s_A s_A FactSale FactSale FactSale FactSale FactSale FactSale s_A s_A s_A s_A s_A s_A FactSale FactSale FactSale FactSale FactSale FactSale s_B s_B s_B s_B s_B s_B FactSale FactSale FactSale FactSale FactSale FactSale s_B s_B s_B s_B s_B s_B FactSale FactSale FactSale FactSale FactSale FactSale s_B s_B s_B s_B s_B s_B FactSale FactSale FactSale FactSale FactSale FactSale s_C s_C s_C s_C s_C s_C FactSale FactSale FactSale FactSale FactSale FactSale s_C s_C s_C s_C s_C s_C FactSale FactSale FactSale FactSale FactSale FactSale s_C s_C s_C s_C s_C s_C FactSale FactSale FactSale FactSale FactSale FactSale s_D s_D s_D s_D s_D s_D FactSale FactSale FactSale FactSale FactSale FactSale s_D s_D s_D s_D s_D s_D FactSale FactSale FactSale FactSale FactSale FactSale s_D s_D s_D s_D s_D s_D FactSale FactSale FactSale FactSale FactSale FactSale s_E s_E s_E s_E s_E s_E FactSale FactSale FactSale FactSale FactSale FactSale s_E s_E s_E s_E s_E s_E FactSale FactSale FactSale FactSale FactSale FactSale s_E s_E s_E s_E s_E s_E FactSale FactSale FactSale FactSale FactSale FactSale s_F s_F s_F s_F s_F s_F FactSale FactSale FactSale FactSale FactSale FactSale s_F s_F s_F s_F s_F s_F FactSale FactSale FactSale FactSale FactSale FactSale s_F s_F s_F s_F s_F s_F FactSale FactSale FactSale FactSale FactSale FactSale s_G s_G s_G s_G s_G s_G FactSale FactSale FactSale FactSale FactSale FactSale s_G s_G s_G s_G s_G s_G FactSale FactSale FactSale FactSale FactSale FactSale s_G s_G s_G s_G s_G s_G FactSale FactSale FactSale FactSale FactSale FactSale s_H s_H s_H s_H s_H s_H FactSale FactSale FactSale FactSale FactSale FactSale s_H s_H s_H s_H s_H s_H FactSale FactSale FactSale FactSale FactSale FactSale s_H s_H s_H s_H s_H s_H Basic Physical DB Design in PDW ReplicatedTables Time Dim Product Dim Date Dim ID Calendar Year Calendar Qtr Calendar Mo Calendar Day Prod Dim ID Prod Category Prod Sub Cat Prod Desc Sales Facts Date Dim ID Store Dim ID Prod Dim ID Mktg Camp Id Qty Sold Dollars Sold Store Dim Store Dim ID Store Name Store Mgr Store Size Mktg Campaign Dim Mktg Camp ID Camp Name Camp Mgr Camp Start Camp End Smaller Dimension Tables are Replicated on Every Compute Node SQL T D S D P D M D SS SS FFFF11 11 SQL T D S D P D M D SS SS FFFF- 1 1 21 SQL T D S D P D M D SS SS FFFF- 1 1 31 SQL T D S D P D M D SS SS FFFF- 1 1 41 Result: Fact -Dimension Joins can be performed locally Storage Node – Drive Array (User Databases) User Database Small Form Factor Drives (SFF) (24) – 8 RAID 1 groups containing a single LUN Large Form Factor Drives (LFF) (11) – 4 RAID 1 groups containing 2 LUNs each LUN1 LUN2 LUN3 LUN4 LUN5 LUN6 LUN7 LUN8 LUN1 LUN2 LUN3 LUN4 LUN5 LUN6 LUN7 LUN8 LUN 9 LUN 9 LUN 9 SFF – 6 disk in a RAID 1+0 group LFF – 2 disk in a RAID 1 group Log LUN 9 LUN 9 LUN 9 SFF - 2 Hot Spares LFF – 1 Hot Spare Hot Spare Hot Spare (SFF) 32 User Database Storage Node – Physical File Layout LUN1 LUN2 LUN3 LUN4 LUN5 LUN6 LUN7 LUN8 FG UDB_Dist_A FG UDB_Dist_B FG UDB_Dist_C FG UDB_Dist_D FG UDB_Dist_E FG UDB_Dist_F FG UDB_Dist_G FG UDB_Dist_H UDBDist.ndf UDBDist.ndf UDBDist.ndf UDBDist.ndf UDBDist.ndf UDBDist.ndf UDBDist.ndf UDBDist.ndf UDB Replicated FG UDBRepl.ndf LUN 9 UDBRepl.ndf LUN 9 UDBRepl.ndf LUN 9 Log LUN 9 LUN 9 LUN 9 UDBRepl.ndf UDBRepl.ndf UDBRepl.ndf UDBRepl.ndf UDBRepl.ndf 8 distribution filegroups 1 replicated filegroup with 8 files 1 log filegroup 33 Parallelism with Distributed Tables 2 Data Rack system • PDW, distributes a single large logical table across 160 tables across the 20 servers. • The distribution is performed by selecting a column in each table and applying a hash function to it. FactOrders OrderID OrderDateKey OrderTimeKey Clustered Index = OrderDateKey FactTrades Partition = OrderDateKey TradeDateKey TradeTimeKey Partition = OrderDateKey CurrentOrderID CustomerKey TotalQty TradeID Clustered Index = OrderDateKey Distribution = OrderID OldOrderID … Distribution = CurrentOrderID 34 Parallelism with Distributed Tables Distribution Compatible Join Node 1 Fact Orders Fact Trades Fact Orders = TradeID OrderID OrderDateKey OrderTimeKey CustomerKey Node 2 Fact Orders Fact Trades = …. TotalQty FactTrades TradeDateKey TradeTimeKey CurrentOrderID OldOrderID Node 20 Fact Orders Fact Trades = SELECT OrderID, COUNT(TradeID) FROM FactOrders JOIN FactTrades ON FactOrders.OrderID = FactTrades.CurrentOrderID WHERE …… 35 Parallelism with Distributed Distribution Incompatible Join Node 1 Shuffle Move Fact Orders Fact Trades Fact Orders = OrderID OrderDateKey OrderTimeKey Node 2 Fact Orders Fact Trades CustomerKey …. TotalQty TempDB = FactTrades TradeID TradeDateKey TradeTimeKey CurrentOrderID TempDB OldOrderID Node 20 Fact Orders Fact Trades = TempDB SELECT OrderID, COUNT(TradeID) FROM FactOrders JOIN FactTrades ON FactOrders.OrderID = FactTrades.OldOrderID WHERE …… 36 Massive Scale and Performance with Parallel Data Warehouse ? ? ? Compute Nodes ? ? Control Nodes ( Active / Passive ? ? ?? ?? ?? Management Servers Load Data Rack Landing Zone LLLLL Backup Node ???? ? ??? ???? ? ??? ? ??? ??? ???? ? ??? ?? ?? ? ??? ???? ? ??? ???? ? ??? ???? ? ??? ?? ?? ? ??? ?? ?? ? ??? Storage Nodes Dual Fiber Channel ? Control Rack Dual Infiniband ? ? Spare Database Server Multiple queries are simultaneously executed across all nodes. PDW supports querying while data is loading. PDW Data Loader Illustrated Distributed Tables Control Rack Data Rack Control Node Active/Passive Database Server Nodes Storage Nodes DWLoader Invoked from Command line DMS Reads Load File and buffers records to Sends to Compute Nodes roundrobin SQL Server PDWEngine Load Manager Creates Staging Tables Load Manager Export Manager DMS Manager DMS Infiniband DMS DMS Ser er Converter Sender Receiver Writer Each row is converted for bulk insert and optionally hashed Hashed row is sent to appropriate node receiver for loading Landing Zone Load File Load Client DMS DMS Converter Sender Receiver Writer Received row is pushed onto writer thread Row is bulk inserted into staging table Distributed Data Warehouse Architecture Departmental Reporting Central EDW Hub Accessible from Anywhere SQL Server SQL Database Landing Zone High-Performance Reporting Regional Reporting SQL Server Analysis Services SQL Server FastTrack 3rd Party Data Integration ETL Tools Data Files 3rd Party RDBMS Getting you there with DW Appliance Support With Microsoft SQL Server 2008 R2 Parallel Data Warehouse you can choose from two support packages: Maintenance Maintenance + PMC Maintenance provides core support Single point of contact for the appliance (hardware and software) DELL 7x24 4hr call to repair1 Unlimited support incidents New Product Version rights Appliance centric software servicing and support lifecycle Premier Mission Critical3 for appliances is designed to maximize business continuity for your most important of solutions: Architectural/Operations review2 Maintain Dedicated Service Engineer (400hrs per year) 1 Risk and Health Assessments per year Enhanced Repair 1 Designed to improve the overall success of the project 30 minute response (priority phone number access) Enhanced Critical Situation escalation process Escalation Manager Executive incident visibility Faster access to the software engineering team Provided by the hardware vendor support group. Included in the MCAP- PMC pricing at no cost between July 2010 and June 2011 – May be delivered by MCS or a Microsoft Partner – will also be provided to customers who do not purchase PMC 3 PMC features listed is basic PMC offering, PMC can be customized and configured to the customers environment. 2 PDW Momentum – Customer Wins in FY12 DIRECTEDGE gets 142X Performance Gain with Parallel Data Warehouse (PDW) Appliance Benefits 42 Benefits 43 © 2012 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.